Runway Dev’s Model Router selects the right model for every request, so builders don’t have to guess which media model is best when integrating into their products. Instead, define your preference for cost, quality or latency, and the router handles optimal selection at request time. Model Router evaluates incoming prompts and router configurations against Runway’s benchmarking dataset, which tracks qualitative and quantitative model performance.
This report is the first window into how Model Router configurations reduce cost and hold production usability rates in comparison to a state-of-the-art (SOTA) model baseline. Today, 64% of builders using Model Router on Runway Dev have configured their routers for cost optimization. Early feedback shows builders prefer maximum efficiency, with the cheapest model that still delivers production-ready outputs.
Despite this, many builders using generative media APIs default to using the latest SOTA model for every call to guarantee quality. This "better safe than sorry" strategy presents three key issues:
- Over-provisioning: You often don't need an expensive SOTA model for straightforward generation tasks
- Generalist vs. Specialist Trade-offs: A general SOTA model isn't always the top performer for niche or domain-specific use cases
- Resource Constraints: Finding the cheapest model that meets production quality requires time-consuming evaluation loops that most teams don't have the bandwidth to run
Model Router resolves these issues on a per-request basis while continuously adapting as new models are deployed on Runway Dev.
Experimental Setup & Methodology
To test whether dynamic routing preserves quality while lowering costs, we conducted an experiment comparing a SOTA model as a baseline against three Model Router configurations. We generated videos for each setup across a benchmark dataset of 250 image-to-video prompts (text prompt + reference image).
Experimental configurations
We evaluated four total setups across the prompt dataset:
- Baseline Control: Seedance 2.5 (a SOTA video model)
- Quality-Optimized Router: Configured to select the highest-performing model regardless of cost
- Quality-Optimized Router w/ $1.00 Cap: Configured to prioritize quality while enforcing a strict $1.00 per-generation limit
- Cost-Optimized Router: Configured to select the cheapest model that supports the request
Dataset stratification
To ensure broad coverage across real-world production use cases, the 250 prompts were evenly distributed across 10 functional categories:
- Human & character action
- Animals & nature
- Physics & object interactions
- Camera movement & framing
- Text & signage
- Crowds & complex scenes
- Stylized animation
- Product & commercial
- Dialogue & lip-sync
- Abstract & VFX
Rigor & statistical controls
- Blind Evaluation: Output comparisons were double-blinded to eliminate evaluator model bias
- Statistical Significance: Preferences and pass rates were evaluated using paired two-sided t-tests with Benjamini-Hochberg FDR correction (controlling family-wise error rate at alpha = 0.05)
Example
Here are the prompt inputs for one of the scenarios in our dataset and the video outputs generated for each of our test configurations:
Inputs
Text Prompt: “A white storefront awning clearly displays the word OPEN in large black capital letters. A person walks past from left to right but does not enter. Daylight street.”
Outputs
Evaluating Cost & Usable Quality
To measure real-world performance, we tracked production usability and cost on the complete 250-prompt dataset. The results were as follows:
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline (Seedance 2.5) | 78% | $1.80 | — | Seedance 2.5 (100%) |
| Quality Router | 77% | $1.28 | -29% | Seedance 2.5 (56%), Gemini Omni Flash 1.1 (34%), Gemini Omni Flash 1.0 (9%), Happy Horse (1%) |
| Quality + $1 Cap | 74% | $0.61 | -66% | Gemini Omni Flash 1.1 (65%), Gemini Omni Flash 1.0 (34%), Happy Horse (1%) |
| Cost Router | 38% | $0.30 | -83% | Gen-4 Turbo (99%) |
Performance Takeaways
- Quality + $1 Cap captures a balance of quality and cost: Enforcing a $1.00 per-generation limit on the Quality-Optimized Router cuts cost by 66% while maintaining a 74% usable rate, retaining 95% of the baseline SOTA quality at a fraction of the price
- Uncapped Quality Mode yields automatic savings without compromising output: The Quality-Optimized Router delivers near-identical usability to the fixed baseline (77% vs. 78%) while generating 29% cost savings by intelligently shifting lower-complexity prompts away from expensive SOTA calls
- Cost-Optimized Mode is a targeted trade-off for early-stage workflows: While the Cost-Optimized Router slashes spend by 83%, usable output drops to 38%. This setup is best suited for high-volume brainstorming or drafting workflows that route final passes through a quality-optimized router
Rethinking Default SOTA Integrations
Defaulting to the latest SOTA model for every single generation request has become a costly and unnecessary habit in AI engineering. As our evaluation demonstrates, paying top-tier rates for every prompt isn't required to maintain production-ready quality.
Strategic Takeaways:
- Over-provisioning is real: For a large percentage of everyday prompts, mid-tier specialized models pass production quality bars, without SOTA prices.
- Constrained routers capture the sweet spot: Introducing simple operational guardrails, such as a $1.00 per-generation cap, retains 95% of SOTA usable quality (74% vs. 78%) while cutting generation spend by two-thirds.
- Automated routing beats manual eval loops: Instead of spending bandwidth running ad-hoc evaluations every time a new model launches, dynamic routers continually match incoming prompts to the cheapest model that meets your defined quality bar.
Dynamic routing gives product builders granular control over unit economics without forcing a choice between overpaying or shipping broken outputs.
Ready to stop over-provisioning your media pipeline? Explore Model Router on Runway Dev or read our documentation to deploy your first cost-capped router in minutes.
Appendix: Category-Level Benchmark Data
You can see the breakdown of usability across domain-specific workloads represented by each of our 10 prompt categories in the tables below:
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 84% (21/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 12% (3/25) | $0.30 | −83% | Gen-4 Turbo (100%) |
| Quality | 84% (21/25) | $1.66 | −8% | Seedance 2.5 (88%), Gemini Omni Flash 1.1 (12%) |
| Quality + $1 Cap | 68% (17/25) | $0.61 | −66% | Gemini Omni Flash 1.1 (52%), Gemini Omni Flash 1.0 (48%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 76% (19/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 48% (12/25) | $0.30 | −83% | Gen-4 Turbo (100%) |
| Quality | 76% (19/25) | $1.18 | −34% | Gemini Omni Flash 1.1 (52%), Seedance 2.5 (48%) |
| Quality + $1 Cap | 72% (18/25) | $0.61 | −66% | Gemini Omni Flash 1.1 (60%), Gemini Omni Flash 1.0 (40%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 60% (15/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 24% (6/25) | $0.30 | −83% | Gen-4 Turbo (100%) |
| Quality | 56% (14/25) | $0.66 | −63% | Gemini Omni Flash 1.0 (52%), Gemini Omni Flash 1.1 (44%) |
| Quality + $1 Cap | 52% (13/25) | $0.61 | −66% | Gemini Omni Flash 1.0 (92%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 88% (22/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 40% (10/25) | $0.31 | −83% | Gen-4 Turbo (96%) |
| Quality | 84% (21/25) | $1.32 | −26% | Seedance 2.5 (60%), Gemini Omni Flash 1.1 (40%) |
| Quality + $1 Cap | 84% (21/25) | $0.61 | −66% | Gemini Omni Flash 1.1 (92%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 80% (20/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 64% (16/25) | $0.30 | −83% | Gen-4 Turbo (100%) |
| Quality | 80% (20/25) | $0.94 | −48% | Gemini Omni Flash 1.1 (72%), Seedance 2.5 (28%) |
| Quality + $1 Cap | 72% (18/25) | $0.61 | −66% | Gemini Omni Flash 1.1 (88%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 76% (19/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 56% (14/25) | $0.31 | −83% | Gen-4 Turbo (96%) |
| Quality | 76% (19/25) | $1.51 | −16% | Seedance 2.5 (76%), Gemini Omni Flash 1.1 (24%) |
| Quality + $1 Cap | 80% (20/25) | $0.61 | −66% | Gemini Omni Flash 1.1 (88%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 80% (20/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 39% (9/23) | $0.31 | −83% | Gen-4 Turbo (96%) |
| Quality | 84% (21/25) | $1.51 | −16% | Seedance 2.5 (76%), Gemini Omni Flash 1.1 (24%) |
| Quality + $1 Cap | 78% (18/23) | $0.61 | −66% | Gemini Omni Flash 1.0 (56%), Gemini Omni Flash 1.1 (44%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 72% (18/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 48% (12/25) | $0.30 | −83% | Gen-4 Turbo (100%) |
| Quality | 64% (16/25) | $1.28 | −29% | Seedance 2.5 (56%), Gemini Omni Flash 1.1 (40%) |
| Quality + $1 Cap | 52% (13/25) | $0.61 | −66% | Gemini Omni Flash 1.1 (76%), Gemini Omni Flash 1.0 (24%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 92% (22/24) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 16% (4/25) | $0.30 | −83% | Gen-4 Turbo (100%) |
| Quality | 92% (22/24) | $1.76 | −2% | Seedance 2.5 (96%) |
| Quality + $1 Cap | 96% (24/25) | $0.62 | −65% | Gemini Omni Flash 1.1 (92%) |
| Configuration | Usable Rate | Cost / Clip | Spend vs. Baseline | Selected Model(s) |
|---|---|---|---|---|
| Baseline | 68% (17/25) | $1.80 | — | Seedance 2.5 (100%) |
| Cost | 32% (8/25) | $0.30 | −83% | Gen-4 Turbo (100%) |
| Quality | 76% (19/25) | $0.99 | −45% | Gemini Omni Flash 1.1 (36%), Gemini Omni Flash 1.0 (32%), Seedance 2.5 (32%) |
| Quality + $1 Cap | 84% (21/25) | $0.61 | −66% | Gemini Omni Flash 1.1 (52%), Gemini Omni Flash 1.0 (48%) |