Baseten 发布 Baseten Router 编码任务模型路由器，成本降低 72%
Introducing Baseten Router: a Pareto-optimal model router for coding tasks
Baseten 发布编码任务模型路由器 Baseten Router，现已在 Baseten Grid 开放公测。
Please rotate your device
To continue exploring this website, please rotate your device to a vertical orientation
Products
Resources
Dedicated Inference Inference for your specialized models
Model APIs Serve frontier model APIs on day zero
Training Outperform closed models with your own
Sandboxes Give your agents a secure place to run
Grid Control model access and cut AI spend
Resources
* [Blog](https://www.baseten.co/blog)
* [Guides](https://www.baseten.co/resources/type/guide)
* [Webinars](https://www.baseten.co/resources/type/webinar)
* [Security](https://www.baseten.co/security)
* [Events](https://www.baseten.co/resources/type/event)
* [Changelog](https://www.baseten.co/resources/type/changelog)
* [Contact Us](https://www.baseten.co/talk-to-us)
* [Careers](https://www.baseten.co/resources/careers)
Featured
The building blocks of the open frontier Blog / News
Products
Resources
Resources
* [Blog](https://www.baseten.co/blog)
* [Guides](https://www.baseten.co/resources/type/guide)
* [Webinars](https://www.baseten.co/resources/type/webinar)
* [Security](https://www.baseten.co/security)
* [Events](https://www.baseten.co/resources/type/event)
* [Changelog](https://www.baseten.co/resources/type/changelog)
* [Contact Us](https://www.baseten.co/talk-to-us)
* [Careers](https://www.baseten.co/resources/careers)
Featured
The building blocks of the open frontier Blog / News
Product
Introducing Baseten Router: a Pareto-optimal model router for coding tasks
Baseten Router picks the right model for each coding task, staying within 0.8 pts of Opus 5.5 at 72% lower cost. Now in public preview within Baseten Grid.
Authors
Guillaume Troadec
Matthew Blau
Aidan Phillips
Josh Green
Ervin Wang
Alex Tan
Date
November 4, 2026
Table of contents
How we built Baseten Router
Newsletter
Subscribe to our newsletter
Submit
TLDR
Baseten Router sends each coding task to the right model, open or closed, so you pay frontier prices only when the task needs it. In benchmarks, it came within 0.8 percentage points of Opus 5.5 at 72% lower cost. It’s the only router we tested that beat random routing by a statistically significant margin. Baseten Router is part of Baseten Grid, now in public preview.
Today, we’re sharing our initial results from training an intelligent model router for Baseten Grid.
Not every coding task needs the most capable model. As AI coding spend continues to grow rapidly [1] and cheaper open-weight models close the capability gap [2], choosing the right model for each task becomes increasingly valuable. Baseten Router chooses between models based on their expected performance and cost for each coding task. The goal is frontier performance while paying frontier prices only when the task calls for it.
Baseten Router cuts costs by 72% compared to using Opus 5.5 for every task while remaining within 0.8 percentage points of its pass rate, measured on a test set spanning DeepSWE, SWE Atlas QnA, and Terminal-Bench. Adjusting the routing threshold lets users choose various trade-offs along the cost-performance Pareto frontier.
Among the commercial routers available - including those from Fireworks [3], Together AI [4], and Applied Compute [5] - Baseten Router is the only router we evaluated that beats random routing on these benchmarks at the same cost by a statistically significant margin.
Baseten Router shows only a 0.8-percentage point lower pass rate and 72% cheaper cost per task versus using Opus 5.5 alone.
We’re working closely with design partners to specialize Baseten Router to their production coding workloads and prove out performance in the real world. Early feedback is promising, and we will share those results soon. This post walks through how we built competitive performance into Baseten Router and how we continue to refine our approach.
How we built Baseten Router
Formulating the problem
From the input prompt, Baseten Router decides whether the coding task should be completed by a frontier closed-weight model or a cheaper open-weight model. In these experiments, we route between Opus 5.5 and GLM 5.3 Flash. We chose GLM 5.3 Flash because it is a strong coding model that is approximately 80% cheaper than Opus 5.5 across relevant workloads. Our approach can be extended to any model combination in any harness supported by Baseten Grid.
Our aim is to build a model router that decides between two models and gives a better trade-off between success rate and cost than either model alone. We formulated routing as choosing the model with the highest score, where the score is the predicted success rate minus a penalty proportional to the predicted cost:
$$ \pi_{\lambda} \left(\right. x \left.\right) = arg \underset{m \in \left{\right. \text{Opus},\text{GLM} \left.\right}}{max } \textrm{ } \left(\hat{p}\right){m} \left(\right. x \left.\right) - \lambda \textrm{ } \left(\hat{c}\right){m} \left(\right. x \left.\right) $$
π λ(x)=ar g m∈{Opus,GLM}maxp^m(x)−λ c^m(x)
For a given coding task x, \hat p_m and \hat c_m are learned predictions of model m’s success rate and cost in USD on the task, respectively, and \lambda is a configurable weight that controls the trade-off. Intuitively, this favors Opus when its higher predicted success justifies its additional cost.
Illustrative example of Baseten Router, which currently routes between Opus 5.5 and GLM 5.3 Flash
Why start with coding benchmarks
There are two distinct ways to evaluate a router: on coding benchmarks and in production. The router’s performance in production matters most to us and to customers: how much model routing reduces a team’s AI coding spend while preserving the quality of their engineering work.
However, it is challenging to compare results from online evaluations across routers. These tests involve different users and tasks, different definitions of task success, and different methods for calculating cost savings.
Coding benchmarks provide a controlled and transparent framework, enabling us to compare different routers on shared tasks with clear success criteria and cost calculations. Additionally, we believe that the expertise needed to develop a good router on benchmarks is a subset of the expertise needed to perform well in production. Consequently, we view strong performance on coding benchmarks as an important milestone toward deploying in production.
By being methodical about data quality, quantifying noise, and comparing against meaningful baselines, we have built confidence in Baseten Router’s capabilities as we bring it to our customers’ production workloads.
Building the training and test sets
To train Baseten Router, we generated traces on verifiable coding tasks. We built the training set from public coding datasets spanning bug fixes, feature implementation, codebase question answering, and terminal tasks. For each task, we ran both Opus 5.5 and GLM 5.3 Flash at high reasoning effort and recorded the verifier output and model cost.
Baseten Router’s training data mix, broken down by category and benchmark
To evaluate generalization beyond the training set, we used three benchmarks spanning long-horizon coding tasks (DeepSWE 1.1), codebase Q&A (SWE Atlas QnA), and terminal tasks (Terminal-Bench 2.1) as our test set. We selected these benchmarks because they cover categories highly relevant to production coding work and were included in Artificial Analysis's Coding Agent Index (v1.4) [6] and OpenRouter's Router Benchmark (initial release) [7].
An underrated aspect of this process was cleaning the datasets. We corrected grader errors and broken task environments, added safeguards to prevent cheating, and removed training tasks with repositories present in the test set. Additionally, due to the non-deterministic nature of agentic coding, we quantified the uncertainty between repeats of the same task and repeated evaluation runs to improve the robustness of our results. We discuss this further below.
Controlling for noise
Benchmark results are noisy, meaning a router may perform well due to chance rather than genuine capability. To more rigorously control for this, we designed our evaluation to account for two sources of noise:
- Label noise: The same model can pass the same task during one run and fail it during another.
- Selection noise: A random router can route tasks, by chance, to GLM where it succeeds, making it seem more effective than it actually is.
Label noise
Models in production are non-deterministic, meaning the same model can pass a task on one run and fail it on the next. Consequently, a single run is an uncertain measure of how hard a task is for that model. To quantify this uncertainty, we calculate a signal-to-noise ratio (SNR). The signal, \tau^2, measures how much the true gap between Opus’s and GLM’s pass rates varies from task to task. The noise, \sigma^2_O and \sigma^2_G, measures how much each model’s results vary between attempts at the same task. Averaging k_O Opus runs and k_G GLM runs per task shrinks the noise according to the formula:
$$ \text{SNR} = \frac{\tau^{2}}{\sigma_{O}^{2} / k_{O} + \sigma_{G}^{2} / k_{G}} . $$
SNR=σ O 2/k O+σ G 2/k Gτ 2.
The higher the SNR, the more precisely we can measure a router’s performance. With one run per model, the pooled test SNR was 0.27, meaning the test set had almost four times as much noise as signal. By running each test task four times with Opus and eight times with GLM, we raised the SNR to 1.51. This higher SNR makes it easier to distinguish a router that makes better decisions from one that is lucky.
Every test set pass rate mentioned in this blog post is an average over those runs.
Selection noise
To demonstrate that a router has actually learned useful signals, we must show that it beats random routing by more than chance.
A random router is defined as sending a random subset of tasks to GLM and the rest to Opus. Sending more tasks to GLM lowers both expected cost and expected pass rate, so on average every random router lands somewhere between always using Opus and always using GLM.
The figure below plots pass rate against average cost per task. Always using Opus 5.5 (top right) and always using GLM 5.3 Flash (bottom left) are the two ends of random routing. The dotted line between them is the average random router at each cost.
For each fraction of tasks sent to GLM, we simulate 10,000 random routers, each choosing a different random subset. The gray points are a sample of these routers. The gray band spans the middle 90% of their pass rates, from the 5th to the 95th percentile. We call a router’s performance statistically significant if it lies above the band: its pass rate beats 95% of random routers at the same cost.
!Illustrative example of the random routing gray band. Router X lies above the band, so it beats random routing at the same cost by a statistically significant margin. Router Y lies inside the band, so its performance is indistinguishable from random routing. Illustrative example of the random routing gray band. Router X lies above the band, so it beats random routing at the same cost by a statistically significant margin. Router Y lies inside the band, so its performance is indistinguishable from random routing. ###
Experimenting with architectures
Baseten Router’s problem formulation requires it to find signals in a task that predict the value of choosing one model over the other. This section describes the architectures we tested and how early results shaped our final approach. We compared experiments on the training set using leave-one-benchmark-out cross-validation.
Exploring embeddings
We began by embedding the task instructions and fitting separate heads to predict each model’s probability of success and expected cost. Pre-trained embedding models, such as Qwen3-Embedding [8] and EmbeddingGemma 2 [9], are a natural starting point: they capture semantic features of the task without needing to train a language model.
To understand which predictions mattered, we compared three routing setups: predicting success advantage alone, predicting expected cost alone, and predicting success advantage relative to expected cost. Routing on predicted cost alone captured most, but not all, of the initial router gains over random routing.
One hypothesis is that embeddings capture task scope, which is a clearer signal of cost than of which model will succeed. Subsequent analysis supported this: requirement count and estimated code diff size are strongly correlated with cost, but not with the gap in success rate between models.
Combining predictions for both success advantage and expected cost led to the best cross-validation results, so we kept that setup for later experiments.
Comparison of router pass-rate gain by predicting: only cost, only success rate, both cost and success rate
Looking beyond semantic similarity
Having established that the embedding router improved on random routing, we next explored which features were responsible for driving those gains. Some, such as the task category, are legitimate routing signals. For example, implementing a new feature and answering a question about a codebase have different success and cost profiles. However, embeddings can also capture cues specific to the benchmark a task belongs to. Relying on these cues risks learning shortcuts that do not transfer to new benchmarks or to production.
We trained a linear probe to test whether benchmark identity was encoded in the embeddings. This linear probe achieved 89% balanced accuracy, meaning benchmark identity was available as a routing shortcut. Next, we explored whether the embeddings captured useful task-level information. To route well within a category, a router must separate the tasks where a frontier model’s extra capability justifies its extra cost from the tasks where it does not. We found that tasks nearby in embedding space received similar routing scores, but the observed advantage of Opus over GLM on those tasks was only weakly correlated. This suggests that the naive embedding router struggles to learn meaningful representations necessary to distinguish tasks.
These findings motivated two directions: reducing benchmark-specific cues in the representation, and testing whether more explicit task characteristics provide additional routing signal. We tried multiple approaches, including rewriting prompts to strip out benchmark-specific cues, applying corrections to clusters of similar tasks, and extracting structured task attributes. Structured task attributes, such as a task's scope, the domain knowledge it requires, and its level of ambiguity, were particularly useful because they describe the work a task requires more explicitly. We subsequently explored using decision models, specifically elephant-1 and mammoth-1 from Base Labs, to extract these attributes.
Reading cost signals from model activations
In parallel, we explored whether a model’s internal state contained useful routing information [10-12]. We ran GLM 5.3 Flash on each task without generating a response, captured its intermediate activations, and fitted a prediction head to estimate the number of steps in the subsequent response.
Intermediate activations predicted the number of agent steps with a Spearman correlation of approximately 0.6 on our validation set. The signal strengthened with depth, plateaued, and then declined in the deepest layers, consistent with findings in the literature [12]. This supported using intermediate activations to estimate GLM's run length, and by extension its expected cost.
GLM 5.3 Flash’s intermediate layers are predictive of how many steps its response will take
To reduce the router’s latency incurred by retrieving GLM 5.3 Flash’s internal activations, we explored various quantization techniques and the use of smaller Qwen models as proxies for GLM 5.3 Flash’s internal layers. This significantly reduced the router’s end-to-end latency while preserving routing performance.
Putting the experiments together
Our final router combines the most useful signals from our experiments, including structured task attributes and internal model activations, to predict each model's success and expected cost.
Results
We tested Baseten Router on three benchmarks held out from training - DeepSWE 1.1, SWE Atlas QnA, and Terminal-Bench 2.1 - and compared it with routers offered by other providers. The reported pass rate is an average over multiple runs for each task.
Baseten Router is Pareto-optimal
We observed strong generalization on the out-of-distribution test set. Baseten Router performs within 0.8 percentage points of Opus 5.5 at 72% lower cost, and every threshold setting lies on the Pareto frontier of pass rate against cost among the routers we tested.
Baseten Router lies above the gray band, which covers the middle 90% of random routers. This means it beats random routing by a statistically significant margin at every threshold. No other router we tested does so on this set of benchmarks.
Baseten Router offers a range of thresholds to configure where it lies on the Pareto frontier: Cost, Balanced, and Intelligence. These thresholds can be passed in at request time to the same router, without requiring any retraining.
Results on the test set comprising DeepSWE 1.1, SWE Atlas QnA, and Terminal-Bench 2.1, showing the change in pass rate (percentage points) and cost savings relative to always using Opus 5.5. Thresholds are configurable operating points for the same Baseten Router deployment.
Latency
We measured the end-to-end latency of our router across increasing levels of concurrency on a single B200 GPU deployment. At 4 concurrent requests, the p50 and p95 latencies were 273 ms and 429 ms, respectively. This sub-second overhead routing is small relative to a full coding-assistant turn, which typically takes tens of seconds [13]. We can easily scale up the deployment from a single B200 GPU if additional concurrency is required.
Baseten Router’s end-to-end latency on a single B200 GPU deployment Baseten Router’s end-to-end latency on a single B200 GPU deployment ####
Comparing the routers fairly
These results demonstrate Baseten Router’s strong performance on benchmarks selected before running any evaluations. There are also a few factors to keep in mind when interpreting the results.
First, router performance depends on the underlying training distribution and optimization objective. Although most other routers we evaluated also report results on coding benchmarks, they may target different workloads. Our results establish performance on specific benchmarks, which we believe are representative of real software engineering work. Relative router performance may differ on other benchmarks.
Second, model pools are not identical across routers. All but one router offered both Opus 5.5 and GLM 5.3 Flash. One router was limited to Anthropic models. Our comparison evaluates each router as offered, and model pool selection is a choice that forms part of the router being evaluated.
Proving out performance in production
Our next focus is extending these gains from benchmarks to production. The two serve different purposes: benchmarks let us compare routers on the same tasks with the same success criteria, whereas production lets us measure cost savings and quality on real customer workloads.
Routing in production introduces three new challenges:
- Requests are less explicit. A benchmark may describe a bug, identify the affected code, and explain how to reproduce it. In production, an engineer might simply write “fix this” and paste a stack trace.
- Success is harder to measure. Benchmarks come with tests that verify whether a task was solved. A production task has no equivalent label, so training a router on production coding traces means deriving signals from the session, such as whether the user corrects the agent and whether the code is kept in subsequent turns [14].
- Tasks are inherently multi-turn. In a benchmark, the agent receives a single prompt and works until completion. In production, the user reviews the agent’s work, corrects it, and asks for more work in later turns. This means one session can contain several tasks of different difficulties. The best model can change mid-session, but switching models can mean reprocessing conversation context and forfeiting the benefits of a warm KV cache. Consequently, turn-level routers must weigh the benefit of a mid-session model switch against its additional cost.
We’re addressing these challenges by building training and evaluation environments specialized for production requests, a labeling pipeline that derives success signals from real coding traces, and policies for KV-cache-aware turn-level routing.
We’re also working with design partners to specialize Baseten Router to their coding workloads and extend the cost-quality gains from benchmarks into production. Our goal remains the same: frontier performance on every task, at frontier prices only when the task calls for it.
If you’d like to see what Baseten Router can do for your workloads, we’d love to talk.
References
- Menlo Ventures, 2025: The State of Generative AI in the Enterprise
- Epoch AI: Open models lag state-of-the-art closed models by 4 months
- Fireworks AI, Introducing FireRouter with Opus
- Together AI, Together Link: frontier-quality open models in the harness you already use
- Applied Compute, Shift-1, a SOTA coding router
- Artificial Analysis, Coding Agent Index Methodology
- OpenRouter, Model Router Benchmarks
- Zhang et al., 2025: Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- Google, EmbeddingGemma 2
- Varshney et al., 2026: LLM Router: Rethinking Routing with Prefill Activations
- Lugoloobi et al., 2026: LLMs Encode Their Failures: Predicting Success from Pre-Generation Activations
- Skean et al., 2025: Layer by Layer: Uncovering Hidden Representations in Language Models
- Anthropic, 2026: Measuring AI agent autonomy in practice
- Cursor, How Cursor Router chooses the right model for the task
Products
Resources
Legal
Developer
The open frontier is built on Baseten
© 2026 Baseten
The platform for inference, training, and execution | Baseten
来源：Baseten 工程博客 · baseten.co