信任自动模型路由需要什么：O'Reilly 分析编码助手中的路由与可检验性

When a new coding model comes out, I want to try it. New releases always carry a promise for me. Maybe it can handle a change that needed too much manual intervention before. Maybe it can find the problem an earlier model kept missing. Maybe it can handle some coding scenarios that my favorite model can’t handle today. To discover a new model’s capabilities, I need to spend time with it to find out what I can reasonably hand over to it.

Having said all that, I understand the frustration when a new model becomes available, but your assistant won’t let you choose that model. Somebody has decided which models you can use and when you can use them. You are left trying to decide whether the answer is good enough and can’t get your hands on the new shiny toys. At the same time, I’m quite happy to let an assistant choose a model if that makes my budget last longer. New toys tend to be expensive and top-line models don’t make a difference here. There are plenty of steps in development where a cheaper model might do everything required. Using it for those steps could leave more of the budget available for a difficult investigation later. If routing lets me keep using capable assistance throughout the day, I have a good reason to use it. I want routing to give me more assistance for the same budget. The quality of that assistance depends on how the models, tools, and context handling fit together. And I need enough information about what the assistant did to judge the result.

Getting more help from the same budget

A fixed AI budget puts a fairly ordinary constraint on a team. Or better, on the affected individual.  You have an amount you can spend, and of course you want as much help as possible from it. Each request uses some of that amount. And it really gets frustrating when repeating a request, correcting an answer, and asking the assistant to try again consume even more of the budget.

Automatic routing gives the system a way to manage that spending for you. It can use a cheaper model where that model is likely to succeed and spend more where the additional capability is likely to help. You, the developer, can then concentrate on the task while the assistant handles some of these decisions and you don’t have to constantly worry about which model to pick for which task.

The researchers behind RouteLLM trained routers to choose between a stronger model and a cheaper one using preference data. They reported more than a twofold reduction in cost with minimal impact on response quality on the benchmarks they evaluated. You still need to check how it performs on your own tasks.

The next question is how those savings get to me in my day-to-day jobs. If requests become cheaper to serve, the same subscription could support more of them. A team could keep its assistant available to more people within an existing spending limit. The saving could also make it practical to use a more capable model more often when the task calls for it. So much for the theory. In most assistants that behavior is little more than a “switch.” That connection to better results needs to be visible in the product. If I choose automatic routing, I want to understand what I get from it. Perhaps I can send more requests before I reach a limit. The results still need to meet my expectations, of course. This is a fairly reasonable thing to ask of a feature that spends part of the budget on my behalf.

The time we spend with the assistant belongs in that calculation too. Think about how a cheaper request can lead to another attempt, followed by another explanation, followed by a correction. If this happens often enough, I spend more time getting the code ready to accept than I would have with a more capable model. So when we compare routing policies, we need to follow the task through those interactions. Then we can look at both the money spent and the time we put into getting there and compare both together.

Consider a Java service that sends an order to another service. Occasionally, the request times out, the caller retries, and two orders appear. You ask the assistant to investigate. It can begin by finding the retry code and following its callers, which may be manageable with a cheaper model. As it reads further, it finds a possibility that needs more investigation: The receiving service may have saved the first order before the response was lost. The caller sees a timeout, but that doesn’t tell it whether the order was created.

The assistant now needs to inspect how the receiving service handles a repeated request. A proposed fix should be checked against the failure we are concerned about: save the order, lose the response, and repeat the request without creating a second order. If the assistant can only inspect the caller, it should leave the receiving service’s behavior as an open question. Spending more on a model may help with the reasoning once the relevant code is available, but the assistant still needs access to that code or evidence of its behavior.

The task only becomes clearer as the assistant reads more code. That gives the router more information than it had when it saw the original prompt. It may make sense to spend more on the next part of the investigation, even if the initial steps were straightforward. A recent SWE-Router paper takes this approach: A cheaper model makes a short initial attempt. The router then uses what happened to decide whether to continue with that model or switch to a stronger one. Its benchmark results show that these early actions can improve the routing decision. That seems like a sensible direction for coding assistants: Let what the agent discovers influence how it continues.

The model operates inside a harness

Following the retry through the code also shows how much of this depends on the harness, the software surrounding the model. That’s not only the client running on your machine; it includes the router and backend logic. They provide context and tools, manage the conversation, and control how the agent proceeds.

The assistant needs to find the code that sends the order. It also needs to find out how the receiving service handles a request it has already processed. Once the model has that information, it can reason about the interaction. The harness has to help the agent get there.

This includes some really practical details. Can the agent search the relevant repository? Does it retain the information it found as the conversation gets longer? Can it run a test against the proposed change? If that test fails, does the failure reach the model in a form it can act on? These details affect both the result and the cost of reaching it. An assistant that loses relevant context may need to find the same information again. We pay for those attempts, and we may need to explain the problem again too. When the assistant finds the right code and runs an appropriate test, the result is way easier to review.

A model benchmark tells us how a model performed under particular conditions. The assistant we use has its own instructions and tools. It decides what information to provide to the model and what to do with the response. We experience all of that when we ask it to do something. A new model can improve that experience considerably. I want to use it with a harness that can find the relevant code, retain context, and run the tools the task needs. I also want the people integrating it into my assistant to check how it behaves with their harness.

The same applies when a company runs an assistant on its own infrastructure. It might connect to a model through an approved cloud end point or use a model running on its own hardware. The developer may see the same interface in both cases. Behind that interface, the model may have different tools or different access to the code. We made this case in our O’Reilly book Applied AI for Enterprise Java Development.

Returning to the order example: Suppose the assistant proposes another retry without examining whether the first order was created. The model may have struggled with the problem. It may also have been missing information about the receiving service. Looking at which files it examined and what it checked helps us investigate the failure.

This is exactly the kind of information I would like my assistant to make easy for me to find. It gives me somewhere to start when the result is disappointing. I can try another model, and I can also check whether the assistant found the information it needed. I believe I am less helpless in front of an already very magic system.

But what can a developer judge?

We have a reasonable chance of judging an assistant’s answer when we know the system and can examine what it has done. In a familiar service, I can follow the changed behavior and read the tests. I can then decide whether the change addresses the problem. With an unfamiliar service or programming language, I need more time to understand the code before I can make that decision. The assistant’s explanation can help me get there. It can point to the relevant code and explain how the change affects it. If it says that repeating the order request is safe, I can follow that claim into the receiving service. I need to see how that service handles a request it has already processed.

A clear explanation makes a proposal easier to understand. Checking the proposal takes another step. With that, I have seen confidence develop faster than that understanding. And I always wondered why. In an earlier study of an AI coding assistant, participants with access to an assistant produced less secure code while being more likely to believe their code was secure. The experiment used an older model and specific security tasks, so was not a general-purpose study, but even with that limitation, it shows why confidence about a change deserves attention. Assistant interfaces should show which claims have been checked and which still depend on an assumption. They should also keep an unresolved question visible as the conversation continues. “Review the result carefully” becomes easier to act on when the assistant helps me find the parts that really need review.

Deciding whether the router chose well takes a little more effort. Spotting a mistake tells me something about the answer in front of me. To find out whether another model would have done better, I need a reasonably fair comparison. It should start with similar code, context, and tools. If I change several of those at once, it becomes hard to tell what made the difference. This leaves me in an awkward position: I am responsible for accepting a result, while some of the decisions that shaped it are effectively hidden from me. A model name gives me little more than a reference. After using a model for a while, you develop expectations about what it handles well and when another attempt is likely to be necessary. Those expectations come from your own tasks and experience. If the service does not expose a precise model version, you cannot be sure that the model behind the same name has remained unchanged.

Despite all those drawbacks, access to a new model gives me something to try when my assistant is stuck. So I really can understand why people want that choice. It’s one of the few concrete ways to influence what happens next.

And of course the choice itself comes with expectations. But is this a real effect? In a study of AI model choice, people planning a holiday chose between a model presented as more capable and another as more energy efficient. Both choices routed to the same model, GPT-4o mini. People who chose the supposedly stronger option rated its responses more highly. They selected their own option, though, and had separate conversations. Some of the difference in ratings could have come from the people taking part or from those conversations. A second study gave 48 people functionally identical chatbots labeled “old” and “new.” The researchers varied which appearance received each label and which version people tried first. Thirty-seven preferred the “new” version. The suggestion that one version was newer was enough to influence their preference. Having some control may also change how people feel about a system. In experiments with social media feeds, people reported greater satisfaction when controls were available, even when the controls changed the feed randomly. Later, researchers told participants in the interview study that one feed behaved that way. Most could then identify it. Once they knew what to examine, they could recognize the difference.

These three studies involved general users doing tasks outside software development. We still need to understand how those findings carry over into coding. But for me, the studies suggest that the model name and the choices we make influence our impression of an answer and its quality. But we are fighting the wrong fight here. The real value would be in giving developers a way to check the answer.

The minimum control I want in a coding assistant could be quite modest. Let me ask it to spend more of the budget on a difficult investigation. Let me request a more capable model when repeated attempts are going nowhere. The harness can use that request together with what it has already learned about the task. I want to be able to leave routine routing decisions to the system, and still intervene when I have a reason. Each investigation gives me an opportunity to learn what the assistant handles well. I need to see enough of what happened to make that judgment, and the conversation is a good place to start.

Make the conversation easier to assess

When I look back at a past conversation, I want to understand what the assistant had available, what it did, and what it checked. That gives me a way to decide whether to accept a change or investigate further. The interface could help by showing more of what happens around the model, but it does not necessarily have to.

At a minimum, I need a context breakdown. The messages I type are part of the information the model receives. The harness also supplies system instructions, descriptions of tools, and guidance such as project rules and skills. All of that takes up visual space, but it shows how much space each part uses and how much is reserved for the response so I can act more appropriately in my next steps in the conversation.

Ultimately this makes developer mistakes visible earlier. When you realize that, even a short exchange can involve quite a lot of information. If tool descriptions take a large share, I can look at which integrations the task needs. Perhaps the assistant is configured for several kinds of development MCP servers, while a simple task only needs a few of those tools. I can try a more focused setup and check how it handles a similar task. It still needs access to the information and tools that help it finish the task.

In experiments I’m driving at IBM, I’m connecting this context information with a record of what the assistant did. I can see context changes alongside actions such as reading a file or loading a skill. From there, I can open the relevant part of the conversation. If the assistant delegated part of a task, I can follow that task as well. I have somewhere to begin looking when a long session becomes difficult to follow.

For example, the assistant might read the same file several times and receive identical content. The view can draw my attention to those reads and let me inspect what happened around them. Of course, the assistant may have had a reason to check the file again. I may also decide to ask it to summarize what it has found and continue from there. I can make that decision after looking at the messages and results behind the observation.

Alongside the activity history, I want to see the current context size, the session’s total token usage, and the cost of its requests. A context snapshot tells me how much space was occupied at a particular point. The usage recorded over a session tells me how many tokens its requests consumed, and cost tells me what I paid for those requests. These figures answer different questions, and I need to know which one I’m looking at. An estimate should be labeled as an estimate. If part of the history is missing, that gap should be visible too.

With the history available, I can also follow a claim back to what the assistant actually checked. Take the order example again: I want to see which code the assistant read and whether its test covered losing the response after the order was saved. If it only examined the caller, I still need to understand how the receiving service handles the repeated request. The conversation should keep that question open so I can provide the missing information or ask the assistant to investigate further.

A report that tests passed needs a little context as well. I want to know what those tests covered and which version of the code they ran against. If the assistant edited the code again afterward, I need a new check. And when it wrote both the implementation and the tests, both may follow the same mistaken assumption. I still need to compare them with the behavior we wanted.

In my article on specification, I argued for agreeing on enough of that behavior to make the result checkable. The assistant can keep that agreement available as the conversation gets longer. When it says that a requirement is satisfied, it can point to the code and the check supporting that claim. If a question remains open, I want to see it before I accept the change. It should also be easy to tell what stage the assistant has reached. It might be proposing a change, reporting that it edited a file, or showing the result of a test run. Those are different points in the task. A clear account of what happened helps me decide what to do next.

In practice, most of this can stay fairly brief. Put the checks and open questions beside the proposed change, with links to the actions and results behind them. I can read the summary, then open the part that needs more attention. As I use the assistant on similar tasks, those checks give me a basis for deciding how much I can rely on it.

I also want to understand how routing spends the budget. I already struggle to see token usage in some interfaces. Showing me how many tokens remain in my budget helps me decide how much more I can do. When one session consumes a large part of it, I want to understand where it went. Showing the recorded usage alongside the assistant’s actions would help me connect the spending to what it attempted.

A team can use this information when choosing a routing policy. Try the assistant on tasks that resemble everyday development. Keep the starting conditions similar and follow each attempt through to a result the team can review. Include the corrections and the time developers spent getting there. Then compare how much assistance the same budget provided and whether the resulting code met the team’s expectations. That comparison may also point to something in the harness that needs attention. An assistant that repeatedly retrieves the same information might need better context handling. One that lacks access to a relevant service might need a different set of tools. Those are things we can investigate once we can see what happened during the task.

Automatic model routing can be great when it gives developers more access to capable assistance for the money they already spend. I’m happy to let the assistant choose models for routine steps and spend more when a difficult investigation calls for it. The whole assistant needs to support that arrangement. Its models, tools, and context handling all contribute to the result.

As the industry develops these systems, we need to keep finding better ways to help developers judge the answers they receive. I want to be able to look at a conversation, understand what happened, and decide what to do next. That is how I can build trust in the assistant while letting it take on more of these decisions.