Markus Eisele 实测智能体编码后提出按验证难度与学习价值决定委托多少代码

I recently used a coding agent to build Task Viewer, a small community helper for IBM Bob that we use internally at IBM. It’s a convenience plug-in to Bob, with no critical production role. Despite helping some users internally, for me it’s a real-world experiment on a project to find out how much of the implementation I can delegate. How much to delegate also depends on what the person doing the task still needs to learn. Someone familiar with the application may already know how to check the result, while a developer learning the system may need to work through part of the implementation to develop that understanding.

Even when I hand all the coding to an agent, my basic requirements for myself are to check functionality and the design manually. I want to use the application and see whether it behaves the way I intended. Does the design make sense when I actually have to find something? Is the information presented in a way that I can follow? Those are questions I still need to spend time with, even when the application is small and the implementation was largely delegated.

For Task Viewer, I also created plenty of UI tests to check that the data appeared correctly. That gave me a way to repeat some of the checks as the application changed. There was still a separate question about the calculations, though. Seeing a number in the right place on the screen does not tell me whether it’s the right number. I ran SQL queries against the underlying data to check the math, going back to the records the application was supposed to represent.

There is quite a bit of understanding involved in that seemingly ordinary step. To check a calculation, you need to know what the result is supposed to mean and which records should contribute to it. You need some expectation against which to compare what the application displays. Thankfully, I had help from the team in uncovering relationships in the data that I would not have been able to derive or even explain on my own.

For this particular application, I could inspect both its behavior and the data behind it, and I had people to turn to when my own understanding was incomplete. Its limited role also gave me room to experiment without making the result part of a production system. Building that confidence took time, because I had to check the behavior, establish what the numbers should mean, and resolve the questions I could not answer myself. All of that, for me, belongs in the assessment of how much coding I could delegate.

This is why I find descriptions such as “The agent got me 80% of the way there” increasingly difficult to interpret. I understand the feeling behind them. You have something running, most of the interface exists, and it looks as though the remaining effort should be the smaller part. But a calculation based on the wrong records may look perfectly reasonable until you find a way to check it. Before correcting anything, you have to discover where your understanding and the application’s behavior diverge. The agent may help with that investigation too, although the amount of code it has already produced tells you very little about how long it will take.

DORA’s analysis of 1,110 responses from Google software engineers describes some of the time saved during code generation being used for auditing and verification, alongside productivity benefits. I would expect to check an application I had implemented myself too, so any comparison needs to include the time it takes to reach a result we are prepared to use. Otherwise, we can make agentic development look unusually expensive by counting checks we leave out of the human version, or unusually cheap by stopping the clock when the agent says it has finished.

Some of that effort can make later changes easier to accept. Once a UI test exercises an important behavior, it can keep doing so as the application changes, and a calculation checked against a small set of records can become a regression test. I wrote about this in my earlier article on the right amount of specification. Turning an expectation into an executable check lets us reuse some of the thinking that went into it. We still have to establish that expectation, though. If the application and its tests share the same mistaken interpretation, running the tests more often will keep confirming the same mistake.

For Task Viewer, those expectations depended on relationships in the data that colleagues helped me understand. I knew how to run a query, but I still needed their explanations before I could judge what it should return. Looking only at the time I spent checking would leave out the knowledge they brought to those conversations, just as looking only at the generated code would leave out my checks. The confidence we need to accept a change can depend on several people understanding different parts of the system.

When someone is still learning one of those parts, the implementation itself may be how they begin to understand it. Following a value from the database to the screen gives them reasons to ask why certain records are selected and how the application uses them. If an agent completes that whole path for them, they may reach a working feature without having answered those questions themselves. The same task can be a sensible delegation for an experienced engineer and an important learning opportunity for someone else.

Shen and Tamkin’s study about how AI impacts skill formation reports on a randomized experiment with 52 developers learning an unfamiliar Python library. Participants with AI assistance scored lower on an immediate assessment of their understanding. But the researchers also observed that those participants who stayed actively engaged with the AI had better learning outcomes, although that observation does not establish which way of using AI led to better learning. A short exercise cannot tell us how an engineer develops over several years, but it gives us a reason to look beyond whether the assigned feature was completed when we assess what someone gained from a task.

We can make a decision about autonomy before the agent starts, agreeing on which parts it should carry through on its own and which parts a developer should attempt. Sometimes the reason to limit autonomy is that we do not yet have a convincing way to check the result. Sometimes we can check the result quite easily, but we want the person doing the task to develop a particular skill. In that second case, deliberately keeping part of the implementation with the developer can be the right choice even when the agent could produce acceptable code.

The right choice can probably be made by asking these questions: How much effort will the team need to establish that this result is correct for its intended use? And what does this developer need to learn by doing this part of the task? A feature may be easy for a colleague to verify because they already know the application, while the person implementing it still needs to learn how it works before taking responsibility for future changes. We do not need to turn every routine change into an exercise, but we can choose tasks that give someone a chance to develop understanding they will need again.

Table 1 applies to a particular task, or a part of one, with a particular developer. “Straightforward to verify” means we have a credible way to assess correctness for the intended use. A passing test suite does not settle that question if we are unsure what the tests should assert. When checking is difficult or we do not yet know how to do it, the first step is to make the uncertainty smaller, with help from someone who understands the area. Giving the whole task back to an inexperienced developer would leave that uncertainty unresolved.

Imagine a new hire adding a “show only open issues” filter to an issue list in a small Java application. An agent could generate the request parameter, the query condition, and the changes to the screen. For someone who already knows the application, that may be a sensible delegation. Someone learning the system also needs to find out which statuses count as open, where the application selects the records, and how the team expects the change to be tested. Discussing those questions with a colleague is part of learning how to contribute to this application.

For this assignment, the developer could follow the existing request through the application and write the first test before asking the agent to implement the filter. Suppose the list shows twenty issues at a time, and the test data contains twenty closed issues followed by five open ones. Selecting “show only open issues” should display those five. If the application first loads twenty issues and then filters them in the browser, the screen will be empty. Working through that difference with a colleague gives the developer an opportunity to understand where filtering and pagination happen. The agent can help with unfamiliar framework code while the developer investigates why the expected issues are missing.

Once the developer can explain the behavior and has a test that catches the mistake, the agent can handle more of the surrounding implementation. A later request to filter by assignee gives them a chance to use what they learned. Can they work out which issues should appear and whether the existing tests cover the combination of filters? Their explanation during review gives the team a basis for deciding how much of that next change to delegate. The first assignment has also given the new hire experience asking about requirements and discussing a change with another developer.

An experienced engineer can also have good reason to limit the agent’s autonomy in an unfamiliar area. I still needed help with Task Viewer’s data, despite being comfortable writing the queries. When a task takes us into a part of the system we do not yet understand, it makes sense to reconsider how much to hand over. A team can make verification easier by improving its tests and sharing what its members know. It can also choose to preserve a learning opportunity after verification has become easy, because another person is going to need that understanding on future changes.

A developer who has been asked to learn through a task needs time to follow the behavior through the application and someone to discuss it with. If we agree on that purpose and then judge the result only by how quickly the feature appeared, we make it difficult for them to keep the learning step for themselves. We may also leave more of the checking with the same experienced colleagues, while giving others fewer chances to develop the knowledge needed to share it.

Before starting, the developer and the colleague supporting them can agree on what the agent will handle, what the developer will do and why, and what would justify changing that division of labor on a later task. For the status-filter example, this means allowing time for the developer to trace the request and write the first test before the agent implements the filter. As that understanding develops, they can revisit the division on a related change.

With Task Viewer, I could delegate much of the coding because I had ways to check the result and colleagues who could help where my understanding was incomplete. I want us to keep creating those conditions as we ask agents to do more. Better checks can reduce the effort needed to accept a change, while time spent learning can help another developer become able to make that judgment. Sometimes that will mean deliberately leaving the first test and investigation with a person, even though the agent could finish them. When we expect that person to take responsibility for the next change, giving them the opportunity to understand this one belongs in our decision about how much to delegate.