Tessl 博客：AI 代码工厂的瓶颈不是 Review 而是风险

An AI software factory can generate a change, run tests, and open a pull request before a human has finished reviewing the previous one. The queue grows. Review looks like the bottleneck because it is where the work stops. But the harder question sits behind that queue: what would make us comfortable letting this change proceed without someone reading every line?

Answering that requires more than a better reviewer. It requires a system that can prevent some mistakes, demonstrate outcomes, recognize consequential decisions, and contain failures. Those are the conditions under which an agent can earn authority to act.

Why review feels like the bottleneck

Review carries several jobs at once. We ask reviewers to catch defects, interpret intent, check architecture, notice security problems, and judge whether a change is safe to release. When AI increases the supply of code, all those responsibilities converge on the same approval button. Adding reviewer agents may clear part of the queue, but it leaves the underlying decision intact: how much confidence is enough, given what could go wrong?

A clean diff cannot answer that on its own. A one-line change to authorization can carry more risk than a large refactor of an internal reporting tool. Even correct code can cause trouble when it meets production data, unexpected traffic, or a dependency behaving badly. Reading more carefully helps, but it cannot establish every condition under which the software will run.

For an AI factory, the useful shift is from reviewing every diff toward verifying outcomes and evidence, with deeper review reserved for decisions whose consequences warrant it. That shift needs four kinds of control. Each answers a different question, and none can substitute for all the others.

1. Reduce the likelihood

Start by making mistakes less likely. Tests, static analysis, security checks, and policy checks should run as part of the factory’s normal execution. Architectural constraints should be enforceable wherever possible: allowed dependency directions, restricted interfaces, and boundaries around sensitive operations. An instruction in an agent’s context is weaker than a constraint the agent cannot accidentally bypass.

Reviewer and adversarial agents can challenge an implementation, search for missing cases, and probe assumptions. They need room to disagree with the coding agent, including access to the original intent and a way to report unresolved concerns. Agreement between agents is useful evidence only to the extent that they have done meaningful, distinct work. Several agents repeating the same assumption do not make it safer.

Isolated execution gives the factory a place to explore without exposing production systems or credentials. Restricted permissions and disposable environments limit what a mistaken tool call can touch. Together, these controls improve the odds before a change reaches a release decision. They still leave a residual question: did the change accomplish what was asked?

2. Verify the outcome

A passing test suite establishes what the suite checks. It does not automatically establish that the requested outcome has been delivered. Acceptance scenarios should connect the original intent to observable behavior: what users can now do, which failure case is fixed, and which existing behavior must remain intact. Before-and-after evidence is particularly useful because it makes the claimed improvement inspectable.

An independent verification agent should evaluate those scenarios against the running result, with access to the requirements rather than just the implementation agent’s account of its work. Depending on the change, evidence might include test results, API responses, a replay of the failing case, or video of a user flow. A video can show that a workflow succeeds; it cannot, by itself, prove that access controls are correct. The evidence must fit the claim.

The factory should assemble an evidence bundle linked to the intent and the exact version being considered for release. That gives an autonomous decision a traceable basis and gives a human something more useful than a confident completion message. Review can then focus on whether the evidence supports the outcome, what remains untested, and whether that uncertainty is acceptable.

3. Locate the risk

The same evidence can justify different decisions in different parts of a system. A change to an internal dashboard and a change to payment authorization should not inherit the same release policy simply because both passed their tests. The factory needs context about system criticality, customer exposure, data sensitivity, and the blast radius of failure.

Dependencies matter as much as the component being edited. A small utility used across many services may be more consequential than a larger service with few consumers. Dependency centrality, shared infrastructure, and downstream behavior help reveal where a local mistake could spread. Incident history adds context that a repository alone may lack: fragile integrations, recurring failure modes, and areas where previous assumptions have proved unreliable.

Reversibility belongs in this assessment too. Reverting code does not necessarily undo a data migration, an external transaction, or information exposed to the wrong customer. Risk classification should account for the action’s consequences and the available recovery path. When that context is missing, the factory has a reason to narrow its authority or ask for human judgment.

4. Reduce the impact

Resilient architecture belongs near the top of the autonomy discussion. If one faulty change can take down the entire service, the architecture makes every release a high-stakes decision. Isolation boundaries and cells can contain failure to a limited population. Circuit breakers can prevent a struggling dependency from exhausting its callers. These properties determine how much room the factory has to be wrong.

Feature flags and progressive exposure let a change reach a small audience before it reaches everyone. Rate limits and quotas bound the damage an automated action can cause. Tested fallbacks and rollback paths provide ways to recover, while automatic rollback can shorten the time between detecting a regression and containing it. Each mechanism needs to match the failure: switching off a feature is helpful only if doing so actually stops the harmful behavior.

Monitoring and auto-recovery complete the loop. The factory needs signals tied to user outcomes and operating limits, plus clear conditions for stopping, rolling back, or escalating. Those recovery mechanisms also need verification. An agent should not receive more autonomy because a rollback procedure exists on paper; it should receive it because the system has demonstrated that the relevant failure can be detected and contained.

Autonomy is a risk-budget decision

Autonomy is granted over specific decisions: edit code, merge a change, expose it to customers, modify data, or recover a service. Each permission spends some of the organization’s tolerance for uncertainty and failure. The four controls make that decision concrete: how likely is an error, how strong is the outcome evidence, where would failure matter, and how much damage could escape before recovery?

A factory might automatically release a well-tested change to one isolated cell, watch the results, and expand exposure within agreed limits. A change involving sensitive data or an irreversible operation might require stronger verification and a human decision. The distinction comes from the consequences and controls around the action. A single confidence score from the agent cannot capture all of that.

Humans remain responsible for setting those limits and revisiting them as evidence accumulates. Successful runs, near misses, and incidents should inform where authority expands or contracts. This creates a practical route to autonomy: grant bounded permissions, observe what happens, and improve the controls that justify the next step.

Make mistakes survivable

The next improvement to an AI software factory may be a better test, a clearer acceptance scenario, a dependency map, or a smaller failure boundary. Those investments can remove the uncertainty that currently sends every change to a reviewer. They also make the system safer when review misses something, as it eventually will.

AI factories won’t become autonomous by making review perfect. They’ll become autonomous by making mistakes survivable.