AI 正在改变 CI
Anthropic 的工程师平均每个季度提交的代码量是 2021-2025 年间的 8 倍。其中 80% 的代码由 Claude 编写，而且 Claude 在 PR 的审查和批准中也发挥着重要作用。
除此之外，我们代码库中的测试数量增长了 10 倍，而工程师只增加了少量。这一切导致 CI 任务在六个月内增长了 25 倍（如果你想算这笔账：并非每个测试都会在每个 PR 上运行，我稍后会解释）。
这曾多次差点压垮我们的测试影响分析服务。为了避免成为下一个瓶颈，我们把整个系统推倒重来，重新设计了该服务的架构。但走到这一步的道路并不平坦：我们先做了三次快速修补，分别只坚持了 70 天、29 天，然后不到一天。
随着智能体不断加速代码生成和审查，更多工程团队可能很快就会面临 CI 扩展的挑战。我预计，横向扩展的测试选择架构将成为行业标准，因为运行智能体的团队会产生更多的 PR 和更多的测试。
在本文中，我将讨论我们在 Anthropic 如何扩展测试影响分析服务，以及我惨痛学到的教训：永远要为指数增长做好规划。具体的扩展技术——购买更大的机器、并行化进程、或者重启服务（是的，这一招至今依然出奇地管用）——都很常见，并不是本文要传达的洞见。
关键在于，这些技术如今买来的时间收益只是它们一年前的一小部分。另一方面，彻底重构并重新设计一个服务同样只需一小部分时间，而且在编写代码不再是瓶颈的今天，这种方式更具可持续性。
你越能预见这种压力并规划架构如何随之演进，你在权宜之计上浪费的时间就越少。
测试影响分析架构
我的许多同行所在的组织，每次变更时仍然运行所有测试。这在一定程度上行得通，但无法扩展：CI 门禁变得越来越漫长、昂贵且不可信。
此外，人类很擅长判断哪些测试失败与自己无关，而智能体则需要更多的上下文和指引。当它们拿到一组特定的有效测试时，就能更有效地自我验证和迭代。
在 Anthropic，我们构建了一个确定性的测试影响分析（即测试选择）服务，它基于历史表现和包相关性来判断每次变更需要运行哪些测试。这并非罕见做法，已有一类供应商提供该领域的产品。
我们的服务依赖两个确定性组件保持同步：
- 一个“监听器”记录每次 CI 运行的测试结果。
- 一个“选择器”读取测试结果历史，并决定在哪些已打开的 PR 上运行哪些测试。
这很有效，但当每秒有多个 CI 任务运行时，监听器会越来越落后于 PR 队列。对于 AI 原生 SDLC 来说，哪怕一点滞后也可能产生重大影响。例如，监听器滞后 20 分钟可能意味着数以万计的测试更新未能应用到选择器上。
- 如果一次糟糕的变更被合并，某个测试就会开始对其他所有人都失败，导致多次不必要的排查。
- 如果某个依赖开始出现不稳定，那么不稳定的红色失败就会开始阻塞合并。
- 如果某个测试被修复或新增了一个测试，在监听器追上之前它都不会运行，从而带来回归风险。
所有这些都运行在单个进程中，因为为每个测试保留运行历史意味着需要由单一写入者来应用结果。这种 v0 设计使我们无法进行水平分片。
重新设计的坎坷之路
去年 10 月，该服务就已显现出吃紧的迹象，我们连续两天收到告警传呼。
补丁 1：换更大的机器
第一个修复很简单：我们把运行该服务的核心数翻倍。我们也清楚这只是权宜之计。
即便趋势线已经很明显，责任归属仍然模糊不清。没有人愿意再接管另一块基础设施。此外，CI 团队还有更要紧的事要处理。
补丁 2：分片
此时，该服务监听器不断积压的延迟已让我们相当频繁地收到告警传呼。为了推动一些长期修复，我在内部版 Claude Tag 中开启了一个长期会话，专门用于监控该服务。每当监听器积压超过 50,000 个任务时，Claude 就会提醒我，并继续我们关于下一步措施的对话。
这种状况持续了好几个月，而不必不断向它提醒过去的工作或背景信息，这很有帮助。Claude 常常主张进行大重构，但我们通常最终选择再打一个补丁。
到了二月，CI 任务量的指数级增长再次让服务不堪重负。这一次，我们决定做并行化。
监听器并不需要一个单一写入者来正确排序测试结果，它需要的是每个包各有一个写入者，以便正确排序代码库每个部分的测试结果。Claude 为我们生成了代码，把每个包的状态拆分成一个带有自己 worker 的分片。
我们也知道这个修复只是权宜之计，但没想到它只为我们争取了 29 天。
补丁 3：每日重启
到了三月，进程在大多数工作日的中午过后不久就达到内存上限。我们再次寻找快速修复方案，但是：
- 我们只找到了四个 bug。
- 把内存分配器换掉这种快速 hack 毫无效果。我们曾试图优化垃圾回收，但那并不是真正的解决方案。
- 我们不想冒险对一个已经承受高负载的单例（singleton）做内存分析。
- 重启带来的效果还不到一天。
我们还发现，每日重启导致服务逐渐落后得越来越多。当落后超过一个小时时——这种情况发生过几次——大量任务结果没有被监听器记录下来。
需要说明的是，这并不意味着这些 PR 上从未运行过 CI，也不意味着未经测试的代码被推到了生产环境。它意味着监听器没有接收到某些结果，也就是说，我们的测试选择组件在使用过时的数据来决定 PR 上要运行哪些测试、不运行哪些测试。大多数情况下，这体现为我们运行了那些本来就已经非常不稳定或全面失败的测试。
重新设计
是时候（早就该）重新设计这个服务了，我们采纳了 Claude 的建议：给测试选择服务配了一个数据库，确切地说是一个内存数据存储。这样做之后，我们实际上把单例以前承担的一大部分内存处理工作卸载了出去。
现在，任何监听器工作进程都可以处理任何结果，将其追加到内存存储中的日志里，然后继续处理，无需在内存中保存任何东西——无状态，因而可以水平扩展。一个单独的小型消费者进程每隔几秒钟把日志汇总成每个测试的历史记录，选择器就可以快速查询相关的结果历史。
这种分布式架构的运行成本更高，但比起一个摇摇欲坠的单例，它的扩展和内存分析要容易得多。这个项目由一名工程师花了三周完成。一年前，这大概需要接近一个季度。
期间有一些微调（调整 journal 的大小和 worker 数量），这 largely 由 Claude 自主完成，但我们的服务自那以后一直保持稳定。
我会做出的不同选择
如果我被送回 2025 年 10 月，凭借现在所了解的情况，我会以不同的方式处理这个项目和其他项目。
第一个不同之处是，我会把 AI 的指数级增长考虑在内。随着每位工程师平均使用的智能体数量上升，以及加速 PR 审批变得更加成熟，CI 作业会呈指数级增长。
这在 Anthropic 已经随时间改变了 PR 的形态，因为 Claude 更喜欢更小、更细粒度的 PR（这也是不应对每个 PR 运行所有测试的又一个好理由）。这转化为每天更多的 CI 作业。此外，由于智能体会在夜间和周末推送代码，活跃度的下限被抬高了，但由于人类工程师仍然驱动并审批大量 PR，负载仍保持突发性的特点。
我给工程团队的建议是：无论你是自建还是采购，都要假设你的架构会在两个季度内达到 25 倍的负载。“过度设计”这一概念正开始略微淡出，或者说至少门槛已经大幅提高了。只要预算允许，你现在在 v0 设计中就可以为 10-20 倍于预期规模的负载做好准备。
为你的服务配置可观测性工具，让它们成为 Claude 的眼睛和耳朵。这让 Claude 能够爬山式地增量修复问题，比我们手动操作要快得多、好得多。特别是，要确保进入的 CI 任务数量与完成的数量一致。
从一开始就要把状态隔离在进程之外。我还会避免以单实例方式运行任何关键服务，除非你能够对其进行度量以及任何金丝雀变更。CI 的演进速度太快了，不这样做寸步难行。
更多 CI 相关资源
我还写过一篇关于我们如何加速 使用 Claude Tag 的 CI on call（测试版）的文章。
AI is evolving CI
Anthropic engineers on average ship 8x as much code per quarter as they did from 2021-2025. Claude authors 80% of that code and it also plays a large role in reviewing and approving PRs as well.
On top of that, the amount of tests across our codebase grew 10x and we added a nominal amount of engineers. This all led to a 25x increase in CI jobs over a six month period (in case you are trying to do the math, not every test runs on every PR as I will explain).
This threatened to overload our test impact analysis service several times. To avoid becoming the next bottleneck, we blew up the whole thing and reimagined what the service's architecture looks like. But getting there was a bumpy path that started with three quick fixes, which lasted 70 days, then 29 days, and then less than a day respectively.
Scaling CI is a challenge more engineering teams are likely to soon face as agents continue to accelerate code generation and review. I anticipate horizontally scaled test selection architecture will become industry standard as teams running agents create both more PRs and more tests.
In this article, I’ll discuss how we scaled our test impact analysis service at Anthropic and the lesson I learned the hard way: always plan for the exponential. The specific scaling techniques–buying bigger machines, parallelizing processes, or restarting the service (yeah, this one still works surprisingly well) – are common and not the insights to take from this article.
The point is that each of these techniques bought a fraction of the time they did a year ago. On the other hand, overhauling and completely redesigning a service also takes a fraction of the time and is much more sustainable now that writing code is no longer the bottleneck.
The more you can anticipate this strain and plan how your architecture will evolve with it, the less time you will waste on half-measures.
The test impact analysis architecture
Many of my peers work at organizations where every test is still run on every change. This works up to a point, but doesn’t scale: CI gates get increasingly long, expensive, and untrustworthy.
Additionally, humans are great at determining which test failures don’t apply to them while agents will require more context and direction. When they get a specific set of valid tests, they can self-verify and iterate more effectively.
At Anthropic, we built a deterministic test impact analysis or test selection service that determines which tests run on each change based on past performance and package relevance. This isn’t an uncommon practice, and there is a category of vendors with offerings in this area.
Our service depends on two deterministic components staying in sync:
- A “listener” records the test results from every CI run.
- A “selector” reads the test result history and determines which tests run on which opened PRs.
This is effective, but when there are multiple CI jobs running every second, the listener starts to increasingly fall behind the PR queue. For an AI-native SDLC, a small lag can have a big impact. For example, 20 minutes of listener lag can translate into tens of thousands of test updates not being applied to the selector.
- If a bad change gets merged, then a test will start failing for everyone else causing multiple unnecessary investigations.
- If a dependency starts flaking, then flaky reds start blocking merges.
- If a test gets fixed or a new one gets added, it won't run until the listener catches up risking a regression.
All of this ran as a single process because keeping a running history per test meant a single writer needed to apply the results. This v0 design prevented us from being able to horizontally shard.
The bumpy road to redesign
By October of last year the service was already showing signs of strain, and we got paged two days straight.
Patch 1: A bigger machine
The first fix was easy: we doubled the cores running the service. We also knew it would be fleeting.
Even when the trend line was clear, ownership was murky. No one wanted to own another piece of infrastructure. Also, the CI team had bigger fish to fry.
Patch 2: Sharding
At this point we were getting paged pretty frequently by the lag building up in the listener of this service. To drive some long-term fixes, I started a long-running session in an internal version of Claude Tag dedicated to monitoring the service. Anytime the listener lag would get more than 50,000 jobs behind, Claude would ping me and resume our conversation on next steps.
This would go on for months, and it was helpful not having to constantly remind it of past efforts or context. Claude often argued for an overhaul, but we usually settled on another patch.
In February, the exponential growth of CI jobs started to strain the service once again. This time, we decided to parallelize.
The listener didn’t need a single writer to order test results correctly, it needed a single writer per package to order the test results for each section of our codebase correctly. Claude generated the code for us to split each package’s state into a shard with its own worker.
We also knew this fix would be fleeting, but we didn’t realize it would only buy us 29 days.
Patch 3: Daily restarts
In March, the process reached its memory limit by mid-afternoon on most weekdays. Again, we looked for quick fixes but:
- We only found four bugs.
- Swapping the memory allocator as a quick-hack did nothing. We were trying to optimize garbage collection but that wasn’t really the solution.
- We didn’t want to risk memory profiling a singleton already under a heavy load.
- Restarting bought us less than a day.
We also discovered daily restarts were resulting in the service gradually falling further behind. When it fell behind for more than an hour, which happened several times, a ton of job results weren’t recorded by the listener.
To be clear, this doesn’t mean CI never ran on those PRs, or that untested code was pushed to production. What it meant was that the listener didn’t pick up some results, which meant our test-selection component was using stale data to decide what to run and what not to on PRs. Mostly this translated into us running tests that were already super flaky or widespread-failing across the board.
The redesign
It was (past) time to redesign the service, and we took Claude’s advice: we gave the test selection service a database, or an in-memory data store to be exact. By doing so, we effectively offloaded a huge chunk of in-memory processing that the singleton used to do.
Now, any listener worker can process any result, append it to a journal in the in-memory store, and move on without holding anything in memory - stateless and hence, horizontally scalable. A small separate consumer process rolls the journal up into per-test history every few seconds, and the selector can look up relevant result history quickly.
This distributed architecture is more expensive to run, but it is much easier to scale and memory profile than a shaky singleton.This project took three weeks for a single engineer. A year ago it would have been closer to a quarter.
There was some fine tuning (sizing the journal and number of workers) which Claude did largely autonomously, but our service has remained stable since.
What I would do differently
If I was sent back in time to October 2025, I would have approached this and other projects differently with what I now know.
The first difference is that I would account for the AI exponential. CI jobs increase exponentially as the average number of agents per engineer rises and as accelerated PR approval becomes more sophisticated.
This has changed the shape of PRs over time at Anthropic as Claude prefers smaller, more granular PRs (another good reason not to run every test against every PR). This has translated into more CI jobs in a given day. Also, the activity level floor is raised as agents push overnight and on weekends, but it remains bursty as human engineers still drive and approve a significant amount of PRs.
My advice to engineering teams is, whether you build or buy, assume your architecture will be at a 25x load within two quarters. Over-engineering as a concept is starting to slightly fade away, or at least the bar is moving much higher. You can now start to account for 10-20x the perceived scale in your v0 designs as long your budget allows for it.
Instrument your services to act as Claude’s eyes and ears. It allows Claude to hill-climb and fix problems incrementally much better and faster than we could manually. In particular, ensure that the same number of CI jobs coming in equals the same going out.
Keep state out of the process from the start. I’d also avoid running any critical service as a single instance unless you can measure it and any canary changes. CI is evolving too quickly to proceed any other way.
Additional CI resources
I’ve also written how we accelerated CI on call using Claude Tag (beta).