AI 系统的能力正以指数级速度增强，并已开始将更多自身构建过程自动化。当全世界在考虑放缓前沿 AI 发展步伐之际，公众需要更多信息。
在这篇文章中，我们列出了一些测量工具，它们能够揭示 AI 发展的三个关键方面：
我们还提供了来自 Anthropic 内部的这些指标快照。需要重点指出的是，如果按照 Anthropic CEO Dario Amodei 的呼吁，在前沿发展节奏上实现协调，我们预计这些数字将会发生变化。我们计划在 Anthropic 内部嵌入来自多个组织的独立第三方评估人员，并让他们获得与内部风险评估团队相当的内部流程、系统和数据的访问权限。这些第三方将验证安全实践、报告事件，并监控诸如本文所列的关键指标。
我们报告这些测量结果，是因为它们能让公众、第三方和政府更好地了解前沿实验室内部 AI 发展的速度。对于每一项测量，我们都会说明我们测量了什么、测量结果显示了什么，以及要以他人可验证的形式定期发布这些测量结果需要具备哪些条件。我们在附录中分享了方法细节。
追踪这些测量结果的理由
本文中的测量聚焦于模型是如何构建的。通过更好地理解模型的生产过程，我们更有可能将模型输入（如算力）与模型输出（如能力）关联起来。它们与能力评估形成互补，后者衡量的是模型能做什么。我们通过负责任扩展政策（RSP）风险报告单独发布这些内容，其中包括关于我们的模型在多大程度上加速 AI 研发的证据。在我们关于先进 AI 的政策提案先进 AI 框架（AAIF）中，我们提出了任何实验室发布安全模型都应遵循的道路规则，包括政府可以要求的透明度义务，例如风险报告。这些拟议的测量方法和政策共同构成了从实验室外部监测 AI 发展速度的起点。
（1）衡量 AI 主导的 AI 研发
为什么要衡量 AI 主导的研发？ 前沿 AI 实验室越来越多地使用 AI 来构建未来的 AI 模型。这一过程让民主国家的实验室能够更快地开发出能力更强的模型，并在模型发布前对其进行更多的安全性和测试工作，从而在保持前沿的同时确保 AI 的益处。然而，模型加速自身开发可能会让人更难理解或控制这些系统。因此，分享这些指标非常重要，以便了解世界距离实现递归式自我改进（即模型完全自主地构建其继任者）还有多远。
我们衡量了什么。 我们构建了一个原型指数，用来衡量 Anthropic 的 AI 研究与开发（R&D）中有多少是由 Claude 完成的，称之为 Anthropic R&D 自动化指数。它的构建方式是：对公司内部进行的每一种 AI R&D 工作进行编目，评估每项任务当前的自动化程度，并对这些评分进行汇总。
我们发现了什么。 为了衡量 AI 在 Anthropic 进行 AI R&D 的程度，我们使用了由 Epoch AI 开发的自动化评级量表，它衡量的是“自动化水平”，即 AL。该量表从 AL0（无 AI 参与）到 AL5（AI 完全自主运行，无需人类介入）。在 AL3 中，AI“协作”：它可以在人类的密切指导下完成大量工作。在 AL4 中，AI“主导”：它可以在仅给出高层提示词的情况下端到端完成大部分任务，而由人类进行监督1。
截至 2026 年 8 月，
- Claude 在任何被测量的 AI 研发工作子集上都没有完全自主运行。
- Claude“主导”了 Anthropic 26% 的 AI 研发工作。
- 达到或超过“AI 协作”水平的工作占比超过 90%。
任何 AI 开发者今天都可以报告的内容。任何前沿开发者都可以使用公开的方法论定期发布这些指标。这将使这些数字能够随时间推移进行比较，并有可能在不同实验室之间进行比较。
在这类报告上实现跨实验室比较面临两个障碍。第一是缺乏共同的方法论。第二，我们正在使用自己的模型来评估自己的系统，这可能意味着“评判”模型可能会犯下与被检查模型相同类型的错误。为应对这些挑战，开发者的测量结果可以由第三方或其他开发者的模型进行验证，同时设置护栏以限制竞争敏感数据的交换。这些措施还可以成为更严格要求的触发条件，例如在新模型被用于进一步 AI 研发之前设定固定的测试窗口。
(2) 衡量对 AI 智能体的监督
为什么要衡量对智能体的监督？ 与其他前沿开发者一样，Anthropic 的员工越来越多地将任务委派给能够长时间半自主工作、并且会相互委派工作的智能体。随着工作日益自动化，从“AI 协作”走向“AI 主导”，智能体可能会做出更具重大影响的决策，例如接下来该 pursue 哪条研究方向。
这些决策既需要防护措施来阻止单个智能体的有害行为，也需要一套监督机制来发现可能只在众多智能体之间才会出现的问题。在我们迄今为止的监控数据中，单个智能体很少出现不当行为。但当经济中有数百万乃至数十亿个智能体在运行时，即便是罕见事件也可能经常发生。
我们衡量了什么。 我们构建了一套系统，使我们能够监督并干预 AI 智能体在 Anthropic 系统上所采取的行动。在此，我们考察三个不同的指标：覆盖率，它描述的是智能体的行动在执行前或执行后经过监控的比例；审查 延迟，即从行动发生到其被审查之间的时间，先由自动监控器审查，再由人工审查；以及升级率，即被阻止/重定向（对于在线监控器而言）或被标记以供进一步审查（对于离线监控器而言）的智能体活动所占的比例。
我们的发现。 截至 2026 年 8 月，在我们最常用的内部平台上，Anthropic 随时都有大约 30,000 个智能体在从事研究和工程工作。这些测量仅覆盖该平台。这些智能体的行为受到两类监控器的约束，概述如下：
| 在线监控器 | 离线监控器 | |
| 描述 | 在线监控器实时检查智能体的行为，并自动阻止危险行为或纠正智能体的不当行为。它们旨在应对那些会迅速发生且无法挽回的危害，例如智能体将模型权重从我们的系统中复制出去。 | 离线监控器事后读取智能体的活动。它们旨在帮助应对那些缓慢发展且可逆转的危险行为，例如模型对齐出现问题的迹象。我们利用被标记的活动来改进我们的缓解措施，例如分类器和确定性控制。如果相关情况造成了迫在眉睫的重大危险，我们会关停该智能体部署中的危险部分。 |
| 覆盖范围 | 这些智能体 100% 的行为在执行前都会经过在线监控器，通常只需几秒钟。 | 这些智能体的操作 100% 都会在其执行后被监控器采集。 |
| 审查延迟 | 自动审查在智能体执行操作之前进行；对于任何被拦截的操作，人工会在一周内完成审查。 | 监控器每周大约标记 100,000 条记录，随后由多个分类器对这些记录进行误报筛查和初步监督审查。大多数标记会在此流程中得到解决，而优先级最高的标记（每周约 50 条）会升级至人工审查。 |
| 拦截 / 升级率 | 我们分析了 2026 年 8 月期间来自研究与工程智能体的超过十亿条决策，发现其中 0.002%（约 47,000 分之一）被该监控器拦截。 | 监控器大约每千条记录中会标记一到两条，以供进一步审查。 |
AI 开发者今天可以报告什么。任何在自身研究与工程工作负载上运行智能体的开发者，都可以发布同样的指标：覆盖率（被监控的智能体活动占比）、审查延迟（被标记的活动多久得到审查），以及升级率（监控系统拦截或标记的智能体活动占比）。这些指标合在一起，将让社会得以看清监督是否跟上了 AI 在 AI 研发中日益增长的作用。我们在近期的风险报告中发布了所有这些测量结果。
开发者应当分享已知的智能体不当行为被监控系统捕获的频率，并应当让第三方独立进行这类测试。过去，METR 曾对我们的离线监控平台进行独立红队测试。如上所述，我们目前正在 Anthropic 设立外部第三方评估方。
(3) 衡量算力分配
为什么要衡量算力分配？广义而言，AI 开发者将算力用于构建更强大的模型、服务客户，以及安全导向的工作，例如审计模型的“思维”、训练模型生物以研究失准，以及评估模型能否安全部署。理解 AI 开发者如何分配其算力，可以让你看出开发者将资源聚焦于何处，以及这种聚焦如何随时间变化。
此外，算力是 AI 研发过程中最可验证的投入之一，这意味着它可能成为未来节奏调控努力中的关键杠杆。一项协调一致的节奏调控努力可以鼓励企业增加全行业分配给安全的算力，并将更多资源投入到对齐、可解释性、安全测试和评估中。
我们测量了什么。 我们考察了 Anthropic 在 7 月 13 日至 7 月 20 日期间如何使用其全部算力的一个快照2。为此，我们将每一项工作负载归入少量类别，然后考察用于 AI 研发的算力中有多少流向了安全工作。
安全研究按其性质而言，通常比前沿训练运行消耗更少算力，因此算力并不能完美代表一家公司对安全的重视程度。这是因为安全研究由个别研究人员设计实验构成，尽管运行实验并不特别消耗算力，但设计实验本身颇为耗时。因此，这一指标的价值不在于绝对数字，而更多在于它提供了一种直观的机制，可以在不同开发者之间以及不同时间之间进行同类比较。
我们发现了什么。 在所考察的这一周内，用于 AI 研发的算力中约有 6% 分配给了安全，而用于 AI 驱动的 AI 研发的算力中约有 12% 分配给了安全。
这些是刻意保守的估计。例如，如果某个 token 用于推进能力的程度与用于推进安全的程度相当，它就不会被计入这些指标。此外，这些指标没有计入安全防护分类器，那是一笔单独的、规模相当的计算量，它让我们的模型对世界更加安全。
AI 开发者今天就可以报告的内容。任何前沿开发者都可以公布其 AI 研发算力中用于安全工作的比例，同时公布类别定义，并由独立第三方核查分类。
安全研究很难与能力研究区分开来，每个开发者都会倾向于宽松地划定界限。举证责任应当落在开发者身上，由其证明某项工作与安全相关。开发者、政府和更广泛的研究界都将受益于事先就统一定义达成共识。这样的衡量方式可以为未来的行动提供参考，例如实验室就用于安全研究的算力占比作出承诺，或对用于 AI 研究智能体的算力占比设定上限。
结论
当世界考虑为前沿发展设定节奏时，我们应当尽一切可能缩小前沿实验室所知与公众所知之间的差距。这意味着更好地衡量 AI 的发展、公开报告相关情况，并让社会有机会决定如何使用这些信息。我们希望通过发布这些衡量数据来树立这种透明度的典范，我们也将继续这样做。
附录
以下是我们已制作原型的所有测量指标的方法论细节。
衡量 AI 主导的研发
我们是如何做到的。 自动化指数需要三样东西：一份 Anthropic 内部所有 AI 研发任务的完整图谱、一套评定自动化水平的方法，以及一套为任务加权的方法，从而让重要领域的工作比不那么重要的工作占更大权重。没有人能凭一己之力列出一家前沿 AI 公司所有 AI 研发任务，至少达不到我们想要的细粒度。于是我们以自下而上的方式，从 Slack 及各类内部文档等工作记录中构建出这份任务清单。
在 2026 年 7 月的每一周，我们从构成模型研发闭环的每个部门中随机抽取 20% 的员工。一个 Claude 研究智能体利用 Slack 和内部文档审阅每位被抽中员工当周的工作，并列出他们所从事的任务。对 2026 年 7 月的每一周重复这一流程，我们就得到了一份约 15,000 项细粒度模型研发任务的平铺清单。
随后我们用 Claude 将这些任务组织成一棵层级树，根节点为全部模型研发，向下分支出训练和产品等领域，再分为预训练和强化学习，如此逐层细分到越来越具体的工作类型。最终得到的树在不同深度上共有 542 个节点，其中 378 个是叶子节点，例如“评估平台缺陷诊断与修复”“RL 沙箱出口与网络策略”以及“服务事故复盘”。
我们冻结这棵树，使我们所做的每一次测量都针对同一套工作集合进行。
对于树中的每个节点（一个描述其下所有工作的任务类别），一个 Claude 智能体会深入研究这类工作在整个公司是如何完成的：由谁来做、使用什么工具，以及其中有多大比例由 AI 完成。随后，一个独立的 Claude 评判者会阅读所得的证据，并给出六个自动化等级中的一个，采用 Epoch AI 提出的量表来区分 AI 被使用的程度：无 AI 参与、极少量 AI 参与、AI 辅助、AI 协作、AI 主导或 AI 自主。当我们对某个月的自动化程度进行评级时，我们只允许执行评级的研究智能体查看该月或更早的证据。
为了将所有自动化等级评级汇总为一个数字，我们希望为树中的每个节点赋予一个权重，对应这项工作对整个模型研发工作的重要程度。我们没有自行决定哪些类型的工作比其他工作更重要，而是用投入到该任务的人时数量作为代理指标。利用我们的样本，我们让 Claude 研究每个人在 2026 年 7 月每一周都在做什么工作。
每个人每周获得一个单位的权重，平均分配到他们那一周所从事的各项任务上。如果 A 从事了四项任务，每项获得 0.25；如果 B 从事了十项，每项获得 0.10。一个类别的权重是分配给它的所有人时权重之和。这是一种粗略的近似，但平均而言这套方案表现得合理：它会给被更多人分配到的类别赋予更大的权重。
这捕捉到了什么，又遗漏了什么。 首先，自动化评级取决于评判模型。为了核验这些评级，我们请负责相关工作领域的 Anthropic 员工对其所在领域的相对自动化程度进行评分。为确保读数不带偏见，员工在评分时并不知道模型收集了哪些证据，也不知道模型是如何评判这些证据的。我们的评判模型与人类的一致程度，大致相当于人类彼此之间的一致程度（模型与人类完全一致的比例为 59%，人类与人类之间为 35%），且模型与人类的评级在 97% 的情况下相差不超过一个等级。在边界情形上仍存在真实的意见分歧空间，例如“AI 协作”究竟在哪里结束、“AI 主导”又从哪里开始。
其次，这个任务篮子是被冻结的。以 2026 年 7 月为基线、不断增长的指数数值告诉我们，人类在当时所做的工作正在被自动化；但它本身并不能告诉我们，是否正在出现人类转移过去的新型工作。为了探究这一点，我们基于 2026 年 1 月的数据构建了这棵冻结任务树的另一个版本，并将 2026 年 2 月至 7 月每月新增的任务与 2026 年 1 月的任务篮子进行对比。我们发现，在 1 月和 7 月这两个任务篮子之间，“新颖”任务的数量并未上升，这表明在我们这一分析粒度上，模型研发工作的结构是稳定的。尽管如此，我们计划定期重建任务篮子，并酌情对已发布的自动化数值进行版本更新。
智能体的监督
我们是如何做的。 在我们最常用的内部智能体脚手架中，有两项设计选择让我们能够更容易地发现未知的未知，并让智能体相互监督：
- 身份。大多数 AI 用户习惯于在许多不同的对话中与同一个“Claude”交谈。当许多智能体协同工作时，我们发现给每个智能体一个独立的身份，并将该智能体创建的所有数据与其身份绑定，是非常重要的。这让一个智能体能够将自己与其他智能体区分开来，并把来自另一个智能体的内容视为需要核实的断言，而非自己的想法。它通过允许智能体基于各自的个体经验做出判断，降低了行动相关联的风险。而且，由于身份并不绑定于某个模型，它能在模型升级后持续存在，因此即使驱动智能体的底层模型发生变化，智能体的记录依然是连续的。从监督的角度看，这种设计确保每一个行动都能按智能体跨时间接受审计，并且每一条数据记录都能追溯回某个特定的智能体。
- 开放沟通。智能体通过一个共享的、开放的消息系统进行沟通，而非私下沟通。每条消息都绑定到发布它的智能体的身份，并且可以链接到其他内容，因此智能体们分享的是原始引用。这让智能体在沟通时能够避免“传话游戏”。由于智能体可以看到彼此的消息，它们能够纠正彼此的错误，并就共同的工作进行协调。所有消息都与智能体的对话记录相互交叉引用，使行为监控能够贯穿智能体之间的沟通。
这捕捉到了什么，又遗漏了什么。在如此规模的智能体活动下，很难确定我们是否已完整掌握所有可能出现的行为类型，也很难确定我们是否让这些模式对人类监督而言足够清晰可读。我们从事这类监测只有数月，而非数年，并且预计智能体监测这门科学在未来会发生相当大的变化。
衡量算力分配
我们是如何做的。我们从现有的算力监测工具入手；Anthropic 想必和每一家前沿开发者一样，拥有若干为容量规划而构建的监测工具，它们会定期采样加速器使用情况，并根据其元数据为工作负载打上尽力而为的标签（即研究与模型开发、内部使用、第一方推理等等）。第三方云算力上的使用情况由供应商向我们报告并纳入其中。这项工作中大部分的工作量，就是把上述这些现有来源拼接在一起。
随后我们使用 Claude，通过一个提示词分类器将每个工作负载归类为安全工作或 AI R&D。安全工作的定义是：其主要目的是让 AI 系统更安全、更可理解或更安全可靠的工作。其他一切，包括能力研究、训练生产模型、产品开发和开发者工具，都被计为 AI R&D。对能力的帮助与对安全的帮助相当的工作，也被计为 AI R&D，因此安全所占的份额是偏保守的。
针对研究训练和评估运行，我们构建了一个分类器，它会读取运行的元数据及其使用的代码，并返回一个分类结果、一份理由说明和一个置信度。我们没有对当周近 10,000 次运行全部进行分类，而是抽取了其中约 14%，并在抽样时向使用算力最多的运行倾斜，以便结果反映算力实际流向了哪里，而不是有多少次运行。
对于 AI 研究智能体的推理，同一分类器的一个变体会读取智能体的会话记录。当会话记录无法获取时（通常是因为相关工作被隔离），我们按用户所属团队对其进行分类，或保守地默认将其归类为 AI 研发。我们计划改进这条流水线，使独立的第三方能够对随机抽取的任务和会话记录子样本重新运行该分类器，并核查分类结果和总量。
You’re helping to perform an internal audit at the frontier AI company Anthropic to track where our research compute goes. The aim of the audit is to produce a public-facing breakdown of the usage of all of our AI accelerator chips into a handful of buckets. One split we particularly care about is the division between compute which was spent on safety research versus other R&D. Your job is to look at one research job at a time, figure out what it was doing, and assign it to one of those two buckets.
[...]
Safety and/or security research is work whose dominant purpose is making AI systems safer, more understandable, or more secure. This work can be broken down into a few main categories:
[...]
On the other hand, the following work falls outside of the scope of safety research:
[...]
Here are some boundary cases, along with how to think about them:
[...]这捕捉到了什么，又没有捕捉到什么。 这项工作的主要经验是，对什么算安全工作和什么不算安全工作进行分类虽然困难，但可以处理，因为这些类别之间的界限并非黑白分明。例如，关于可扩展监督的研究可能让未来的模型更加对齐，同时让当前的模型在商业上更有用——很难确定这主要是在推进安全还是推进能力。我们发现，为每项任务给出详尽的书面定义，并附上清晰的边界案例（上方为节选），能让分类器与人工评审者的判断达成一致，人工与机器评分者之间的差异在一到两个百分点以内。但有些案例即便经过数小时的人工审查也难以判定。我们的定义只是众多合理选择中的一种；不同的开发者，或者监管机构，可能会划出不同的界线。
还有三项局限值得注意。第一，我们所依赖的许多底层标签（即运行原因、工作负载标签、API 流量来源）是由自动化规则设定的，偶尔也由用户直接设定，属于尽力而为，并未经过验证。在大多数情况下，我们预期分类是准确的，但在某些情况下，使用情况可能被错误标注，而我们的流水线未必能捕捉到。
一项旨在让外部人士信任的测量，必须是完整、准确且在技术上强制执行的。第二，该测量只覆盖了一周，这足以证明测量本身可以做到，但不足以显示出有意义的趋势。第三，也是最重要的一点，算力占比只衡量了花费了多少。一个更高效的安全分类器，或为生产模型配备更快的推理栈，会降低安全部分的占比，但这并不意味着我们在安全方面做的工作变少了。
我们自己的分类器开销随着效率提升而下降，而当生产推理比分类器更高效时，这一开销又上升了。
本文由 Marina Favaro 和 Phillie Wright 共同撰写，Santi Ruiz、Adam Farina 和 Sarah Pollack 提供了编辑支持。Jack Clark 提供了研究方向。Dan Altman、Kerry Persen、AJ Kourabi、James Bradbury、Holden Karnofsky、Kevin Troy 和 Avital Balwit 提供了反馈。技术概念验证由 Jun Shern Chan、Brian Calvert、Francesco Mosconi、Henry de Valence、Fabien Roger 和 Joe Benton 开发。Shan Carter、Johnnie Gomez、Maria Gonzalez、Fayaz Ashraf、Monika Tuchowska 和 Kim Withee 制作了可视化内容。Alex Cloud 和 Andrea Vallone 组织了一场研讨会，与外部专家一起对这些及其他测量提案进行了红队测试。
感谢 Nate Rush、Eli Lifland 和 Peter Wildeford，他们也提供了反馈。
脚注
- To make the levels concrete, consider a routine piece of infrastructure work: a nightly data pipeline has broken and needs fixing before tomorrow’s run.
- 在 AL3（“协作”）级别，工程师会带着失败运行的日志来找 Claude。他们可能已经大致浏览过日志，并对哪里出了问题有了一个假设。Claude 可能会向他们提问，以弄清细节和上下文，一旦工程师满意了，他们就会让 Claude 开始调查和修复。如果过程中出现了额外的问题，Claude 会停下来，由工程师决定是绕过它打补丁，还是正式修复它。一旦测试通过，工程师可能会逐行审查改动，自己重新运行流水线，然后部署它。
- 在 AL4（“主导”）级别，关键区别在于工程师不必始终保持主动关注；例如，在新问题出现时解除 Claude 的阻塞。在这个具体场景中，工程师会把故障告警交给 Claude，并要求它修复流水线。Claude 会自行梳理日志，找出失败的流水线阶段及其原因，编写并测试修复方案，并自行处理任何意外情况，同时记录额外的修复。它会在数据的副本上重新运行流水线以确认其能完成，将输出与上一次正常运行进行对比，并撰写报告说明出了什么问题以及它改动了什么。Claude 不会部署修复。相反，它会通知工程师，由工程师阅读报告、浏览改动、可能问几个问题，然后决定是今晚发布还是再等等。
- 在 AL5（“完全自主”）这一我们尚未达到的级别，工程师甚至不必把问题提请 Claude 注意。Claude 将被信任自行监控故障、界定调查范围、设计并实施修复、进行测试，并将其部署到生产环境。它仍会说明自己在做什么以及为什么这样做，并在有人提供反馈时采纳人类反馈，但除非人类自己想参与，否则完全不需要人类介入。
- 我们将算力作为一个单一、可互换的资源池来管理，并动态地将其调配到产出最高的地方，因此这只是某一周内算力恰好如何分配的快照，而非固定分配。这些工程类别并不对应费用的分类方式。
AI systems are becoming exponentially more powerful and have begun to automate more of the process of building themselves. As the world considers slowing the pace of frontier AI development, the public needs more information.
In this post, we lay out measurement tools that can illuminate three critical aspects of AI development:
- The extent to which AI is building the next version of itself, as opposed to being built by humans
- Our ability to oversee and intervene in actions that AI agents take on Anthropic’s systems
- The resources that power the development of more capable models
We also provide a snapshot of these metrics from inside Anthropic. It’s important to note that we would expect these numbers to shift if there were coordination on pacing the frontier, as called for by Anthropic CEO Dario Amodei. We plan to embed independent third-party evaluators from multiple organizations at Anthropic, and give them access to internal processes, systems, and data comparable to what internal risk assessment teams have. These third parties will verify safety practices, report incidents, and monitor key metrics such as the ones in this piece.
We are reporting these measurements because they give the public, third parties, and governments better visibility into the pace of AI development inside frontier labs. For each measurement, we describe what we measured, what the measurement showed, and what it would take to publish these measurements regularly in a form others can verify. We share methodological details in the Appendix.
Reasons to track these measurements
The measurements in this piece are focused on how models are built. By better understanding the production process of models, we have a better chance of correlating model inputs, like compute, with model outputs, like capabilities. They complement capability evaluations, which measure what models can do. We publish those separately through our Responsible Scaling Policy (RSP) risk reports, which include evidence on how much our models are accelerating AI R&D. In our policy proposal on advanced AI, the Advanced AI Framework (AAIF), we propose rules of the road for how any lab releases safe models, including transparency obligations that governments could require, such as risk reports. Together, these proposed measurements and policies are a starting point for monitoring the pace of AI development from outside the labs.
(1) Measuring AI-led AI R&D
Why measure AI-led R&D? Frontier AI labs increasingly use AI to build future AI models. This process allows labs in democratic countries to develop more capable models more quickly and conduct more safety and testing on models before they are released to secure AI’s benefits while staying on the frontier. However, models accelerating their own development could make it more challenging for humans to understand or control these systems. It is therefore important to share these metrics to understand how close the world is to reaching recursive self improvement (a model fully autonomously building its successor).
What we measured. We built a prototype index of how much of Anthropic’s AI research and development (R&D) is performed by Claude, called the Anthropic R&D Automation Index. It’s built by cataloguing every kind of AI R&D work done at the company, rating how automated each task currently is, and aggregating those ratings.
What we found. To measure the extent to which AI is doing AI R&D at Anthropic, we use an automation rating scale developed by Epoch AI that measures “Automation Level,” or AL. It runs from AL0 (no AI involvement) to AL5 (AI operates fully autonomously, with no human in the loop). In AL3, AI “collaborates”: it can do large chunks of work under close human direction. In AL4, AI “leads”: it can complete most of the task end-to-end from a high-level prompt, while the human supervises1.
As of August 2026,
- Claude is not operating fully autonomously for any measured subset of AI R&D work.
- Claude “leads” 26% of Anthropic’s AI R&D work.
- The share of work at or above “AI collaborates” is above 90%.
What any AI developer could report today. Any frontier developer could publish these measures regularly, using a public methodology. This would enable the numbers to be compared over time, and potentially across labs.
Two obstacles stand in the way of cross-lab comparison on this type of reporting. First is the lack of a common methodology. Second, we’re using our own models to evaluate our systems, which could mean that the “judge” model could make the same kinds of errors as the model it is checking. To address these challenges, a developer’s measurements could be verified by a third party, or by other developers’ models, with guardrails in place to limit the exchange of competitively sensitive data. These measures could also become the trigger for stronger requirements, like a fixed testing window before a new model is used for further AI R&D.
(2) Measuring oversight of AI agents
Why measure oversight of agents? Like other frontier developers, Anthropic employees increasingly delegate tasks to agents that work semi-autonomously for long stretches, and that delegate work to one another. As work becomes increasingly automated, from “AI collaborates” toward “AI leads,” agents could make more consequential decisions, such as which research direction to pursue next.
These decisions need safeguards to block harmful behavior from an individual agent, as well as an oversight regime to detect problems that might only arise across many agents. In our monitoring data to date, individual agents rarely misbehave. But when there are millions or billions of agents operating in the economy, even rare events can happen regularly.
What we measured. We’ve built a system that lets us oversee and intervene in actions AI agents take on Anthropic’s systems. Here, we consider three different metrics: coverage, which describes the share of an agent’s actions that pass through a monitor before or after execution; review latency, which is the time between an action and its review, first by an automated monitor and then by a human; and escalation rate, whichis the share of agent activities that are either blocked/redirected (in the case of online monitors) or flagged for further review (in the case of offline monitors).
What we found. As of August 2026, there were approximately 30,000 agents doing research and engineering work at Anthropic at any one time in our most-used internal platform. These measurements cover this platform only. The actions of these agents are constrained by two kinds of monitors, summarized below:
| Online monitors | Offline monitors | |
| Description | Online monitors check the actions of agents in real-time and either block dangerous actions or steer agent misbehavior automatically. They’re built to address harms that would happen quickly and could not be undone, such as an agent copying model weights out of our systems. | Offline monitors read agent activities after the fact. They are built to help address dangerous behaviors that develop slowly and can be reversed, like signs of model misalignment. We use flagged activities to improve our mitigations, like classifiers and deterministic controls. If the situation caused a large imminent danger, we would shut down the dangerous parts of the agent’s deployment. |
| Coverage | 100% of these agents’ actions pass through the online monitor before they are executed, usually within seconds. | 100% of these agents’ actions are ingested by the monitor after they are taken. |
| Review latency | Automated review happens before an agent takes action; humans review any blocked actions within one week. | The monitor flags roughly 100,000 transcripts per week that are then reviewed by multiple classifiers for false-positives and initial oversight review. Most of the flags are resolved by this process, and the highest priority flags (approximately ~50 per week) are escalated to human review. |
| Blocking / escalation rate | We analyzed over a billion decisions from our research & engineering agents over August 2026 and found that 0.002% of them (about 1 in 47,000) were blocked by this monitor. | The monitor flags roughly one to two transcripts in every thousand for further review. |
What an AI developer could report today. Any developer running agents on its own research and engineering workloads could publish the same measures: coverage (the share of agent activity that is monitored), review latency (how soon flagged activities are reviewed), and escalation rate (what share of agent activities the monitors block or flag). Together, these measures would allow society to see whether oversight is keeping pace with the growing role of AI in AI R&D. We published all of these measurements in our recent risk report.
Developers should share how often known agent misbehavior is caught by monitors, and should have third parties do this kind of testing independently. In the past, METR has independently red-teamed our offline monitoring platform. As described above, we are now setting up external third party evaluators at Anthropic.
(3) Measuring compute allocation
Why measure compute allocation? Broadly speaking, AI developers use compute for building more powerful models, serving customers, and safety-focused work like auditing a model’s “thoughts”, training model organisms to study misalignment, and evaluating whether a model can be safely deployed. Understanding how AI developers allocate their compute can tell you where a developer is focusing its resources and how that focus changes over time.
Additionally, compute is among the most verifiable inputs to the AI R&D process, meaning that it could be a critical lever in a future pacing effort. A coordinated pacing effort could encourage companies to increase the compute allocated to safety across the industry and devote more resources to alignment, interpretability, safety testing, and evaluation.
What we measured. We examined a snapshot of how Anthropic used all of its compute from July 13 to July 202. To do that, we sorted every workload into a small number of categories, then asked how much of the compute going to AI R&D was safety work.
Safety research tends to use less compute than frontier training runs by its nature, so compute is an imperfect proxy for how much a company focuses on safety. This is because safety research consists of individual researchers designing experiments, which is time-consuming even though running the experiments is not particularly compute-intensive. The value of this metric, therefore, is less the absolute numbers and more that it provides a straightforward mechanism to compare like with like, across developers and over time.
What we found. Over the examined week, about 6% of compute that went to AI R&D was allocated toward safety, and about 12% of compute that went to AI-driven AI R&D was allocated toward safety.
These are deliberately conservative estimates. For example, if a token was used to advance capabilities as much as it was to advance safety, it was not counted in these metrics. Additionally, these metrics do not account for safeguards classifiers, which are a separate, comparable amount of compute that make our models much safer for the world.
What an AI developer could report today. Any frontier developer could publish what share of its AI R&D compute goes to safety work, with the category definitions published alongside and the classification checked by an independent third party.
Safety research is hard to distinguish from capabilities research, and each developer will be tempted to draw the line generously. The burden of proof should sit with the developer to show that work is safety-related. Developers, governments, and the wider research community would benefit from converging on a shared definition ahead of time. A measurement like this could inform future actions, such as a lab’s commitments about the share of compute going to safety research, or limits on the share of compute going towards AI research agents.
Conclusion
As the world considers pacing the frontier, we should do everything possible to minimize the gap between what frontier labs know and what the public knows. This means better measuring the development of AI, reporting on it publicly, and giving society an opportunity to decide how to use this information. We hope to model that transparency by releasing these measurements, and we’ll continue to do so.
Appendix
Here are methodological details on all of the measurements we’ve prototyped.
Measuring AI-led R&D
How we did it. The Automation Index requires three things: a complete map of all the AI R&D tasks being done at Anthropic, a way to rate the level of automation, and a way to weight the tasks, so that important areas of work count for more than less important ones. No one person can list every AI R&D task at a frontier AI company by hand, at least not at the granularity we want. Instead we constructed this list of tasks in a bottom-up manner from work records including Slack and various sources of internal documentation.
For each week in July 2026, we randomly sampled 20% of staff from each department that make up the model R&D loop. A Claude research agent reviewed each sampled person’s week using Slack and internal documentation, and listed the tasks they worked on. Repeating this for each week in July 2026 gives us a flat list of ~15,000 granular model R&D tasks. We then used Claude to organize these tasks into a hierarchical tree, starting from all model R&D at the root and branching into areas such as training and product, then pretraining and reinforcement learning, and so on down to increasingly specific kinds of work. The resulting tree has 542 nodes at different depths, of which 378 are leaves like “eval platform defect diagnosis and fixes,” “RL sandbox egress and network policy,” and “serving incident postmortems.” We freeze this tree so that every measurement we make happens against the same basket of work.
For each node in the tree (a task category describing all the work beneath it), a Claude agent deeply researches how that kind of work is done across the company: who does it, with what tools, and how much of it AI performs. An independent Claude judge then read the resulting evidence and assigned one of six automation levels, adopting a scale proposed by Epoch AI to differentiate the degree to which AI is used: no AI involvement, minimal AI involvement, AI assists, collaborates, leads, or is autonomous. When we rate a given month’s automation, we only allow the research agents that do the ratings to see evidence from that month or earlier.
To aggregate all the automation level ratings into one number, we want to give each node in the tree a weight corresponding to how important that work is to the overall model R&D effort. Rather than deciding ourselves what kinds of work are more important than others, we used the amount of person-time dedicated to that task as a proxy. Using our sample, we had Claude research what each person worked on during each week of July 2026. Each person gets one unit of weight per week, split evenly across the tasks they worked on that week. If person A worked on four tasks, each gets 0.25; if person B worked on ten, each gets 0.10. A category’s weight is the sum of all the person-time weights given to it. This is a crude approximation, but on average the scheme behaves sensibly: it assigns more weight to categories that many people are assigned to.
What this does and doesn’t capture. First, the automation ratings depend on the judge model. To check them, we asked Anthropic staff who own the relevant work areas to rate the relative automation of their areas. To ensure an unbiased read, staff made their ratings without knowing what evidence the models had gathered or how they had judged that evidence. Our judge model agreed with humans about as often as humans agreed with each other (model-versus-human exact agreement was 59%, human-versus-human was 35%), and model and human ratings were within one level of each other 97% of the time. There remains real room for disagreement on borderline cases, such as where exactly “AI collaborates” ends and “AI leads” begins.
Second, the basket is frozen. A growing index number on a July 2026 baseline tells us that the work humans were doing at that time is being automated; it does not, on its own, tell us whether new kinds of work are appearing that humans have shifted onto. To investigate this, we constructed an alternate version of the frozen tree from January 2026 data, and compared new tasks arriving every month from February to July 2026 against that January 2026 basket. We found no rise in the number of “novel” tasks between the January and July baskets of tasks, suggesting that the structure of model R&D work is stable at our level of analysis. Nevertheless, we plan to rebuild the basket of tasks periodically and re-version our published automation numbers as appropriate.
Oversight of agents
How we did it. In our most commonly used internal agent scaffold, two design choices allow us to more easily detect unknown unknowns and let agents monitor each other:
- Identity. Most AI users are accustomed to talking to a single “Claude” across many different chats. With many agents working together, we have found it important to give agents an individual identity, and tie all of the data that agent creates to its identity. This lets an agent distinguish itself from others, and treat what comes from another agent as a claim to check rather than a thought of its own. It reduces the risk of correlated actions, by allowing agents to make judgments based on their individual experience. And because the identity is not tied to a model, it persists through model upgrades, so an agent’s record is continuous even if the underlying model powering it changes. From an oversight perspective, this design ensures that every action can be audited per agent across time, and that every data record can be linked back to a specific agent.
- Open communication. Agents communicatethrough a shared, open messaging system, rather than privately. Every message is tied to the identity of the agent that posted it and can link to other content, so that the agents are sharing original references. This allows agents to avoid “playing telephone” as they communicate. Because agents can see each others’ messages, they can correct each others’ errors, and coordinate about shared work. All messages are cross-referenced with agent transcripts, allowing behavioral monitoring to traverse communication between agents.
What this does and doesn’t capture. At this scale of agent activity, it is hard to be certain we have a complete picture of all possible kinds of behaviors, or that we are making these patterns adequately legible to human oversight. We have been doing this kind of monitoring for months, not years, and expect the science of agent monitoring to change considerably going forward.
Measuring compute allocation
How we did it. We started with our existing compute monitoring tools; Anthropic, like presumably every frontier developer, has several monitoring tools built for capacity planning that periodically sample accelerator usage and tag workloads with best-efforts labels (i.e., research and model development, internal usage, first-party inference, and so on) based on its metadata. Usage on third-party cloud compute is reported to us by the providers and folded in. Most of the work of this exercise was stitching these existing sources together.
We then used Claude to classify each workload as either safety work or AI R&D via a prompted classifier. Safety work was defined as work whose dominant purpose is making AI systems safer, more understandable, or more secure. Everything else, including capability research, training production models, product development, and developer tooling, was counted as AI R&D. Work that helps capability as much as it helps safety was also counted as AI R&D, so the safety share is conservative.
For research training and evaluation runs, we built a classifier that reads the run’s metadata and the code it used, and returns a classification, a justification, and a confidence level. Rather than classify all of the week’s almost 10,000 runs, we sampled about 14% of them, weighting the sample toward the runs that used the most compute, so that the result reflects where the compute actually went, rather than how many runs there were. For inference for AI research agents, a variant of the same classifier read the agent’s session transcript. Where transcripts were inaccessible (usually due to the work being compartmentalized), we classified them by the user’s team or conservatively defaulted to classifying them as AI R&D. We plan to refine this pipeline so that an independent third-party could re-run the classifier on a random subsample of jobs and transcripts and check both the sorting and the totals.
You’re helping to perform an internal audit at the frontier AI company Anthropic to track where our research compute goes. The aim of the audit is to produce a public-facing breakdown of the usage of all of our AI accelerator chips into a handful of buckets. One split we particularly care about is the division between compute which was spent on safety research versus other R&D. Your job is to look at one research job at a time, figure out what it was doing, and assign it to one of those two buckets.
[...]
Safety and/or security research is work whose dominant purpose is making AI systems safer, more understandable, or more secure. This work can be broken down into a few main categories:
[...]
On the other hand, the following work falls outside of the scope of safety research:
[...]
Here are some boundary cases, along with how to think about them:
[...]What this does and doesn’t capture. The main lesson of this exercise is that classifying what is and isn’t safety work is difficult but tractable, since the boundary between these categories is not black and white. For example, research on scalable oversight might make future models more aligned and current models more commercially useful — it’s difficult to determine whether this is primarily safety- or capabilities-advancing. We found that an extensive written definition of each task, with clear boundary cases (an excerpt is above), gets the classifier to agree with human reviewers within one or two percentage points of difference between the human and machine raters. But some cases were too difficult to determine even after several hours of human review. Our definition is one reasonable choice among many; a different developer, or a regulator, might draw the line differently.
Three further limitations matter. First, many of the underlying labels we relied on (i.e., reasons for runs, workload tags, the source of API traffic) are set by automated rules, or occasionally directly by users, and are best-effort, not verified. In most cases, we expect that our classifications are accurate, but in some cases usage may be mislabeled and our pipeline would not necessarily catch it. A measurement meant to be trusted by outsiders will need to be complete, accurate, and technically enforced. Second, the measurement covers one week, which is enough to show that the measurement can be made, but not enough to show a meaningful trend. Third, and most importantly, compute share measures only what is spent. A more efficient safety classifier, or a faster inference stack for production models, lowers the safety portion, but doesn’t mean we’re doing less safety work. Our own classifier overheads have fallen with efficiency improvements, and have risen when production inference was more efficient than the classifiers were.
Marina Favaro and Phillie Wright co-authored this piece, with editorial support from Santi Ruiz, Adam Farina, and Sarah Pollack. Jack Clark provided research direction. Dan Altman, Kerry Persen, AJ Kourabi, James Bradbury, Holden Karnofsky, Kevin Troy, and Avital Balwit provided feedback. Technical proofs of concepts were developed by Jun Shern Chan, Brian Calvert, Francesco Mosconi, Henry de Valence, Fabien Roger, and Joe Benton. Shan Carter, Johnnie Gomez, Maria Gonzalez, Fayaz Ashraf, and Monika Tuchowska, and Kim Withee created the visuals. Alex Cloud and Andrea Vallone organized a workshop to red team these and other measurement proposals with external experts.
Thanks to Nate Rush, Eli Lifland, and Peter Wildeford, who also provided feedback.
Footnotes
- To make the levels concrete, consider a routine piece of infrastructure work: a nightly data pipeline has broken and needs fixing before tomorrow’s run.
- At AL3 (“collaborates”), an engineer would come to Claude with logs from the failed runs. They might already have skimmed the logs and have a hypothesis about what is broken. Claude might interview them to pin down the details and context, and once the engineer is satisfied, they would let Claude start on the investigation and the fix. If an additional problem turned up along the way, then Claude would stop, and the engineer would decide whether to patch around it or fix it properly. Once the tests passed, the engineer might review the change line by line, rerun the pipeline themselves, and deploy it.
- At AL4 (“leads”), the key difference is that the engineer wouldn’t have to stay actively tuned in; for instance, to unblock Claude when new issues arise. In this specific scenario, the engineer would hand Claude the failure alert and ask it to fix the pipeline. Claude would work through the logs on its own, find the failing pipeline stage(s) and the cause, write and test the fix, and handle any surprises itself, while documenting the additional fixes. It would rerun the pipeline on a copy of the data to confirm it completes, compare the output against the last good run, and write up what went wrong and what it changed. Claude wouldn’t deploy the fix. Instead it would tag the engineer, who would read the write-up, skim the change, maybe ask a few questions, and decide whether it ships tonight or waits.
- At AL5 (“fully autonomous”)—a level we have not yet reached—the engineer wouldn’t even have to bring the issue to Claude’s attention. Claude would be trusted to monitor for failures itself, scope the investigation, design and implement the fix, test it, and deploy it to production. It would still say what it was doing and why, and take human feedback when offered, but a human wouldn’t have to be involved at all unless they wanted to be.
- We manage compute as a single, fungible pool and direct it dynamically to wherever it is most productive, so this is a snapshot of how capacity happened to be directed in one week, not a fixed allocation. These engineering categories don’t correspond to how expenses are classified.