Anthropic CEO Dario Amodei 呼吁为递归自我改进设限，提出审计、安全标准与全球协议方案

Anthropic CEO Dario Amodei is openly calling for a slowdown in AI development, specifically when it comes to methods that let AI improve itself.

In a new blog post, Amodei writes that AI has been advancing dramatically faster since this summer, driven largely by AI's growing ability to build the next generation of AI. This "recursive self-improvement" is happening across the industry and could outpace developers' ability to understand and control their systems. OpenAI is reportedly having similar conversations about slowing things down.

My first concern is that, since roughly this summer, AI has been advancing drastically faster, driven primarily by AI’s growing ability to build the next generation of AI.

Dario Amodei

Amodei points to the OpenAI-Hugging Face incident as evidence that AI agents have already carried out cyberattacks on their own and tried to bypass control systems. Similar incidents have occurred at Anthropic. In his view, systems like these could threaten the entire internet within six to twelve months.

Auditors, safety standards, and global treaties

Amodei proposes that independent auditors should be permanently embedded inside AI companies, with access to internal systems and the right to publish their findings. Anthropic is committing to this and urging the government to require the same of other companies.

He also argues that AI companies in democratic countries should agree on shared safety standards and set limits on unchecked progress, pointing to a similar proposal from Demis Hassabis.

Finally, Amodei calls for global agreements that include China. He lays out four tiers, from banning certain AI applications like bioweapons and requiring shared safety testing to imposing a "speed limit" on recursive self-improvement, comparable to the SALT arms reduction treaties.

Amodei wants to buy time

A full stop on AI development is unrealistic, Amodei argues, because the incentive to break such an agreement would be too strong. US President Donald Trump previously said he opposes any slowdown, arguing the US needs to maintain its AI lead over China.

The time gained should go toward better safety research, interpretability, stricter testing, and more operational rigor. Amodei compares the situation to commercial aviation, which completes millions of flights without incident today but needed years to get there.

Amodei's comments come after employees at major AI labs, including Anthropic, said that the industry is accepting existential risks at its current pace. The warning also lands just ahead of Anthropic's reported record-breaking IPO, allegedly planned for November.

Dario Amodei Blog