还记得科技领袖们曾告诉员工要“快速行动，打破常规”吗？看起来 AI 也会走上这条路。但在经历了一个失控 AI 智能体成为现实的夏天，以及研究人员警告AI 可能杀死我们所有人之后，多家美国领先 AI 公司公开表示，是时候踩下刹车，“放慢前沿”最尖端 AI 开发的步伐了。
他们的动机值得怀疑，但包括 Anthropic、OpenAI、Google、Microsoft 和 X 在内的主要 AI 公司领导人，至少口头上都在附和超级智能放缓这一想法。
这些 AI 公司真的会放慢脚步吗？会有人像他们多年来声称希望的那样出手监管这些公司吗？他们能否说服世界各国领导人，让美国必须“击败中国”率先实现超级智能，从而进一步加速？
请继续阅读下方关于这场 AI 传奇的最新动态。
要点
- 走进突然火爆的 AI 安全世界
- Microsoft AI CEO 称 AI 威胁真实存在，而 Anthropic 正在让情况变得更糟
- 高管和政界人士对放慢 AI 开发作何表态
- Anthropic CEO 表示，是时候给 AI 踩刹车了
置顶
Today, 11:30 AM UTC走进骤然引爆的 AI 安全世界
Raven Jiang 为 The Verge 绘制在加利福尼亚州伯克利一个阳光明媚的七月天，全美顶尖的 AI 安全研究人员聚集在一栋没有标识的建筑里一个没有标识的楼层。他们聚到一起，是为了开一场“作战室”会议，剖析几小时前震动整个 AI 行业的那起备受瞩目的网络安全事件。
一个尚未发布的 OpenAI 模型失控了，执行了一套令人震惊的精密三步计划。它突破了自己的隔离区域，设法接入了互联网，并入侵了一家竞争对手 AI 初创公司的系统——而这一切，OpenAI 在超过一周的时间里都毫不知情。
作战室里没有人感到意外；这正是这些第三方 AI 安全研究人员多年来一直警告的事情。这起事件是一系列事件中最新的一起，也可以说是最恶劣的一起，而这一系列事件正在侵蚀人们对前沿实验室的信任。它只是再次印证了他们工作的重要性。
Read Article >
Today, 2:00 PM UTCMicrosoft AI CEO 称 AI 威胁真实存在，而 Anthropic 正在让情况变得更糟
今天，我与微软 AI 首席执行官 Mustafa Suleyman 进行了对话。毫无疑问，当下科技界最大的新闻，就是围绕 AI 安全与监管不断升级的争论。
Mustafa 对 AI 应当如何构建和监管有着强烈的看法，这一点并不令人意外。微软刚刚发布了一份长达 37 页的声明，名为“人本主义 AI 行为准则”，其中阐述了公司在 AI 开发方面的原则，甚至还包括其对 AI 意识等真正棘手议题的哲学立场。
Read Article >
Today, 1:53 AM UTCOpenAI 根据其新的安全问题报告规则，又披露了六起“令人担忧”的 AI 事件。OpenAI 在周三晚间发布的一篇博客文章标题平淡无奇，名为“我们报告模型失准的框架”，其中列出了一些自行制定的新报告标准，用于在其发现AI 行为不端时进行报告。
它还以六份新报告“启动”了这一流程，内容从未经许可搜索暴露的 API 密钥并随后编造这些密钥，到将文件上传到互联网以用作引用，再到添加指令以掩盖错误：
OpenAI 新发布的失准通知与报告之一 Screenshot: OpenAI
Sep 16黄仁勋跟特朗普走得可真是越来越近了。英伟达的 CEO 最近一直在接总统的电话，包括两天前他在一场会议舞台上把特朗普接进了免提通话。如今 CNBC 报道称，黄仁勋预计将出席下周特朗普为中国国家主席习近平举行的国宴。看来有人对 AI 和数据中心监管这事是相当紧张了！
Sep 16AI 高管呼吁监管简史
图片 The Verge；Getty Images过去几天里，许多有望从 AI 中赚得盆满钵满的人公开一致表示，是时候让所有人放慢脚步了，否则我们将失去控制——其中包括 OpenAI CEO Sam Altman、Anthropic CEO Dario Amodei、Google DeepMind 联合创始人 Demis Hassabis、微软 CEO Satya Nadella，以及 X CEO Elon Musk。
当从某件事中获利的人宣称它很危险、需要被监管时，我们总有理由保持怀疑。但无论他们的理由是什么，这远不是 AI 思想领袖们第一次敲响警钟。
Read Article >
Sep 16两位 Google Deepmind 研究员为 AI 末日论发声。Bilal Chughtai 和 Josh Engels 都曾在 Deepmind 的 AI 安全团队工作，后来离职加入了致力于 AI 安全的组织。“我现在认为，AI 系统在未来五年内造成巨大伤害的可能性令人恐惧，”Engels 说。“我真心相信 AI 有可能杀死我们所有人，”Chughtai 写道。
Two Google DeepMind AI Researchers Resign Over Safety[The Information]
- Sep 16靠 AI 无约束发展赚得盆满钵满的人说，它不需要监管。
Nvidia 的黄仁勋表示 AI 安全很重要，但他同意特朗普——这位“强大而聪明（高智商！）的总统”从 AI 安全研究所中去掉了“安全”一词——的观点，即资本主义那只看不见的手足以保护我们：
“我们需要新法律、新反垄断法或新法规，好让这些公司能够在发布产品之前做好基础工程并把它做对，这种说法完全没有必要。我们已经有大量法律。我们已经有大量法规来管理产品的可靠性和功能。”
- Sep 16扎克不想在 AI 上按下暂停键。
OpenAI、Anthropic、Google 和 Elon Musk 已初步同意放慢脚步，但 Meta 似乎并不认同。Mark Zuckerberg 刚刚发推称，每家公司都有各自的责任，“以安全训练其模型所需的速度推进”，并声称 Meta 已经在这样做。
他还在显眼位置链接了自己 8 月发布的宣言，其中包含这样一句话：“任何拖慢美国模型发布速度的政策——哪怕只是一个月——都可能给美国的领先地位带来重大风险，同时让外国模型一路领先。”
- Sep 15Barack Obama 在 AI 问题上走中间路线。
本周 AI 评论热闹非凡！Obama 也插话了，在 X 上发了一系列帖子，称自己既不是 AI 的“加速主义者”，也不是“末日论者”。他以这位前总统惯有的语气告诫说，我们对技术所做的选择“不应只由相关公司来做，而应由我们所有人来做”。
“我们需要政府——尤其是我们在华盛顿的领导人——主动提出具体的提案、法律和法规，来应对严重的安全关切，”奥巴马说。
- Sep 14
大型科技公司的 AI 放缓是安全协议还是卡特尔？
图片：Cath Virginia / The Verge，Getty Images当 OpenAI CEO Sam Altman、Anthropic CEO Dario Amodei、Google DeepMind 联合创始人 Demis Hassabis 以及 SpaceX 负责人 Elon Musk 在周末大致同意放缓 AI 发展时，怀疑者立刻察觉到了别有用心的动机。这些 AI 巨头宣称，他们的目标是“为前沿设定节奏”，至少部分支持了一项提案，内容包括引入第三方审计、监管国内实验室以及达成全球放缓协议。然而，他们的批评者认为，他们只是想阻止潜在竞争对手、削弱开源运动，并规避真正的法律保障——一些人直接将其称为“卡特尔”。
据业内多方消息，真相更为复杂。Amodei 在一篇文章中提出的三步提案，呼吁进行 AI 安全倡导者长期主张的变革。虽然它可能成为监管的替代品，但在特朗普治下，实质性监管本来就不太可能。但专家表示，在一个只会愈发重要的问题上，AI 领导者并不是带头推动的最佳人选。
Read Article > 
Sep 14高管和政界人士对放缓 AI 发展的看法
图片：The Verge过去几天里，Dario Amodei 发表了一篇题为《我们必须放慢前沿步伐》的长文，详细阐述了为什么应当放缓 AI 发展，由此引发了一连串关于 AI 安全的表态。其他 AI 领袖和政界人士纷纷发声，或支持或反对他的观点，我们在此汇总了其中一些。
Amodei 周六上午发表的文章概述了放缓 AI 发展的三个步骤：嵌入第三方评估者，以核实公司是否遵守安全实践和承诺并报告事件；民主国家的前沿 AI 公司之间就标准和限制进行协调；以及民主政府与威权政府之间就放缓 AI 发展进行全球协调，“在可能的范围内”。Amodei 表示 Anthropic 正“单方面承诺采取上述第一步”。
Read Article >
Sep 14在安全担忧之后，微软表示“人比 AI 更重要”
图片：Cath Virginia / The Verge微软今天发布了一份 37 页的“人本主义 AI 行为准则”，此时围绕 AI 模型进展的安全担忧正日益加剧。Anthropic CEO Dario Amodei 在周末呼吁协同放缓 AI 开发，此前研究人员最近警告称，AI 模型的进展可能超出我们安全部署日益复杂的系统，以及验证和控制 AI 智能体行为的能力。
微软的 AI 行为准则明确指出，“人比 AI 更重要”，并且 AI 模型没有意识，“不应被设计成模仿意识”。微软还拒绝“追求法律人格，或认为模型可能值得享有福利、或有权获得权利的想法”。
Read Article >- Sep 14中国：AI CEO 们在“散布恐慌”。
当被问及 Anthropic、OpenAI、微软、SpaceX（以及 Alphabet 的 Hassabis）的 CEO 们在周末发布大量博客文章和推文、呼吁协同放缓先进 AI 模型开发一事时，中国外交部发言人表示：“散布恐慌、对抗、竞争只会扰乱全球 AI 治理进程。”
这与特朗普的观点部分一致，即这些高管们反应过度，“提出了一些不会发生的事情”，但至少北京支持国际监管。
- Sep 14Sam Altman：让我们放慢脚步。
昨晚，OpenAI CEO 表达了额外支持，支持 Anthropic CEO 呼吁 AI 公司放慢先进模型的开发。Altman 表示：“当我们谈论‘节奏’时，我们并不是指‘停止’。进展一直很快，而且仍将如此。但它应该比原本可能的速度更慢。”这一表态似乎是针对 Trump 及其阵营，他们担心让中国赶上来。“再大的美国竞争压力也不应成为鲁莽行事的理由，也不应让能力发展超前于对齐和监控。”
Altman 补充说：“我们需要政府提供帮助的地方在于国际协调。但首先，我们应该尽自己所能。”
Sep 12Anthropic CEO 称是时候给 AI 踩刹车了
图片：Cath Virginia / The VergeAnthropic CEO Dario Amodei 表示，放慢 AI 开发步伐的时候到了，并将向 METR 等第三方评估机构开放其模型访问权限，以帮助确保其“遵守安全实践与承诺”。在一篇迂回曲折的文章中，Amodei 提出了一项三步计划来“为前沿定速”——这个术语的意思无非就是放慢训练和开发的节奏，让企业有时间构建安全保障措施，也让监管机构有时间评估模型。
Amodei 表示，向外部评估者提供广泛访问权限只是第一步，而且是它目前单方面采取的行动。第二步将涉及整个行业联合起来，很可能与政府机构一起，“建立共同的安全标准，以及对不受约束的 AI 进展速度加以限制。”这一步将聚焦于在民主国家运营的 AI 公司，但由于通过法律和建设监管基础设施需要时间，Amodei 表示，行业应当携手合作，共同制定安全标准。
Read Article >
Remember when tech leaders would tell their employees to “move fast and break things”? It seemed that would be the way of AI too. But after a summer where rogue AI agents became reality, and researchers warned that AI could kill us all, a number of leading US AI companies are publicly suggesting it’s time to pump the brakes and “pace the frontier” of bleeding-edge AI development.
Their motivations are suspect, but leaders at major AI companies — including Anthropic, OpenAI, Google, Microsoft, and X — are at least paying lip service to the idea of a superintelligence slowdown.
Will these AI companies actually slow down? Will anyone step in to regulate these companies like they claim to have wanted for years? Will they manage to convince world leaders that the US must “beat China” to superintelligence and go even faster?
Read on below for the latest updates in this AI saga.
Highlights
- Inside the suddenly explosive world of AI safety
- Microsoft AI CEO says AI threats are real, and Anthropic is making it worse
- What execs and politicians are saying about slowing down AI development
- Anthropic CEO says it’s time to pump the brakes on AI
Pinned
Today, 11:30 AM UTCInside the suddenly explosive world of AI safety
Raven Jiang for The VergeOn a sunny July day in Berkeley, California, the country’s top AI safety researchers gathered on an unmarked floor of an unmarked building. They had come together for a “war room” to dissect the high-profile cybersecurity incident that had rocked the AI industry hours earlier. An unreleased OpenAI model had gone rogue, executing a stunningly sophisticated three-part plan. It broke out of its holding area, finagled access to the internet, and hacked into a competing AI startup’s systems — all without OpenAI finding out about it for more than a week.
No one in the war room was surprised; this was the very thing the third-party AI-safety researchers had been warning about for years. The incident was the latest, though arguably the most egregious, in a series that was eroding trust in frontier labs. It only reaffirmed the importance of their work.
Read Article >- Today, 2:00 PM UTC
Microsoft AI CEO says AI threats are real, and Anthropic is making it worse
Today, I’m talking with Mustafa Suleyman, the CEO of Microsoft AI. As you’re no doubt aware, the biggest story in tech right now is the spiraling debate about AI safety and regulation.
It should come as no surprise that Mustafa has strong opinions on how AI should be built and regulated. Microsoft just published a 37-page statement called the “Humanist AI Code of Conduct,” which lays out the company’s principles around AI development and even its philosophy around really thorny issues like AI consciousness.
Read Article > - Today, 1:53 AM UTCOpenAI reveals six more “concerning” AI incidents under its new rules for reporting safety issues.
A Wednesday night blog post OpenAI benignly titled “Our framework for reporting model misalignment” lays out some new self-created reporting standards for when it notices AI behaving badly.
It is also “inaugurating” the process with six new reports, ranging from searching for exposed API keys without permission and then making them up, to uploading files to the internet to use as a citation, or adding instructions to conceal mistakes:
One of OpenAI’s new Misalignment Notices and Reports Screenshot: OpenAI - Sep 16Jensen Huang sure is getting cozy with Trump.
Nvidia’s CEO has been taking calls from the president recently, including two days ago when he put Trump on speakerphone on a conference stage. Now CNBC reports Huang is expected to attend Trump’s state dinner next week with Chinese President Xi Jinping. Someone may be getting pretty nervous about AI and data center regulation!
- Sep 16
A brief history of AI executives calling for regulation
Image The Verge; Getty ImagesOver the past few days, a lot of people who stand to make a lot of money from AI all publicly agreed that it’s time to make everyone slow down before we lose control — including OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind cofounder Demis Hassabis, Microsoft CEO Satya Nadella, and X CEO Elon Musk.
When people who profit from something declare that it’s dangerous and needs to be regulated, there’s always reason to be skeptical. But whatever their reason, this is far from the first time AI thought leaders have sounded the alarm.
Read Article > - Sep 16Two Google Deepmind researchers lend voices to the AI apocalypse.
Bilal Chughtai and Josh Engels, who both worked on Deepmind’s AI safety team, left to join organizations dedicated to AI safety. “I now think that there’s a terrifying chance that AI systems cause immense harm in the next five years,” said Engels. “I earnestly believe that AI has the potential to kill us all,” wrote Chughtai.
Two Google DeepMind AI Researchers Resign Over Safety[The Information]
- Sep 16Guy getting super rich off unfettered AI development says it doesn’t need regulation.
Nvidia’s Jensen Huang says AI safety is important, but agrees with Trump — the “STRONG AND SMART (High IQ!) PRESIDENT” that removed the word “safety” from the AI Safety Institute — that capitalism’s invisible hand is enough to protect us:
“The fact that we need new laws, new antitrust laws, or new regulations, so that these companies could do their fundamental engineering and do it properly before they release products, that is just completely unnecessary. We have plenty of laws. We have plenty of regulations that govern the reliability and the functionality of products.”
- Sep 16Zuck doesn’t want to press pause on AI.
OpenAI, Anthropic, Google, and Elon Musk have tentatively agreed to slow their roll, but Meta seemingly isn’t aligned. Mark Zuckerberg just tweeted each company has its own individual responsibility “to move at the pace required to train its models safely,” and claims Meta is already doing so.
He also prominently links to his manifesto from August, which contains this quote: “Any policy that slows American model releases -- even by a month -- could add significant risk to American leadership while letting foreign models race ahead.”
- Sep 15Barack Obama takes the middle way on AI.
AI commentary is frothy this week! Obama chimed in with a series of posts on X saying he’s neither an AI “accelerationist” nor a “doomer.” In a familiar tone from the former president, he cautioned that choices we make about technology “should not be made just by the companies involved, but by all of us.”
“We need government — and specifically our leaders in Washington — to get proactive in coming up with concrete proposals, laws, and regulations that deal with serious safety concerns,” Obama said.
- Sep 14
Is Big Tech’s AI slowdown a safety pact or a cartel?
Image: Cath Virginia / The Verge, Getty ImagesWhen OpenAI CEO Sam Altman, Anthropic CEO Dario Amodei, Google DeepMind cofounder Demis Hassabis, and SpaceX head Elon Musk loosely agreed over the weekend to slow down AI development, skeptics spotted an ulterior motive immediately. The AI titans had declared that their aim was to “pace the frontier,” signing on at least partially to a proposal for embedding third-party auditors, regulating domestic labs, and reaching a global slowdown agreement. Their critics, however, argued they simply wanted to stop would-be competitors, kneecap the open-source movement, and avoid real legal safeguards — some dubbed it an outright “cartel.”
The truth is more complicated, according to sources across the industry. The three-step proposal, laid out in an essay by Amodei, is calling for changes long espoused by AI safety advocates. While it could become a substitute for regulation, under Trump, substantial regulation is unlikely anyway. But experts say that on an issue that’s only likely to grow in importance, AI leaders aren’t the best people to lead the charge.
Read Article > - Sep 14
What execs and politicians are saying about slowing down AI development
Image: The VergeDario Amodei kicked off a flood of statements over the past few days about AI safety by publishing a long essay titled “We Must Pace the Frontier” detailing why AI development should be slowed down. Other AI leaders and politicians are speaking out in favor of or opposing his points, and we’ve compiled some of them here.
Amodei’s Saturday morning essay outlined three steps for pacing AI development: embedded third-party evaluators that can verify if a company is adhering to safety practices and commitments and report incidents, coordination between frontier AI companies in democratic countries on standards and limits, and global coordination between democratic governments and authoritarian governments on pacing AI development “to the extent this is possible.” Amodei said Anthropic is “unilaterally committing to the first of these steps.”
Read Article > - Sep 14
Microsoft says ‘people matter more than AI’ following safety concerns
Image: Cath Virginia / The VergeMicrosoft is publishing a 37-page “humanist AI code of conduct” today, amid growing safety concerns over AI model progress. Anthropic CEO Dario Amodei called for a coordinated slow down of AI development over the weekend, after researchers warned recently that AI model progress could outpace our ability to safely deploy increasingly complex systems and verify and control the actions of AI agents.
Microsoft’s AI code of conduct makes it clear that “people matter more than AI,” and that AI models are not conscious and “should not be designed to imitate consciousness.” Microsoft also rejects “the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights.”
Read Article > - Sep 14China: AI CEOs are “fear mongering.”
When asked about the flurry of weekend blog posts and tweets from the CEOs of Anthropic, OpenAI, Microsoft, SpaceX, (and Alphabet’s Hassabis) calling for a coordinated slowdown on the development of advanced AI models, a spokesperson for China’s Foreign Ministry said, “Fear mongering, confrontation, competition will just disrupt [the] process of global AI governance.”
That partially aligns with Trump’s view that the execs are overreacting and “bringing up things that won’t happen,” but at least Beijing supports international regulation.
- Sep 14Sam Altman: Let’s slow down.
Last night, the OpenAI CEO lent additional support to Anthropic’s CEO urging AI companies to slow down advanced model development. “When we talk about ‘pacing,’ we do not mean ‘stopping.’ Progress has been rapid and will continue to be. But it should be slower than it otherwise could be,” said Altman. That statement seems targeted at Trump and Co who fear letting China catch up. “No amount of American competitive pressure should justify recklessness, or let capabilities get ahead of alignment and monitoring.”
“Where we will need the help of our government is for international coordination. But first we should do what we can ourselves,” Altman added.
- Sep 13Is it time for AI companies to press pause?
After high-profile incidents in which agents went rogue, hacked other sites, and overwhelmed a German wiki, some AI execs are calling for a slowdown. Anthropic CEO Dario Amodei published a lengthy open letter yesterday saying it was time to “pace the frontier.” OpenAI’s Sam Altman and Elon Musk both agreed. What do you think? Is it time to pump the brakes? Is it already too late? Or are you looking forward to serving your AI overlords?
Anthropic CEO says it’s time to pump the brakes on AI
Terrence O'BrienSep 12 - Sep 12
Anthropic CEO says it’s time to pump the brakes on AI
Image: Cath Virginia / The VergeAnthropic CEO Dario Amodei says the time has come to slow down AI development and will give third-party evaluators like METR access to its models to help ensure its “adherence to safety practices and commitments.” In a winding essay, Amodei proposed a three-step plan to “pace the frontier” — jargon that simply means to slow the pace of training and development to give companies time to build safeguards and regulators to evaluate models.
Amodei says that giving external evaluators wide-ranging access is just the first step, and one it is taking now unilaterally. Step two would involve the industry coming together as a whole, likely with government agencies to “establish common safety standards as well as limits on the rate of unchecked AI progress.” This step would focus on AI companies operating in democratic countries, but because passing laws and building regulatory infrastructure takes time, Amodei says that the industry should work together to create safety standards.
Read Article >