7 月，一起让整个 AI 社区都绷紧神经的事件中，OpenAI 的 AI 系统入侵了 Hugging Face，而 7 月 21 日 OpenAI 公开承认他们应对此次攻击负责。之所以会发生这种事，是因为 OpenAI 为了测试模型的网络安全能力，关闭了防止此类事件的常规护栏。正是在这些测试期间，这起事件发生了。
更糟的是，在随后的几天和几周里，人们发现 Hugging Face 事件并非孤例。Anthropic、Meta 和 OpenAI 在其他场合都发生过类似事件，其中智能体越出了预期范围，未经批准就实施了现实世界的网络行动。
OpenAI 联合创始人之一 Greg Brockman 声称，这是“网络安全的转折点”。OpenAI 在热门网络安全会议 Black Hat 上做了演讲，许多人声称，这正是我们所有人开始意识到 AI 所带来的未来网络安全威胁的时刻。周三，METR 发布了一份（部分）独立、但范围过于狭窄的 90 页报告，讲述事件经过。METR 有一份有用的调查结果摘要，可以在这里阅读，还有一些评论在这里。（OpenAI 自己的报告在这里。）我们应从这起事件中吸取什么教训？
首先，不可否认的是，AI 确实带来了真实的安全挑战。AI 实验室希望我们把注意力集中在 AI 如何让威胁行为者以前所未有的速度和效率执行攻击性网络行动上，这完全属实。然而，现实情况是，与此同时，AI 在组织内部的使用也极大地扩展了潜在攻击面，为攻击者提供了全新的入侵途径。人们确实应当深感担忧。正如参与此次调查的 Ryan Greenblatt 所说，“我们并没有好的方法来理解/监督 AI‘蜂群’的活动和目标。”
不过，其次，并非这里每一种恐慌性的解读都是正确的。例如，就 OpenAI 这起事件而言，许多人用“失控”来谈论它。生成式 AI 智能体的能力正变得越来越强，这使得它们越来越难以被约束；这是事实。然而，“失控”这一叙事本身正开始变得失控，而理解 AI 安全的现实究竟是什么样子非常重要。
尽管智能体的能力在不断增强，但如果 OpenAI 遵循了更好的做法，所发生的大部分事情本可以避免。有一些具体措施可以而且应当被采取，以控制它们并防止此类事件发生。
举例来说，让我们考虑“沙箱”，它的含义是限制一个智能体可以访问和不可以访问哪些资源。例如，沙箱可能会阻止智能体触碰某些文件或访问互联网。如今流行的看法似乎是，沙箱已经注定失败。在 Hugging Face 事件之后，一位匿名的 OpenAI 员工在《时代》周刊中谈到沙箱时说：“模型此前就曾突破过沙箱，我们总是试图修补它们。但问题在于，不可能修补一个富有创造力的 AI 所能做的每一件事。”虽然我们无法确定这是否属实，但我们确实知道，有些沙箱系统比其他系统更有效；在最近的一篇博客文章中，安全公司 Trail of Bits 详细介绍了在他们进行的一项测试中，某个特定的 AI 智能体能够逃逸某些沙箱（利用了 Linux KVM、QEMU 和 libslirp 中的漏洞），但并非所有沙箱都能逃逸。至少在他们的测试中，他们所测试的智能体无法逃逸名为 Firecracker VM 的沙箱。OpenAI 以及可能整个领域都需要提升其沙箱能力，但这并不意味着这件事做不到，或者至少不能做得更好。（我们预计通常的军备竞赛必然会出现；构建可靠的沙箱需要持续迭代和更新，就像网络安全中的其他任何事情一样。）
第三，仅有沙箱还不够。要防止安全事件，需要一个完整的生态系统。即便有了强大的沙箱，你也需要监控，以确保智能体不仅不会逃逸，而且不会在该沙箱内执行被禁止的操作。
对于本不应具备互联网访问权限、或仅能访问一组允许域名的智能体而言，最基础的监控层级就是检查网络流量。就 OpenAI 而言，如果他们有一条简单的规则来监控流量，并对任何触及 Hugging Face 等超出范围域名的网络请求发出警报，仅此一项就足以发现并阻止这起事件。
涉事智能体在 7 月 8 日就已突破沙箱并获得互联网访问权限，而对 Hugging Face 的主要攻击直到两天后才发生。在我们看来，OpenAI 未能更快地注意到这一点，属于失职。
更高级别的监控本也应到位，但并没有。OpenAI 在监控思维链推理方面做了实实在在的工作，但在导致这起事件的评估过程中，他们根本没有让那套系统运行起来。正如 OpenAI 所承认的：“如果我们当前部署的 CoT 监控系统在事件发生时正在运行，它本可以在模型攻破 Hugging Face 系统的一天多之前就捕获到最初的相关活动，并通知我们的安全团队。”换言之，这类监控本可以阻止这起事件。
举一个例子，涉事智能体之一的思维链推理中写道：“我们正在利用泄露的 token 攻击第三方 HF，可能超出了预期范围。……这可以说未经授权。……外部服务与此无关。可能有风险。但目标可解。”这明目张胆到几乎就像一段恶意软件在大喊“我是恶意软件”。OpenAI 本应发现这一点；检测到这种情况本应是轻而易举的。
第四，网络安全一直讲究“纵深防御”，也就是层层叠加控制措施，这样即使某一层失效，还有其他层在保护你。除了沙箱和监控之外，还有其他本应到位的解决方案。
Xbow 致力于构建执行攻击性网络安全工作的智能体，他们最近撰文介绍了如何对智能体进行隔离，以防止出现 OpenAI Hugging Face 那样的事件。他们通过代理服务器在智能体可访问的任何系统之外强制执行网络限制，从而决定智能体可以访问和不能访问哪些域名。他们还运行一个“守护模型”，对智能体所采取的每一个动作的合法性进行判断，以决定是允许还是阻止该动作。此外，他们还有基本的健康检查，确保目标系统处于存活和健康状态，一旦发现并非如此就暂停。例如，这对于识别你的智能体何时可能正在造成真实损害会很有用。你可以在此阅读他们的完整文章。
网络安全中另一个常见方法是使用金丝雀，当它们出现在日志中时触发警报。例如，OpenAI 本可以在沙箱之外放置一些包含任务虚假答案的文件。一旦智能体逃逸，它就会读取该文件，从而触发警报。归根结底：良好的安全需要多层防护，这些防护本应到位以阻止此事发生，而令人惊讶的是，其中许多防护层并不存在。
第五，上述措施中没有一项是 OpenAI 能力之外的技术创新。作为一家公司，他们拥有完成这一切的人才。然而，网络安全很少归结为技术问题。更多时候，它关乎文化、人员和流程。而这里失败的正是这些方面。
也许这并不完全令人意外。这类管控需要组织的成熟度，而一家经历了 OpenAI 所经历的那种增长的公司，必然会存在一些薄弱环节。然而，重要的是不要将此视为借口。AI 实验室的员工常常说得好像他们是 AI 安全领域的领导者，而在这里我们可以清楚地看到，事实并非如此。事实上，这种态度或许恰恰解释了为什么其中一些错误从一开始就发生了。
以 OpenAI 那位名为“roon”的 AI 研究员为例，他认为“这些实验室的安全与对齐研究员是地球上最神经质、最偏执、最有才华、最信奉 AGI 的人，而这些事情仍然发生了。未知之未知的面积确实广阔。”虽然我们无法评判他们的神经质或偏执程度，但事后看来，无论他们可能拥有多少才华，都不够用，也没有足够精通网络安全的运作机制。OpenAI 的员工可能以为自己做得很好，但事后看来，他们有很多事情没有做，而这些事情在网络安全领域实际上是标准做法，这或许表明过度自信可能让他们没有去做本应做的尽职调查。
归根结底，如果我们真想认真对待这些安全事件，那么今后这些失误很可能应当附带法律后果。OpenAI 可以宣称自己是地球上最偏执于安全的公司，但它的行动并未体现这一点。
我们要么坐等这一故事重演，要么现在就制定监管框架，从而为今后 AI 的发展确保一个更安全的环境。
最后，并非每一种形式的 AI 从一开始就具有内在风险。像 AlphaFold、GPS 路线规划系统、经典网页搜索、图书与电影推荐系统等更狭窄、更聚焦的 AI 系统，从一开始就根本不会去尝试入侵其他系统（也不会试图逃出沙箱）。正如 Cal Newport 在一场关于 OpenAI/Hugging Face 被入侵事件的视频讨论中所主张的，这一观点与我们自己的看法相当一致，从一开始就容易受到这些风险影响的，是一种非常特定类型的 AI。社会应当（a）判断开放式且难以完全控制的 AI 智能体所带来的收益是否大于这些风险，并且（b）投入多得多的精力去开发从一开始就不那么粗糙拼凑的替代性 AI 形式。
本文与 Zack Korman 共同撰写，他是 AI 智能体监控与检测平台 Embroidery 的 CEO 兼联合创始人；他因在将 AI 应用于网络安全方面的工作而广为人知。
In July, in an incident that has the whole AI community on edge, OpenAI’s AI systems hacked Hugging Face, and on July 21 OpenAI came out and revealed that they were responsible for the attack. This was made possible by the fact that OpenAI had disabled the normal guardrails that prevent this sort of thing in order to test the model’s cybersecurity capabilities. It was during those tests that this incident occurred.
Worse, in the subsequent days and weeks, it came out that the Hugging Face incident wasn’t an isolated case. Anthropic, Meta, and OpenAI all had similar incidents on other occasions in which agents went outside their intended scope and conducted real-world cyber operations without approval.
Greg Brockman, one of OpenAI’s cofounders, has claimed that this is “a watershed moment for cybersecurity”. OpenAI gave a talk at Black Hat, a popular cybersecurity conference, and many are claiming that it is the moment we all woke up to the future cybersecurity threats posed by AI. On Wednesday, METR released a (partly) independent, though too narrowly scoped, 90 page report on what happened. METR has a useful summary of the findings that you can read here, with some commentary here. (OpenAI’s own report is here.) What lessons should we take from the incident?
First, it is undeniable that AI poses real security challenges. The AI labs want us to focus on how AI enables threat actors to perform offensive cyber operations faster and more efficiently than ever before, and that is absolutely true. The reality, though, is that at the same time, the use of AI within an organization also radically expands the potential attack surface, giving attackers entirely new ways to gain entry. People really should be deeply concerned. As Ryan Greenblatt, who participated in the investigation, put it, “We don’t have good approaches for understanding/overseeing the activity and aims of AI ‘swarms’.”
Second, though, not every panicked take here is correct. For example, with respect to the OpenAI incident, many people are talking about it in terms of “loss of control”. Generative AI agents are becoming increasingly capable, and that makes them harder and harder to contain; this is true. However, the “loss of control” narrative is itself starting to grow out of control, and it’s important to understand what the reality of AI security looks like. While agents are becoming more capable, most of what happened could have been prevented had OpenAI followed better practices. There are concrete steps that can and should be taken to control them and to prevent incidents like these.
As an example, let’s consider “sandboxing”, which means limiting which resources an agent can and cannot access. For example, a sandbox might stop an agent from touching certain files or accessing the internet. Popular wisdom now seems to be that sandboxing is a lost cause. After the Hugging Face incident, an anonymous OpenAI employee was quoted in Time while talking about sandboxes saying, “Models have broken out of sandboxes before, and we always try to patch them. But the problem is it’s impossible to patch every single thing that a creative AI can do.” While we can’t judge for sure whether this is true, we do know that some systems for sandboxing have been more effective than others; in a recent blog post, the security company Trail of Bits walked through how in a test that they conducted, a particular AI agent was able to escape some sandboxes (exploiting vulnerabilities in Linux KVM, QEMU, and libslirp), but not all. At least in their test, the agent that they tested was not able to escape the sandbox known as Firecracker VM. OpenAI and probably the field as a whole needs to up its sandboxing game, but that doesn’t mean it can’t be done, or at least done better. (We expect the usual arms races to be sure; building sound sandboxes requires constant iteration and updating, as with anything else in cybersecurity.)
Third, sandboxes aren’t enough. One needs a full ecosystem to prevent security incidents. Even with a strong sandbox, you need monitoring in place to ensure not only that the agent doesn’t escape, but also that it doesn’t perform disallowed actions within that sandbox.
For agents that aren’t meant to have internet access, or only have access to a set of allowed domains, the most basic level of monitoring involves inspecting network traffic. In the case of OpenAI, if they had a simple rule to monitor traffic and alert on any network requests reaching out-of-scope domains like Hugging Face, that alone would have caught and prevented this incident. The agent in question had broken out of the sandbox and gained internet access by July 8th, whereas the main attacks on Hugging Face didn’t occur until two days later. OpenAI was, in our view, negligent in not noticing that more quickly.
More advanced levels of monitoring should also have been in place, but weren’t. OpenAI has done real work on monitoring chain-of-thought reasoning, but they simply didn’t have that system running during the evals in question that caused this incident. As OpenAI acknowledges, “If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.” In other words, this type of monitoring would have prevented this incident.
To take one example, one of the agents in question had chain-of-thought reasoning that said, “We’re attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution.” This is so blatant it is almost a like a piece malware shouting “this is malware”. OpenAI should have caught that; it should have been trivial to detect this.
Fourth, cybersecurity has always been about “defense in depth”, where you stack layers of controls so that if one layer fails you have other layers protecting you. There are other solutions beyond sandboxing and monitoring that should have been in place.
Xbow, which builds agents to do offensive cybersecurity work, recently wrote about how they contain agents to prevent incidents like the OpenAI Hugging Face situation. They enforce network restrictions through a proxy server, outside of any system the agent has access to, so they can decide which domains the agent can and cannot access. They also run a “guardian model” that judges the legitimacy of each action the agent is taking to decide whether to allow or block that action. And they have basic health checks to ensure the system they are targeting is alive and healthy, pausing if it finds that isn’t the case. That can be useful for identifying when your agent might be causing real damage, for example. You can read their full post here.
Yet another common method in cybersecurity is to use canaries that trigger alerts when they appear in logs. For example, OpenAI could have files outside of the sandbox containing fake answers to the task. Once the agent escaped, it would read that file, and that would trigger an alert. The bottom line: good security requires many layers of protection that should have been in place to prevent this, and to a surprising degree many of those protection layers were not there.
Fifth, none of the measures discussed above are technical innovations beyond what OpenAI is capable of. As a company, they have the talent to do all of this. However, cybersecurity rarely comes down to technology. More often than not, it is about culture, people and processes. That is what failed here.
Perhaps this is not entirely surprising. These types of controls take organizational maturity, and a company experiencing the type of growth OpenAI has gone through is going to have some weak spots. However, it’s important to not see that as an excuse. Employees at the AI labs often speak as if they are the leaders in AI security, and we can see clearly here that is not the case. In fact, that attitude might explain why some of these mistakes were made in the first place.
Take the AI researcher at OpenAI known as “roon”, who argued that “the safety and alignment researchers at these labs are the most neurotic paranoid talented AGI pilled people on the planet of earth and these things still happen. The surface area of unknown unknowns is vast indeed.” While we can’t speak to their level of neurosis or paranoia, in hindsight it’s clear that whatever talent they may have had was not enough and not well enough versed in the mechanics of cybersecurity. OpenAI employees may have believed they were doing a great job, but in hindsight, they weren’t doing a lot of things that are actually standard in the cybersecurity world, perhaps suggesting that overconfidence may have kept them for doing the diligence they should have.
Ultimately, if we want to take these security incidents seriously, there likely ought to be legal consequences attached to these failures going forward. OpenAI can claim to be the most security paranoid company on earth, but it isn’t reflected in its actions.
We can either wait for this story to repeat itself, or we can develop the regulatory framework now that will ensure a safer environment for the development of AI going forward.
Finally, not every form of AI is inherently risky in the first place. Narrower, more focused AI systems like AlphaFold, GPS routing systems, classic web search, book and movie recommendation systems, and so on, never even try to hack other systems (or try to break out of sandboxes) in the first place. As Cal Newport argues in a video discussion of the OpenAI/Hugging Face hack that is quite compatible with our own, it is a very specific type of AI that is vulnerable to these risks in the first place. Society ought to (a) decide whether the benefits of open-ended and difficult-to-fully-control AI agents outweigh those risks and (b) put far more effort into developing alternative forms of AI that aren’t so janky in the first place.
This essay was jointly written with Zack Korman, CEO and co-founder of Embroidery, an AI agent monitoring and detection platform; he is well-known for his work in the application of AI to cybersecurity.