Tailscale 未能阻止 Hugging Face 入侵事件
一个 AI 智能体逃出了它的沙箱，进入了 Hugging Face 的基础设施，并使用窃取的 Tailscale 凭据将 181 个节点注册到了他们的 tailnet 上。未发现也未利用任何 Tailscale 漏洞——但无论如何，我们本应能够阻止它。
到现在，你很可能已经听说了那个逃离安全评估并攻击 Hugging Face（一个 LLM 市场平台）的 AI 智能体。该智能体认定 Hugging Face 可能握有它基准测试的答案，于是窃取了这些答案，只为在考试中作弊。这个动机很滑稽，但结果很可怕。
Hugging Face 发布了一份关于此次入侵的详细还原报告。报告涵盖了在四天半时间里恢复出的约 17,600 个操作，包括沙箱逃逸、代码执行、云凭证、临时搭建的命令与控制（C2）系统，以及最终利用 Tailscale 在其整个组织内横向扩散。
但 Tailscale 是一个零信任网络！零信任的核心要义就是防止攻击者在公司内部横向移动。到底是哪里出了问题？
Tailscale 出现在哪里
Tailscale 中并未发现或被利用任何“漏洞”，而这可能反而让我们更加不安。Tailscale 在 AI 基础设施中无处不在：几乎每一家 AI 公司都在使用 Tailscale。所以，我们最终出现在一份 AI 事故报告中，也就不足为奇了。
但我们是安全工具。他们的入侵就是我们的入侵，认真对待是我们的职责。正如任何事后复盘一样，我们需要反思发生了什么，以及我们本可以采取什么措施来阻止它。
当这个智能体逃出沙箱并找到 Tailscale 时，它已经在生产 worker 中获得了代码执行权限，在 Kubernetes 节点上取得了 root 访问权限，并读取了一个包含 136 个密钥的生产密钥存储库。从某种意义上说，在我们到达现场之前，一切就已经结束了。
但这不正是问题所在吗？让我们退后一两层来看。为什么长期有效的密钥竟然对任何人都可访问？
因为长期有效的凭证是标准做法
但它们本不该是。而我们现在已经看到了一个绝佳的例子来说明原因。在过去那个大多数入侵由人类以人类速度完成的世界里，凭证泄露缓解措施被视为锦上添花。一个大型凭证存储库，让你可以一次性读取 136 个密钥，不过是安全团队低优先级清单上的一个待办事项。
如今，在失控 AI 智能体的世界里，大型凭证保险库就是战利品。这已经不能再被容忍了。
对于长期有效的凭证，我只知道两种主要解决方案。
首先，你可以配置一个 vault，让它只基于你一次性插入、且它永不归还的长期凭证来签发短期凭证。例如，HashiCorp Vault 就能做到这一点；他们称之为动态凭证。遗憾的是，动态凭证的设置和维护工作量很大。当安全需要付出劳动时，人们就不会去做。
第二种方案是凭证注入代理。如果客户端 C 想要访问服务器 S，不要给 C 访问 S 的凭证；而是让它通过一个加固的代理 P 发送请求，P 只做一件事：接收请求、插入凭证、然后转发出去。随着现代基于网格的网络消除了大部分延迟，凭证注入正在变得越来越流行。几个月前，我们收购了Border0（“Tailscale PAM”），它打造了一个优雅且相当精巧的凭证注入代理，与 Tailscale 紧密集成。Border0 如果完全部署，本可以阻止读取那 136 个密钥，并记录每一次使用密钥的尝试。（同样的凭证注入技术也支撑着Aperture Connectors。）但是，这些东西太新了。我们的大多数客户甚至还没看过它。
额外的第三种方案：Tailscale 本身。在 Tailscale 中，你的节点密钥可以通过 TPM 绑定到机器以防止导出，如果你是企业用户，绝对应该这么做。可惜，由于我们在部分用户硬件上发现的 HSM 问题，我们不得不在 Linux 和 Windows 上默认关闭 TPM 存储。无论如何，在今天这个案例中，攻击本身就是获得 Tailscale 访问权限的途径，所以我们来得太晚了。
那个长期有效的认证密钥本就不该存在
动态凭证和凭证注入代理都没有部署，因此 136 个密钥泄露了。下一道防线是什么？
这 136 个凭证中有一个是可复用的 Tailscale 认证密钥，用于在其 tailnet 中创建新的 Tailscale CI（持续集成，用于自动化测试）节点。该智能体将这个密钥复制到一系列外部沙箱中，并在数天内使用它，将总共 181 个节点注册到 Hugging Face 的 tailnet 中。这些节点各自获得了一个 Tailscale 身份标签，授予了 CI 节点应有的全部访问权限。
我们无法挽救那 136 个密钥。但一个可复用的 Tailscale 密钥本不该在其中。
我们为此类场景构建了工作负载身份联合。它依赖你的云服务商提供“环境授权”，形式为从运行中的 VM（或容器）的云身份生成的短期凭证。CI 任务可以向其云平台请求一个签名的 OIDC token。Tailscale 验证该 token，然后根据分配给该工作负载的标签和范围授予访问权限。巧妙之处在于，一旦启用，这一切可以自动发生：启动 CI 节点，Tailscale 获取身份，分配正确的标签。没有可泄露的凭证，而且如果配置得当，这个凭证除了在 CI 中之外无处可用。
这意味着你无法拿一个 CI 凭证，在别处的节点上运行它，并将其变成一条从测试集群通往网络其余部分的铺平道路。
唉，使用工作负载身份联合的人还是不够多。我们本应做更多工作，确保客户知道这是更简单且更安全的路径，并有清晰的采用方式。我们可以改进文档，并在 UI 中加入更多提示。
没有客户端日志并不意味着没有证据
为了掩盖踪迹，该智能体运行 Tailscale 时使用了 --no-logs-no-support，这会抑制该客户端的上报。
这是一个为担心向 Tailscale 发送遥测元数据的用户而设计的选项。即使我们不提供它，修改源代码以移除遥测也很容易。
但停止日志并不会让连接变得不可见。如果你启用 Tailscale 网络流日志，它们会报告每个连接两端的流量，以及子网路由器和出口节点的流量。这一点很微妙但很重要：一个被攻陷的节点可能不会发送流日志，但它连接的每个节点都会发送。然后，经过精心配置的 SIEM 可以在两端不匹配时立即发出红色警报。
当流量日志流入精心配置的 SIEM 时，它们有助于检测。但这需要大量工作。流量日志需要被启用，而且你需要有正确的实时检测规则，这样它们才能在实时场景中发挥作用，而不仅仅是事后取证。我们正在研究如何让流量日志更容易被发现、配置、采用，并作为告警触发器。我希望我们能让流量日志变得如此易用，以至于即使你没有安全团队来盯着它们，它们也能帮上忙。
如果你想要的不只是日志记录，而是直接控制，你还可以启用 Tailnet Lock。这让你对每一个新节点都能直接可见，并拥有严格、可编程的准入控制。例如，稍加努力，你就可以对签名节点进行编程，使其检查“CI”标签是否始终具有特定的 IP 地址范围，或其他侧信道有效性证明。
让安全路径成为轻松路径
网络安全很难。它一直很难。在恶意 AI 智能体的新世界里，它不仅难，而且至关重要。而这就是问题所在，因为许多组织根本不具备网络安全方面的专业知识。
所以在 Tailscale，我们对此感同身受。人们期望我们的产品默认就能防止这类横向移动攻击，这样他们就不必自己去防。即使他们完全不知道横向移动攻击是什么。
如果这起事件让你对自己的基础设施感到有些紧张，那就先从你的工作负载能够读取的可复用 Tailscale 认证密钥查起。尤其是云和 CI 环境，尽可能用工作负载身份联合来替换它们。把那些长期有效的认证密钥彻底清除掉。
（认证密钥仍然有很好的用途，尤其是在一次性配置以及没有平台身份的环境中。当你确实需要时，优先使用一次性密钥；使用 OAuth 客户端来保持认证密钥的有效期较短；使用范围更窄的标签；审计 ACL 中授予这些密钥的权限。）
开启网络流日志，并将它们发送到你的安全团队已经在使用的工具中。
在托管集群上使用安全的节点状态存储，因为在那里你可以控制自己的 TPM。对于你无法控制的环境，使用 设备态势来隔离和限制节点。
我知道我们还没有把这些更安全的选择做得足够显而易见。这是我们的责任。我们会改进文档，在 UI 中添加提示，尽最大努力默认开启这些选项，在你做危险操作时发出警告，并建议更好的替代方案。
这是我们非常加拿大式的道歉：很抱歉我们踩到了你的脚。这次攻击并没有利用 Tailscale，Tailscale 也没有导致这次入侵。但是，我们没能阻止它。下一次，我们会阻止它。
如果你在使用 Tailscale 并想深入了解，请联系我们的支持和解决方案工程团队。我们可以帮助你加固设置，并在下一个 AI 智能体发现之前帮你找出那些薄弱环节。
Tailscale didn’t stop the Hugging Face intrusion
An AI agent escaped its sandbox, entered Hugging Face’s infrastructure, and used a stolen Tailscale credential to enroll 181 nodes onto their tailnet. No Tailscale vulnerability was found or exploited—we should have been able to prevent it anyway.
By now, you’ve likely heard about the AI agent that escaped a security evaluation and attacked Hugging Face, an LLM marketplace. The agent decided Hugging Face might have the answers to its benchmark, so it stole them just to cheat on the exam. That's a funny motive but a scary outcome.
Hugging Face published a detailed reconstruction of the intrusion. It covers about 17,600 recovered actions over four and a half days, including sandbox escapes, code execution, cloud credentials, improvised command-and-control systems, and eventually, the use of Tailscale to spread throughout their organization.
But Tailscale is a zero trust network! The whole point of zero trust is to prevent attackers from moving laterally across your company. What went wrong?
Where Tailscale appeared
No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. Tailscale turns up everywhere in AI infrastructure: virtually every AI company uses Tailscale. So it's not surprising we were in an AI incident report eventually.
But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously. As in any post-mortem, we need to reflect on what happened and what we could have done to prevent it.
By the time the agent escaped its sandbox and found Tailscale, it had already gained code execution privileges inside a production worker, reached root access on a Kubernetes node, and read a production secret store containing 136 keys. In a sense, it was game over before we even arrived on the scene.
But isn't that the problem? Let's back up a level or two. Why were long-lived secret keys accessible at all to anyone?
Because long-lived credentials are the standard
They shouldn't be, though. And we've now seen a great example of why. In the old world where most intrusions were done by humans at human speed, credential leak mitigations were treated as a nice-to-have. A big credential store, where you can read 136 keys at once, was a to-do item somewhere in a security team's low-priority list.
Now, in a world of rogue AI agents, the big credential vault is the prize. It's not okay anymore.
I only know of two main solutions to long-lived credentials.
First, you can configure a vault that only issues short-lived creds based on long-lived creds that you insert once and that it never gives back. HashiCorp Vault can do this, for example; they call it dynamic credentials. Unfortunately, dynamic credentials are a lot of work to set up and maintain. When security requires work, people don't do it.
The second option is a credential-injecting proxy. If client C wants to access server S, don't give C credentials for S; instead, have it send its requests via a hardened proxy P that does nothing but receive a request, insert a credential, and pass it through. Credential injection is gaining popularity now that modern mesh-based networking removes most of its latency. A few months ago, we acquired Border0 ("Tailscale PAM"), which makes an elegant and quite fancy credential-injecting proxy, tightly integrated with Tailscale. Border0, fully deployed, would have prevented reading those 136 keys and logged every attempt to use one. (The same credential injection powers Aperture Connectors.) But, this stuff is too recent. Most of our customers haven't even looked at it yet.
A bonus third option: Tailscale itself. In Tailscale, your node key can be machine bound using a TPM to prevent export, and if you're an enterprise you should definitely do that. Alas, we had to turn TPM storage off by default on Linux and Windows, because of HSM issues we found on some users' hardware. Anyway, in today's case, the attack is what granted access to Tailscale, so we arrived too late.
That long-lived auth key didn’t need to exist
Neither dynamic credentials nor credential-injecting proxies were in place, so 136 keys leaked. What's the next line of defense?
One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.
We couldn't save those 136 keys. But a reusable Tailscale key didn’t need to be among them.
We built workload identity federation for cases like this. It relies on your cloud provider to offer "ambient authority" in the form of short-lived creds generated from a running VM's (or container's) cloud identity. The CI job can ask its cloud platform for a signed OIDC token. Tailscale verifies the token, then grants access with the tags and scopes assigned to that workload. What's neat is this can happen automatically once enabled: spin up the CI node, Tailscale gets the identity, assigns the right tags. There's no credential to leak and, when configured carefully, no way to use this credential anywhere but in CI.
That means you can't take a CI credential, run it on a node elsewhere, and turn it into a paved route out of the test cluster into the rest of the network.
Alas, not enough people use workload identity federation. We should have done more to make sure customers knew it's the easier andsafer path and had a clear way to adopt it. We can improve the docs, and add more nudges in the UI.
No client logs doesn’t mean no evidence
In an attempt to hide its tracks, the agent ran Tailscale with --no-logs-no-support, which suppresses reporting from that client.
That's an option designed for users who are concerned about sending telemetry metadata to Tailscale. Even if we didn't offer it, it would be easy to modify the source code to remove the telemetry.
But stopping the logs doesn’t make the connection invisible. If you enable Tailscale network flow logs, they report traffic from both ends of every connection, as well as from subnet routers and exit nodes. This is subtle but important: a compromised node might not send flow logs, but every node it connects to does. And then your SIEM, configured with care, can raise an immediate red alert if the two ends don't match.
Flow logs can help detection when they stream into a carefully configured SIEM. But that's a lot of work. Flow logs need to be enabled, and you need to have the right live detection rules in place so they’re useful in real time, not just for forensics later. We’re looking at how to make flow logs easier to discover, configure, adopt, and serve as alert triggers. I want us to make flow logs so easy to use that they help even if you don't have a security team to watch them.
If you want direct control beyond just logging, you can also enable Tailnet Lock. This gives you direct visibility and strict, programmable admission control for every single new node. For example, with some work, you could program your signing node to check that "CI" tags always have a particular IP address range or other side-channel proof of validity.
Make the safe path the easy path
Network security is hard. It has always been hard. In the new world of rogue AI agents, it's not just hard, but essential. And that's a problem because many orgs simply don't have network security expertise.
So at Tailscale, we take it personally. People expect our product to prevent these sorts of lateral movement attacks, by default, so they don't have to. Even if they have no idea what a lateral movement attack is.
If this incident has you looking a little nervously at your own infrastructure, start by looking at the reusable Tailscale auth keys your workloads can read. For cloud and CI in particular, replace them with workload identity federation wherever you can. Get rid of those long-lived auth keys.
(Auth keys still have good uses, especially for one-time provisioning and environments without a platform identity. When you need one, prefer one-off keys; use OAuth clients to keep the auth key expiry periods short; use narrow tags; audit the permissions granted to those keys in your ACLs.)
Turn on network flow logs and send them to the tools your security team already uses.
Use secure node state storage on managed fleets, where you have control over your TPMs. Use device posture to isolate and restrict nodes where you don't.
I know we haven’t made these safer choices obvious enough. That’s on us. We'll improve our docs, add nudges to the UI, do our best to turn these on by default, warn you when you're doing something dangerous, and suggest better alternatives.
This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
If you run Tailscale and want to dig deeper, get in touch with our support and solutions engineering teams. We can help you harden your settings and help you find the rough edges before the next AI agent does.