#安全/对齐

Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.

Agents have a trust problem. Today’s models can do most work. What they need is the right boundaries. In traditional software, those were set at build time. With agents, they have to be set and enforced at runtime. Today we’re introducing Composio Firewall, the runtime boundary for agents. Millions of users and developers already use Composio as the runtime for their agents - giving agents tools, auth, and sandboxed execution. Now with our agent firewall every action an agent takes is evaluated using Jev and custom SLMs tuned by Composio based on real-world usage data. The firewall strengthens that existing boundary in both directions, checking the actions agents take and the information coming back to them through Composio. The advantage of the runtime boundary is that Composio Firewall verifies that the agent respects user intent and the boundaries set by the developer and user. Additionally, this provides the safety layer required to protect the user against malicious prompt injection as well as other attacks. Composio Firewall is live now for all Composio developers. Check it out at http://composio.dev/firewall

Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.

Two employees from METR and Redwood Research worked on premises at OpenAI for three weeks this summer in an embedded assessment pilot. This team evaluated the systems OpenAI was using to monitor for misalignment from internal agents. A brief write-up of their work is now public.

This is a "Defender's gap" chart that OpenAI published recently. It shows the gap between defenders' capabilities and attackers' capabilities from a cybersecurity POV. This also translates to a gap between proprietary and open AI models. What Dario is proposing is closely related: "Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively." In other words, Anthropic and OpenAI want to widen the "gap" between what their AI can do and what the rest of the world can do. This doesn't necessarily mean that they will stop AI development and training. What this leads to is: - Anthropic, OpenAI, and other frontier labs will need to work together to make sure that every lab maintains alignment standards. - These labs will continue using RSI to advance their internal models with "employee-like access". - These models WON'T be released to the public until the "gap" is sufficient and until alignment standards are met. What about China? > "Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently." > "If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important." The assumption behind these measures is simple: without being able to distill frontier models, it will take China significantly longer to close the gap with top-tier models. All the above may fay fail. China may or may not take the lead in AI progress. Yet, it's not a surprise that AI can already be used as a cybersecurity weapon. Note that the top point on the "frontier" line describes defenders' capabilities available to companies with Daybreak access and similar. However, the top point on the "open-weight" line is accessible to everyone. Even with current levels of intelligence, we will start seeing more and more major security incidents around the globe. > “Everything that makes it successful is exactly what makes it dangerous.” Monitoring 👀👀👀