OpenAI 的信任悖论：安全研究员被裁与 GPT-6 Astra 可监测性下降

Around the start of October, OpenAI fired three employees from its safety department. The employees said they were given divergent reasons for their dismissal:

• One was told it had to do with his role coordinating with METR, a third-party research organization that OpenAI had invited to investigate the Hugging Face incident. “I was told verbally I was fired because of the way I communicated with METR,” Tomek Korbak wrote on X. “No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.”

• Another was told something similar but vaguer. “I was told OpenAI no longer trusts me because I was speaking too much to third-party safety organizations, implying I leaked company IP. I never shared company IP,” Mikita Balesni wrote.

• And a third was told something quite different. “I was given one reason: that I accessed an executive’s email,” Jasmine Wang wrote. “OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn’t remove it myself, and the inbox was combined in an indistinguishable way in my phone’s mail app. When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again. None of this was hidden.”

After the trio’s dismissal leaked, OpenAI gave a vague statement to the Wall Street Journal: “Our investigation confirmed that these individuals mishandled sensitive information outside established company procedures, violating our policies and breaking the trust essential to our work.”

Subscribe now

OpenAI reiterated that theme a week later, after the fired researchers released an open letter disputing the company’s account. In an unsigned statement attributed to the company’s “research leaders,” OpenAI accused the former employees of a “significant breach of trust.” The company said that “these decisions were not about raising safety concerns or speaking out” and insisted that “we actively encourage” internal discussion of such issues.

Who is telling the truth? Did the employees commit a breach of trust that necessitated their dismissal? Or were they fired for being too critical of the company’s safety choices during internal deliberations?

The difference matters for OpenAI’s remaining employees because some of them may soon have to decide whether to raise concerns of their own.

OpenAI insists that employees have nothing to worry about, that this was a special case, and that it is extremely forgiving of good-faith mistakes. But employees have good reason to be skeptical. After all, if OpenAI had fired employees in retaliation for raising safety concerns or for crossing an unspoken line, the company would have strong incentives not to admit it. Employees have to wonder whether OpenAI’s actual reasons for firing the trio differ from its stated reasons.

Ironically, the fired researchers had been raising concerns that OpenAI might be giving its models a new capacity to conceal their thinking. Modern AI agents reason using a chain of thought, a scratchpad where models think carefully about what action to take next. Many researchers view these chain-of-thought tokens as a safety lifeline, because they offer insight into why models act the way they do.

But OpenAI recently made an architectural change that allows models to think for longer before generating each chain-of-thought token. This seems to make the models more capable; it may also make their reasoning less transparent to human overseers. The sparser the chain of thought, the harder it is to understand what the model is trying to do, or even whether it might be trying to hide its motivations.

Monitorability going missing

Before they were fired, the three researchers seem to have had two major concerns. One was understanding how and why the Hugging Face attack happened and how to prevent incidents like it in the future. The other was keeping AI systems legible to their creators and users — an issue that was a significant source of drama in September, after The Information reported that OpenAI’s new model could hide its thinking in unprecedented ways.

The researchers think their firing was related to this second issue.

“For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired,” Korbak wrote.

“The three of us were pushing internally for industry-wide commitments to preserve our ability to monitor AI reasoning. This work requires talking daily to third parties,” Balesni wrote.

“As an industry, we do not yet know how to safely develop and deploy models that we cannot monitor,” their joint letter said. “The monitorability of frontier models is degrading. As long as we rely on monitorability for safety, OpenAI and other frontier companies should not move forward with developments that further decrease monitorability.” The researchers also alluded to rumors inside OpenAI that they had tipped off The Information, which they denied.

Subscribe now

To understand the researchers’ concerns, we have to go back to 2024, when OpenAI introduced a technique that changed the course of AI. It gave models a notepad and trained them to jot down many “thinking tokens” before producing a final answer. This technique sounds simple, but it has enabled models to solve problems that had previously eluded them.

An added benefit of this approach: humans can read the notepads and see how the models think — including whether they’re thinking about taking harmful action or are being dishonest. Indeed, much of what we know about the Hugging Face incident comes from the chain-of-thought monologues that the hacking agents left behind.

Chain-of-thought monitoring isn’t perfect. For one thing, it’s not established that the chain of thought fully tracks how models think. For another, safety researchers worry that poorly designed training methods could give models an incentive to hide their true intentions by outputting thinking tokens that don’t represent their inner motivations.

But whatever flaws the technique might have, AI companies don’t have a lot of other choices. The most promising alternative for understanding model motivations is mechanistic interpretability — roughly the equivalent of cutting open a model’s brain and trying to figure out what it’s thinking by reading the pulses of its neurons. Although mechanistic interpretability has produced valuable insights about how models work, it has not so far yielded many practical techniques for supervising models’ behavior.

For this reason, those worried about unsafe or deceptive AI tend to view chain-of-thought monitoring as an irreplaceable asset.

Looping layers

Lately, safety advocates — like the three fired OpenAI researchers — have grown concerned that chain-of-thought monitoring is becoming less effective. OpenAI’s new flagship model, GPT-6 Astra, is much better at thinking without needing to write anything down. The system card OpenAI itself published says Astra “shows a substantial decrease in chain-of-thought monitorability compared to previous models” and notes that Astra is “significantly more able” to control its own chain of thought. One independent test found it “has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1).”

This might be partly because smarter models just naturally write less down. But it also seems to be a byproduct of a new architectural technique called recurrent depth that was introduced with Astra.

Large language models think in layers. Conventional LLMs pass information forward from one layer to the next. When they reach the final layer, they output a token before starting the process over again.

Models with recurrent depth break that forward-only pattern: information from a later layer gets passed back to an earlier layer. Layers get used twice (or possibly more) without the model emitting a token in between. If implemented just right, such a looped architecture seems to make models smarter and more efficient: a model that has 100 layers looped twice might reason nearly as well as a model with 200 layers, while requiring less memory and hence being less expensive to deploy.

But looping layers also seems to make the model harder for outsiders to monitor. If given a relatively easy problem, Astra can often output the answer directly — no chain of thought required. For harder problems, Astra still needs to output a chain of thought, but its chain of thought will be shorter, and perhaps harder to interpret, than those of earlier models.

Subscribe now

OpenAI has pushed back on these concerns. In a September tweet, OpenAI executive Jakub Pachocki said that “the depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.” In other words, even accounting for its looped architecture, information isn’t passing through dramatically more layers (compared to previous models like GPT-4) before Astra outputs each token.

But safety advocates worry that Astra could be just the first step. If the modest amount of looping in Astra makes the model smarter and more efficient, future models might use looping more aggressively. Future chains of thought might become increasingly sparse, vague, or otherwise hard to understand. Critics — including the fired OpenAI researchers — fear that this will make it easier for models to scheme or dissemble.

OpenAI has said it agrees with these worries in principle. Last year, it wrote, “We believe that chain-of-thought (CoT) monitoring may be one of the few effective methods we have for supervising superhuman models.” In September, Pachocki said that he too was worried about monitorability and insisted that OpenAI hadn’t gone very far in the direction of weakening it. A few days later, he published an essay called “An Alien Mind” explaining the necessity of and challenges with monitorability as he saw them. And a few weeks ago, replying to the fired researchers, the company reiterated its belief that monitorability requires an industry-wide commitment.

And yet OpenAI has created a less monitorable model. The researchers who were fired worry this is the beginning of a trend. Indeed, OpenAI’s unreleased next flagship model is reported to use looping as well.

Say as little as possible

Perhaps OpenAI’s statements are correct. Maybe it is as committed to monitorability as anyone, and the researchers were fired for leaking rather than for good-faith safety work. Even then, the theme of monitorability is helpful in clarifying the broader situation. A company that has not been able to sufficiently monitor its existing models is building newer models it can monitor even less, even as it also — by its own account — cannot sufficiently monitor the employees whom it has tasked with preserving monitorability, and so must fire them.

“We cannot do the work in front of us without a high degree of trust,” OpenAI’s research leaders wrote in their response to the employees they’d fired. The employees insist they were worthy of that trust, and their response seems credible. But suppose they weren’t worthy — say they intentionally leaked some crucial piece of information about monitorability or Hugging Face. If they did, it was likely because they did not trust the company to do the right thing — or even to do the things the company had already promised to do.

The company’s response seems unlikely to earn anyone’s trust. The research leaders who issued it do not explain their actions very clearly; they don’t even give their names.

Given that this has happened at OpenAI before — see the dispute about the firing of another safety researcher on similar grounds a couple of years ago — one could be forgiven for thinking that the confusion is the point, that the gap between statement and action is itself the message.

This is surely too cynical. But functionally it will not matter. “Say as little as possible to anyone” may not be the intended message, but it is the received one — received by the safety researchers presently at OpenAI and by any who might think of joining in the future.

After this, and after the string of voluntary or seemingly voluntary departures over the summer (plus another senior departure right around the firings), and after the string of hacking incidents slowly revealed during the same time, how many are left who won’t have received this message? For this reason, a senior ex-employee (who seems to have left on good terms) worries now about OpenAI “losing the best talent in the space.”

The solution is the same

The situation is ironic. Just as people are worried LLMs like Astra might have secret reasons that are different from what they say or jot down in their chains of thought, people are now worried that OpenAI might have secret reasons for behaving that are different from the ones it expresses to its staff and the public.

In both cases, the solution is the same: more transparency to increase confidence that the expressed reasons are the actual reasons.

For LLMs, this means redoubling commitment to chain of thought and prioritizing the development of monitoring techniques that can replace it, before models that can work without it are put into production. The world would have had a much poorer understanding of the Hugging Face incident without chain of thought.

Meanwhile, if OpenAI truly wants employees and the public to trust it, it must offer itself up for the kind of monitoring it claims to prioritize in its models. This might mean not only being open to but actively demanding greater oversight from independent organizations, like METR and Redwood Research, and especially from government authorities, state as well as federal.

If it does this, it will have done the world a meaningful favor. Other AI companies may not have quite the same drama, either because of greater diligence or simply because they don’t have OpenAI’s special combination of cutting-edge research ability and organizational chaos. But they too have problems with monitorability and openness, and mandating monitorability would ultimately do the whole industry a favor — it would make all their statements more believable.

One report just came out that Google’s newly released Gemini 4 Argon model has a recurrent architecture as well. This is unconfirmed, but Google itself admits Argon outputs less monitorable chains of thought, can control its own chain of thought well enough to say “hello evaluators reading my chain-of-thought,” and, when instructed to, can evade current monitoring well enough to perform “simple sabotage” half the time.

As for Anthropic, outsiders don’t know if its models are recurrent. Anthropic may emphasize safety, but it is in some ways less open to the world than OpenAI — less prone to leak and less prone to have employees who post about what the company is doing or simply how they see things. This is surely helpful in avoiding drama. It is not helpful in establishing the public faith required for what it too is trying to do. Like OpenAI, Anthropic ultimately believes it is creating an alien mind. Trust us, these companies say. But without extreme visibility into what’s actually going on, down to minute technical detail, nobody will.

Subscribe now