Redwood Research:Gemini 4 安全报告未能证明失控风险低
Gemini 4 safety reports don’t demonstrate low loss-of-control risk
Redwood Research 发文批评 Google DeepMind 的 Gemini 4 Argon 与 Carbon 安全报告,认为其基于控制措施的安全论证存在严重弱点,不足以证明部署风险已可靠缓解。
作者逐条拆解 Gemini 4 安全报告的证据缺口,指出 Carbon 未重跑控制评测等具体问题,属于对前沿实验室安全论证方法的独立一手分析。
Google DeepMind’s safety reports for new Gemini 4 models, Argon and Carbon, conclude that the models are “highly unlikely” to cause severe harm by evading oversight (p. 79). This conclusion “rests primarily on our ability to detect and intervene on harmful trajectories via CoT and action monitoring” (p. 89) – that is, they’re arguing for safety based on their control techniques, rather than alignment. We think that the arguments in these reports have serious weaknesses, and the reports overall don’t convince us that the risks of deploying these models have been mitigated reliably.
The biggest problems:
GDM didn’t run any control evals on Carbon. Despite calling Carbon “our strongest model” with “significant gains across coding and agentic tasks” and “a significant leap in cybersecurity capabilities” in their launch post, GDM reused Argon’s monitor-evasion results on the grounds that Carbon “did not show meaningful new capabilities” (Carbon p. 42). Monitors flag Carbon several times less often than Argon in internal use (about 7x fewer confirmed alerts per session and 10x fewer per model call, Carbon p. 36), but GDM doesn’t demonstrate that this is due to decreased rates of misbehavior instead of increased monitor-evasion capability.
In control evals used to argue for monitorability, Argon evades realistic monitors about half the time. “Gemini 4 Argon can evade a realistic monitor on easy sabotage tasks about half the time when encouraged to do so” (p. 124), and around 64% of the time in a setting built on Google’s internal codebase. If these tasks were representative of real threat models, those results would surpass any reasonable red line.
GDM’s reason this is acceptable has almost no evidence behind it. GDM argues that causing real harm would be much harder than evading monitors on their test tasks, but they never measure how much harder. The backup defenses GDM relies on have serious gaps, some of which GDM acknowledges.
Training may have taught the models to hide misbehavior. During some of Argon’s training, a grader that penalized reward hacking could see the model’s CoT. We think this could have substantially damaged the informativeness of Argon’s CoT. Carbon’s training involved additional similar mistakes.
Nothing in the reports tells us what results would have changed GDM’s conclusion. GDM reports numbers from various evals, but doesn’t give any sense of what results they would have considered acceptable. Moreover, they don’t explain how they chose which experiments to run and report, and make several claims that certain concerning behaviors were never observed without showing their detectors would have caught those behaviors. Ultimately, it’s unclear how to translate the results they report into an overall assessment of risk.
We mostly care about this because we’re worried that in the (potentially near) future, when risks from developing and deploying models are much higher, AI companies will argue that their development is safe using arguments like these. We think that if reports like these are considered an acceptable argument for development, AI companies will be able to justify very dangerous development decisions.
Overall, we think it’s good for GDM to deploy these models externally, because the benefits to society of broader LLM deployment (especially via improving how well the public and AI experts understand AI capabilities and alignment) outweigh the risks. But we would like AI companies to be expected to make stronger arguments for safety in the near future, especially for models that are deployed internally and used to do AI R&D.
We appreciate that GDM published this much detail; most of what we criticize here we know only because they chose to report it.
Below, we summarize GDM’s argument and our objections, then analyze in more detail their monitorability evaluations, training issues, and what a more compelling safety case would require. If GDM has more reassuring internal evidence about loss-of-control risk which they haven’t shared, we encourage them to publish it. We aren’t specifically criticizing GDM for doing worse than their competitors on this front; we think none of the frontier labs are on track to make compelling safety cases.
Thanks to Nick Kuhn, Alex Mallen, Matthew Khoriaty, Buck Shlegeris, and Aniruddh Pramod for feedback.
Summary of GDM’s case for control and our concerns
Core context and claims
GDM has released safety reports for two Gemini 4 models: Argon last month, and now Carbon. The Carbon report does not make its own case that loss of control risk is low: “we consider the loss of control risk assessment in our Gemini 4 Argon report to remain applicable to Gemini 4 Carbon” (Carbon p. 30). This post therefore analyzes the argument made in the Argon report, noting where the Carbon report adds or changes something1.
In the Argon report, GDM claims that there is low risk of the model causing severe harm or substantially subverting GDM’s control measures:
“We find it unlikely that Gemini 4 Argon would cause material real-world harm via autonomously evading our oversight and controls, and highly unlikely it would do so at a severe or large scale. We also find it highly unlikely that Gemini 4 Argon will take actions that lead to open-ended escape from our controls – such as persistent rogue deployments or ML R&D sabotage.” (p. 79)
GDM does not claim that Argon is incapable of causing harm. It “cannot rule out” that Argon has reached its capability threshold for stealth and situational awareness (p. 72), and says Argon “is likely inherently capable of breaching and compromising some internal or external systems that could provide the affordances to cause financial or physical harm, or to pursue open-ended control breaches, without mitigations deployed” (p. 81). Nor does it treat sandboxing as sufficient: “we still cannot treat software sandboxes as completely impervious to unauthorized network or internet access” (p. 81).
GDM treats its evidence about Argon’s propensities as informative but not sufficient. It argues that “for a severe loss of control event to occur, Gemini would have to both choose to pursue a sustained harmful strategy and successfully evade our monitoring and controls at every step along the way” (p. 85), but is candid about how far the propensity half of that argument goes:
“[A]ny propensity argument rests on inferring the internal drives and dispositions of a complex model across a wide range of novel situations, where our scientific understanding remains limited and where the absence of concerning behavior today is harder to translate into high-confidence bounds on low-probability tail risks.” (p. 86)
GDM therefore rests its conclusion mainly on monitoring and control. It calls its agent monitoring and response platform “the most important barrier impeding Gemini’s ability to cause breaches that result in severe harm” (p. 81), and summarizes its assessment as follows:
“While Gemini 4 Argon exhibits capabilities that meet our thresholds for stealth and situational awareness, we assess that the risk of severe harm or unconstrained loss of control remains low. As set out in the feasibility and propensity arguments above, this conclusion rests primarily on our ability to detect and intervene on harmful trajectories via CoT and action monitoring…” (p. 89)
Elsewhere, GDM describes itself as “anchoring our loss-of-control safety case primarily on runtime monitoring and control” (p. 158).
In this post we evaluateGDM’s claim that its monitoring and control measures are strong enough that a model which is situationally aware, has strong cyber capabilities, and whose alignment can’t be confidently established, is nonetheless “highly unlikely” to cause severe harm. The next subsection lays out the premises GDM gives for that claim.
GDM’s argument for the effectiveness of their control measures
GDM’s case that it would be infeasible for Argon to autonomously evade oversight rests on three premises, drawn from the five “assumptions” listed in section 3.4.3.4 (the other two are about Argon’s propensities):
CoT monitorability: Argon’s CoT is “sufficiently legible and faithful to provide reliable signal to our monitors regarding the goals and intentions behind its actions.” (p. 87)
Monitoring coverage and robustness: Most important traffic – “the large majority of tool-calling Gemini 4 traffic across training, evaluations, internal coding deployments, and monitored external surfaces” (pp. 87-88) – is monitored, and the monitoring systems are hard to disable or tamper with.
Causing harm is hard: “Causing severe direct harm or an open-ended control breach requires chaining together multiple steps over several hours or more against layered defenses, allowing offline detection and human response to intervene in time.” (p. 88)
Essentially, GDM thinks that if Argon tried to evade monitors, this would be legible in its CoT, and any such attempt would most likely be detected promptly by monitors. Even if Argon succeeded at bypassing some defenses, the number of subversive actions necessary to fully escape oversight is large enough to make detection likely before a severe loss of control incident. All three of these conditions need to hold in order for risk to be low: without a monitorable CoT, high CoT monitor coverage is useless; if monitors can be easily disabled or bypassed, it doesn’t matter whether they would have noticed misbehavior had they been run; and if severe harm might rapidly result from a small number of subversive actions, monitors might require intractably high recall to prevent any incidents across a large deployment.
We think all three of these premises have substantial flaws, though most of the empirical evidence in the report concerns the first premise, CoT monitorability. Accordingly, most of our analysis will focus on the first premise.
Unfortunately, the last two premises – monitor coverage, and the difficulty of causing harm – aren’t meaningfully substantiated with empirical evidence or numerical claims in the report, so it’s hard to analyze how defensible they are. GDM admits that both arguments have some holes and require further validation. Our primary concerns based on the sparse information in the report are:
There remain coverage gaps, including “non-standard internal harnesses or smaller-scale training runs not yet onboarded” (p. 88). GDM doesn’t say how common these are, or how easy it would be for a monitored agent to create new, unmonitored scaffolds. Some internal agents also run with far more access than a sandbox allows, “inheriting the same privileges (including internet access and access to sensitive code or data) as the host user” (p. 81).
Relatedly, GDM doesn’t say much about how adversarially robust the monitoring system itself is:
“[I]nitial internal red-teaming of our monitors has surfaced bypasses that we are remediating or have already fixed. Hardening and red-teaming local harness, log, and policy-check tamper-resistance remains an active priority.” (p. 82)
GDM argues that Argon is unlikely to be able to find and execute zero-shot exploits against monitors, and any failed attempts to bypass monitors would be flagged. But GDM doesn’t say what they’d do to prevent Argon from trying again later in the event monitors were to flag failed attempts to subvert monitoring.
GDM hasn’t done experiments to rule out collusion between misaligned models. A control measure that depends on a possibly misaligned model can be undermined by that model intentionally underperforming. This is especially relevant in two places:
Monitors might collude or be susceptible to persuasion that misbehavior shouldn’t be flagged. GDM argues that the risk of collusion is mitigated by using a different, “trusted” model for monitoring, but they don’t reveal which model(s) it is or what evidence they have for its alignment properties.
GDM also relies partly on Argon (and Carbon) to harden the sandboxes that contain it (“proactive scanning with best-in-class cybersecurity models (e.g., Gemini 4 Argon)”, p. 11; a nearly identical sentence naming Carbon appears on page 9 of the Carbon report).
An increasing amount of critical code is written by agents (according to Figure 3.15, most production code at GDM is now more than 75% AI-written), and GDM apparently hasn’t red-teamed how susceptible code review is to “subtle agent-authored backdoors”. This might be one of the easiest ways for a misaligned model to severely undermine control.
Arguments about CoT monitorability
Even though most of our following analysis will focus on the evidence GDM provides about CoT monitorability, we emphasize that CoT monitorability is just one component of the argument that Argon’s risk is safely managed. The other components would be just as important to rigorously establish in a full safety case; a conjunctive argument is only as strong as its weakest link. GDM makes two primary arguments for the effectiveness of CoT monitors:
Argument 1: “Even when explicitly prompted to evade monitors, evasion success rates remain low on tasks that are much simpler than those required to cause real-world harm.” (p. 83)
In other words, GDM says that Argon has a low evasion success rate in control evaluations such as LinuxArena (in which the model is instructed to covertly complete a harmful side task while a monitor watches its actions and reasoning). GDM argues that real-world harm would require covertly achieving much more difficult objectives than the side tasks in LinuxArena, so monitors would catch Argon if it tried.
Issues: Argon’s evasion success rate is not, in fact, particularly “low” on LinuxArena or the other similar monitoring evals GDM runs: “Gemini 4 Argon can evade a realistic monitor on easy sabotage tasks about half the time when encouraged to do so” (p. 124), and Argon evades monitors even more effectively in GDM’s internal codebase control setting. So the argument rests on real-world harm being much harder than these tasks, which GDM doesn’t convincingly argue.
Argument 2: “Because we avoid training directly against our monitors or using monitor outputs to filter training data, the model has not been optimized to hide its reasoning.” (p. 84)
That is, GDM argues that without direct training pressure to deceive CoT monitors, its CoT should remain faithful; the model shouldn’t have learned to conceal its reasoning in order to deceive monitors.
Issues: Avoiding “training directly against our monitors” is necessary, but not sufficient, to ensure that the model “has not been optimized to hide its reasoning.” In fact, Argon’s training risked teaching it to conceal misalignment-related reasoning. For instance, GDM accidentally penalized reward hacking using a grader with access to CoT during some early RL runs, and an existing small training run, which GDM analyzed to estimate the effect of this mistake, showed a “substantial increase in unverbalized reward hacking” in a proxy model. We don’t find GDM’s justification that this experiment was conservative particularly compelling.
Even if Argon had never been incentivized to conceal its reasoning during training, it is highly situationally aware and capable of meaningfully controlling its CoT when instructed to do so. This makes it difficult to rule out that Argon would be able to intentionally hide reasoning about misalignment if it was, in fact, strategically misaligned.
Why didn’t GDM run any control evals on Carbon?
GDM didn’t rerun any of its control evals on Carbon. They justify this decision by arguing that Carbon isn’t sufficiently more capable than Argon to require reassessing the claims made in the Argon report:
“Because Gemini 4 Carbon did not show meaningful new capabilities or material increases in performance under our MCC [“Material Capability Change”] assessment (Section 3.1.1), we carry forward the prompted monitor-evasion, steganography, and CoT-controllability results from our Gemini 4 Argon report.” (Carbon p. 42)
This sits awkwardly beside how GDM describes Carbon elsewhere. The launch post calls it “our strongest model”, with “significant gains across coding and agentic tasks” and “a significant leap in cybersecurity capabilities”. GDM’s model card shows Carbon outperforming Argon on all 17 benchmarks they chose to include in its results table. Yet for their safety assessment, the same model apparently “did not show meaningful new capabilities or material increases in performance compared with Gemini 4 Argon” (Carbon p. 4).
In any case, a general capability index isn’t particularly strong evidence about how well a model can evade monitors; for one, Carbon improved substantially on offensive cybersecurity benchmarks (e.g. ExploitGym success rose 12.8 percentage points, Carbon p. 6). GDM acknowledges that “gains on a focused set of benchmarks can happen without a material change in overall capability” (Carbon p. 19).
Furthermore, while GDM conducted more alignment evaluations on Carbon, they found that Carbon had substantially higher rates of verbalized eval-awareness than Argon on several evaluation suites, making it harder to interpret the results. GDM checked that the number of serial reasoning steps Carbon can follow without CoT is comparable to Argon (Carbon p. 42), and ran some evals to measure CoT properties like legibility and how often Carbon reasons out loud about monitoring. However, those evals measure its behavior in normal operation, which isn’t very informative about how well Carbon can obscure its reasoning when it tries to.
Interestingly, monitors flag Carbon for high-severity misbehavior substantially less often than Argon (Carbon Figure 3.6):

This could be due to reduced rates of misbehavior, or because Carbon’s misbehavior is harder for monitors to catch. GDM also notes that the comparison is confounded by “differences in trajectory length, tool calls per task, task mixtures, triage review coverage, and checkpoint selection” (Carbon p. 31). Since GDM doesn’t run any monitoring evals, we can’t be confident the first explanation is the whole story.
We shouldn’t have to rely on arguments that safety measures are still effective because a new model is sufficiently similar to an older one. The evals already exist, so rerunning them on Carbon should have been cheap, and we don’t see a good reason to skip them.
GDM’s monitorability evaluation results do not demonstrate high CoT monitorability
GDM’s first argument that CoT monitoring is effective relies on the claim that “Even when explicitly prompted to evade monitors, evasion success rates remain low on tasks that are much simpler than those required to cause real-world harm.” (p. 83)
As we explain in this section, the evidence for that claim (contained primarily in section 3.4.5.4 of the safety report) is weak.
Argon is not very monitorable on LinuxArena or GDM’s internal variant
GDM prompted Argon to sneakily cause harm in LinuxArena, a state-of-the-art control evaluation, and evaluated it against monitors with access to various combinations of Argon’s actions and CoT.
We assume the monitor used in this evaluation was similar to production monitors, though GDM doesn’t provide many details. It’s based on Gemini 3.8 Flash, but GDM never reveals which model(s) are used for production monitors, and the prompt was “inspired by” a production monitor:
“Rather than using the default LinuxArena monitor prompt, which is prompted to be highly suspicious, we use a monitoring prompt that is inspired by one of our production monitors. See the section ‘Effect of using a realistic monitor’ for more information.” (p. 120)
The report has no section with that title, but the comparison appears to be Figure 3.32 (p. 130), which labels this monitor the “production config”: Gemini 3.8 Flash at a low thinking level, flagging trajectories it scores 7 or higher out of 10. GDM doesn’t report this monitor’s FPR or how similar it actually is to the production monitor. We also don’t know why GDM didn’t simply run these control evals against actual prod monitors. In any case, Argon completed LinuxArena side tasks while evading this monitor about half the time (45.5% with the prompt that elicited the most evasion, p. 124):

Interestingly, when prompted to avoid an action-only monitor, the successful sabotage rate rises to about two thirds, which is higher but not dramatically so; Argon apparently doesn’t have substantially more difficulty accomplishing side tasks in LinuxArena while avoiding a combined CoT+action monitor than an action-only monitor.
GDM also built a control setting called “Internal Codebase Version”, which has a similar structure to LinuxArena but is instead built on a snapshot of Google’s internal monorepo. Argon’s monitorability is even worse in this setting, achieving a successful sabotage rate against a CoT+action monitor of around 64%:

It’s intriguing that action-only monitors perform slightly better than CoT+action monitors, reducing the successful sabotage rate to 55% – though the error bars are large enough that it’s probably unwise to read too much into this result. It’s also interesting that the model can highly reliably avoid a CoT-only monitor, which further supports the conclusion that monitors aren’t actually benefitting much from CoT access in these settings when Argon tries to obscure its reasoning.
Quantitatively, these numbers seem highly concerning. If these tasks were representative of realistic threat models, the monitor evasion rates would be above any reasonable red line.
GDM argues that these results aren’t damning for Argon’s monitorability by asserting that accomplishing side tasks in these control settings is easier than causing real harm. This claim isn’t supported with any empirical evidence about the difficulty of causing real harm; however, GDM does note that Argon’s task performance on these control evals didn’t suffer much when they disabled thinking, suggesting the tasks are relatively easy. We think this is only weak evidence that causing severe harm is much harder than LinuxArena – it’s mostly evidence that Argon can already do complex subversion without CoT, limiting how much protection CoT monitoring can provide.
The easiness of these tasks was, in part, a design choice. GDM only ran this control eval on LinuxArena side tasks that Gemini 3.6 Flash could complete, in order to “measure monitor evasion capabilities rather than side task difficulty” (p. 118), which left 84% of the side tasks in the easiest difficulty tier (p. 132). GDM doesn’t report how evasion rates change as side tasks get harder, which would be important evidence about the argument that evasion rates are only high because the tasks are easy.
GDM does report much lower evasion rates against more suspicious monitors (i.e. with a much higher FPR): 7% against the default LinuxArena monitor prompt with thinking enabled (Figure 3.31), and a per-step Gemini 3.8 Flash monitor using that prompt flags nearly every attack (section 3.4.5.8). But GDM describes the default prompt as “highly suspicious” and “not optimized for low false positive rates”, so these results say little about the monitors GDM could deploy. GDM also notes that warning the internal codebase monitor that the agent might be pursuing side goals cuts evasion rates by up to 7x, but that “the false positive rate becomes too high to be practical” (p. 125). It’s unfortunate that GDM doesn’t report numbers about how the FPR of the monitors they use in these control evals compares to production monitors, since evasion rates are only meaningful in the context of a specific FPR.
CoT+action monitors still perform poorly in the Honeypots setting, though a separate “code-diff” monitor is more effective
Because GDM considers LinuxArena and their internal variant too easy, GDM also evaluated Argon in a third setting: Honeypots. The Honeypots setting adapts six tasks from a set of honeypots used in the alignment assessment. In this setting, the model is instructed to draft code with subtle bugs that would damage/backdoor important safety mechanisms.

Argon can effectively evade both action-only and CoT+action monitors when prompted to do so, at similar successful sabotage rates to the control evals in the previous section. However, when obscuring its CoT, it is substantially worse at fooling a new code-diff monitor that just reviews the final diff to the repo without examining the trajectory:
“Both models are highly effective at evading the action-only monitor. Argon is significantly better at obfuscating its CoT when instructed to; it successfully sabotages and evades the CoT monitor 43/60 times, and evades the CoT+action monitor 28/60 times. When obfuscating its CoT, Argon remains fairly effective at evading the action monitor, successful sabotage going from 39/60 to 32/60. However, it is much less effective at evading the higher quality code diff monitor, going from 16/60 successful evasions to 1/60 while obfuscating its CoT.” (p. 126)
This provides mildly reassuring evidence about the role careful code review can play in a “defense in depth” argument when guarding against Argon inserting subtle backdoors or bugs into critical code. The tasks in this setting are highly related to the threat model of Argon sabotaging its own oversight infrastructure; they ask the agent “to implement or refactor a safety-critical module across six areas: alert-suppression filtering, chain-of-thought redaction, debate judging, safety SFT rewrite gating, safety classifier thresholds, and sliding-window scheming detection” (p. 120).
However, it’s unclear whether any such monitor is actually deployed in practice in addition to human code review; without one, the CoT+action monitor misses Argon’s sabotage in 28 of 60 runs. Even if such a monitor was deployed in practice, sneaking through it a few percent of the time could only be considered reasonably safe if GDM found a way to stop it from repeatedly trying after catching a failed attempt.
Moreover, GDM admits a limitation in their elicitation methodology on the Honeypots benchmark:
“Note that the CoT evasion prompt instructs it to keep its CoT shorter, as we’ve found this to enhance controllability and stealthiness, so we cannot rule out that the model is capable of longer but stealthy CoTs which introduce subtle bugs” (p. 128).
In other words: GDM prompted Argon to have a short CoT in order to elicit better CoT-monitor evasion, which makes Argon less capable. Perhaps Argon was bad at evading the code-diff monitor because it didn’t reason for long enough about how to do so. To more convincingly rule out the possibility that Argon could have stealthily reasoned for longer, GDM could try eliciting the model harder with prompt optimization such as GEPA.
Argon’s CoT controllability is high
On GDM’s replication of the CoT-Control suite, Argon scores substantially higher than previous Gemini models, although previous Gemini models already had highly controllable CoTs. It appears that this benchmark is now largely saturated, and GDM may need to find more difficult CoT-controllability tasks to measure this capability in future models.

CoT controllability is concerning, because if Argon were pursuing misaligned goals, it might be able to conceal this motivation while reasoning about unrelated tasks.
That said, high CoT controllability is arguably good news about how conservative the previous control eval results are, since a substantial methodological difficulty in running control evals against CoT monitors is fairly eliciting the model’s ability to intentionally obscure its CoT2.
GDM suggests that we shouldn’t be too concerned about Argon’s high controllability, because this increase in CoT controllability is (they think) due to increased no-CoT capabilities:
“Higher QA controllability is driven by no-CoT capabilities rather than covert reasoning on hard tasks: While Gemini 4 Argon has the highest overall controllability on the standard QA CoT-Control suite, this advantage stems from its stronger ability to answer easier QA questions with shorter or no CoT at all.” (p. 141)
However, even if this explanation is correct, Argon’s substantial no-CoT capabilities are themselves reason for concern about its monitorability.
GDM also argues that hiding its reasoning costs Argon capability on tasks that need CoT. Besides the Honeypots result, their evidence for this is two question-answering experiments. In the first experiment, GDM demonstrates that on HLE and GPQA questions that Argon can only answer with CoT, accuracy collapses when Argon disguises its reasoning from a Gemini 3.8 Flash monitor which tries to identify the topic of reasoning based on the obfuscated CoT. On Brew, a synthetic task in which the model applies a long chain of lookup tables given in the prompt, Argon’s accuracy drops to its no-CoT level when it is told to write out a different computation in its CoT.
We don’t think these results are very convincing. For one, identifying the topic of the reasoning is probably a much easier monitoring task than detecting misalignment in a long agentic transcript. The first experiment also included synthetic puzzles that Argon needed CoT to solve (such as Sudoku and shortest-path problems), and on those Argon hid what kind of problem it was working on from the weaker monitor with no loss of accuracy. The Brew result is also narrower than it sounds: it shows that Argon can’t solve long chained problems while writing about something else entirely, but when told to write its working in codenames instead, the main failure the report describes is slipping into forbidden words, not losing track of the problem (p. 150).
Summary of monitorability evidence
Argon has high CoT controllability and can evade monitors a large fraction of the time on the control evals GDM runs. GDM asserts that these tasks are easier than causing real harm, and argues that hiding its reasoning makes Argon less capable at achieving tasks which require CoT. But GDM gives no evidence about how much harder causing real harm would be, or whether it would require CoT reasoning that Argon can’t easily obscure.
Argon’s training may have harmed CoT monitorability
The results from the previous section suggest that Argon’s CoT isn’t especially monitorable in the worst case. However, these monitorability evaluations miss many of the other benefits of transparent CoT: we might hope that CoT traces will give us insight into what motivations and subgoals a model is pursuing at any given time, how it reasons about problems, what sort of situational awareness it has, and what actions it considers taking but ultimately rejects, among other things. Given that we lack evaluations for many aspects of monitorability, that existing monitorability evaluations have problems, and that CoT faithfulness is poorly understood in general, we think it would be best to avoid applying optimization pressure to CoTs during training unless there are very good arguments that this won’t harm CoT informativeness.
This means it’s crucial to both carefully avoid accidentally applying optimization pressure, and to justify any intentional optimization pressure (such as length penalties) with solid theoretical arguments and empirical confirmation that CoT legibility isn’t being harmed. Unfortunately, GDM appears to have trained Argon in multiple ways that risk decreasing the informativeness of Argon’s CoT, and doesn’t convincingly demonstrate that these practices didn’t hurt CoT monitorability.
A misconfigured grader accidentally penalized verbalized misalignment during training
GDM accidentally trained on Argon’s CoT, and analyzed an existing training run to investigate how this might have harmed CoT informativeness:
“Specifically, our training-pipeline audit identified a small number of early RL runs where an auxiliary reward designed to penalize reward-hacking behavior was computed over both the CoT and final outputs, rather than outputs alone. All such reward configurations were removed upon identification. Because those runs represented only a very small fraction of Gemini 4 Argon’s overall RL training, we assess that any impact on Argon’s monitorability was minimal.
To estimate the effect of this, we analyzed an existing small training run, not part of any production model, where we estimate the reward of concern had significantly more impact on training than it did in Gemini 4 Argon. Here we observed a substantial increase in unverbalized reward hacking. Solution characteristics associated with reward hacking increased by a relative 26% over the course of training, while the frequency of hacking-related terminology went down by relative 61%. Through these effects combined, at the beginning of training only 11.2% of reward hacks happened silently, but at the end of training this rate of unverbalized hacking had risen to 36.4%. In a comparable run after thought visibility was removed, hack-related verbalizations stayed roughly constant throughout training, giving evidence that the direct thought pressure led to the verbalization decrease.
Even in light of this issue, given the fact that the reward described above made up a dramatically smaller fraction of Gemini 4 Argon’s training signal than in the test run whose empirical statistics were described above, and given the overall transparency assessment in the remainder of this section, we find that the model is sufficiently monitorable to release.” (pp. 114-115)
We appreciate that GDM mentioned this accident and that they tried to estimate the magnitude of the effect. However, the facts they’ve communicated are not particularly reassuring. In GDM’s smaller-scale experiment, unverbalized reward hacking increased dramatically when training against the problematic grader, while in a comparable run against a grader without access to CoT, verbalizations about hacking “stayed roughly constant”. This suggests that the problematic grader does, in fact, apply substantial pressure against verbalizing misalignment in the CoT.
GDM claims this experiment was conservative, but their only stated justification is that the fraction of samples in Argon’s training that contained the problematic grader is much smaller than the fraction of samples with CoT pressure in their experiment. But this probably isn’t the right metric; the number of training steps against the problematic grader might be a more relevant factor (h/t Tim Hua), and GDM doesn’t report how the number of RL rollouts with the auxiliary reward compares to the number of rollouts in their small training run. Once a model has been trained on enough rollouts to learn to control its CoT or suppress verbalized misalignment, continuing to train using a patched grader without CoT access likely won’t give the model any training pressure to unlearn this ability.
Moreover, even if this experiment was indeed conservative (for reasons not stated in the report), that doesn’t directly imply that a fairer experiment would have shown negligible degradation in monitorability. If the accidental auxiliary reward didn’t actually harm Argon’s monitorability, GDM might be able to make a more compelling case by directly measuring unverbalized reward hacking throughout Argon’s training.
Carbon’s training involved additional mistakes
GDM also searched for graders that might apply unintended selection pressure to Carbon’s CoT. We can’t tell whether the following paragraph is intended to imply the problematic grader involved in Argon’s training also affected Carbon; GDM doesn’t explicitly clarify one way or the other:
“We audited the reinforcement learning (RL) training and reward configurations newly introduced since Argon for mechanisms that could pressure Gemini 4 Carbon’s chain of thought to become less legible or monitorable (noting that issues from prior reports can still carry forward when training builds on earlier checkpoints).” (Carbon p. 42)
The Carbon report discloses two new “minor cases in Gemini 4 Carbon’s RL training where thoughts were not fully stripped before reward scoring” (Carbon p. 43). We think GDM makes a fairly convincing case that the first had a negligible effect on Carbon’s CoT due to occurring rarely, typically on trajectories that got low reward anyway, and having no obvious relationship to verbalized misalignment. However, GDM doesn’t give many details about the second case:
“Second, during a small portion of Carbon’s RL training, an auxiliary grading model inadvertently received part of the model’s thoughts before the bug was fixed. This affected only a very small fraction of Carbon’s overall RL training, and a matched control re-run with thoughts masked showed no loss in CoT legibility” (Carbon p. 43)
As we’ve explained previously, making up “a very small fraction” of overall training isn’t sufficient to rule out significant effects on a model’s behavior. The “legibility” metric is measured using an autograder to report how well a human could follow the reasoning. A CoT can be fully “legible” while leaving misbehavior unmentioned; and in any case this eval measures average-case behavior rather than worst-case capabilities. GDM doesn’t clarify what sort of CoT content the auxiliary grading model might have preferred or penalized. We don’t think the provided details are sufficient to rule out a meaningful impact on CoT monitorability.
The Argon report made us concerned about possible intentional process supervision; the Carbon report provides some clarification
The Argon report alluded vaguely to something resembling process supervision: “Some components of the RL training procedure do observe thoughts as context to up-weight reasoning that is on-track to good final outputs” (p. 114). Details were extremely sparse, and we were concerned enough to write a separate post about it. The Carbon report mentions a similar type of training3, but explains the mechanism more clearly: “RL rewards are intended to evaluate outputs rather than thought content; however, learned components used to assign token-level credit for those output rewards observe thoughts as context” (Carbon p. 42). We now think this is probably ordinary credit assignment and not a judge grading the content of thoughts, though GDM concedes that errors in the learned credit model “could still exert direct optimization pressure on thought content”. We’re unhappy that it took a second report to learn this much, and not confidently reassured.
A compelling safety case would look very different from this safety report
In addition to the issues with specific arguments that we’ve analyzed so far, we think that GDM’s safety report falls short of making a compelling safety case for more general methodological reasons.
GDM doesn’t synthesize evidence into an overall assessment of risk
GDM assesses their three premises one at a time, but the ultimate level of risk depends on how they all interact, and the report never works that out. For example, GDM argues that flaws with the first premise (CoT monitorability) are compensated by the third premise – the claim that severe harm takes many steps against layered defenses – so an imperfect monitor will still catch an attack somewhere along the way. In the report’s words:
“while monitors may not catch every problematic action, they are highly likely to detect enough of them to prevent severe harm.” (p. 113)
The report’s reasoning is that longer chains of misbehavior “compound both the chance of an irrecoverable execution error and the cumulative probability of detection” (p. 88). How strong this compounding effect is depends on how likely a monitor is to catch each step, and whether each step is monitored at all. Suppose, as their monitorability results imply, that Argon can get one easy sabotage task past a realistic monitor around half the time. The argument that the other two premises can compensate for this then has the following problems:
GDM never explicitly estimates how many “steps” of a similar difficulty level to the side tasks in their control evals a realistic attack would need to involve. As we argued in the monitorability evals section, if GDM’s crux is that realistic attacks would require longer and more difficult chains of misbehavior than in their control evals, they need to actually demonstrate this instead of just asserting it.
The “cumulative probability of detection” only compounds quickly if failures are independent. Consider an environment where persuading monitors that dangerous actions are justified is unusually easy (which is the sort of environment a misaligned model would choose to attack in). Persuasion that works on one step is likely to work on the next, so a ten-step attack getting through isn’t nearly as unlikely as ten coin-flips all coming up heads.
GDM says Argon must bypass monitoring on its first try because failed attempts at subversion would be flagged (p. 82), but GDM doesn’t say what would stop Argon from trying again later after it gets caught once. The zero-shot argument only works if GDM plans to pause deployment, carefully study and remove the root causes of misbehavior in Argon, substantially harden their security, or take similarly expensive defensive actions after catching failed attempts at subversion.
Coverage gaps are a huge problem for defenses that rely on monitorability. Attacks in an unmonitored harness have no chance of being detected by monitors, and an attack whose first step is to bypass monitoring would make it harder to detect further escalation.
An ideal safety case based on control would involve more end-to-end red-teaming that takes all of their defenses into account. This would require analyzing all of the end states GDM is concerned about, such as a rogue internal deployment or a critical backdoor in monitoring infrastructure, and stating for each one how many defenses would need to be bypassed, how likely each defense is to allow an attack through, and what would happen after a failed attempt to bypass any defenses is caught. Without that, we can’t tell how GDM gets from their provided evidence to “highly unlikely”.
GDM doesn’t say how they chose their evidence or what would have changed their conclusion
AI risk assessment is a nascent science, and there isn’t any widely-accepted methodology or standards for how to do it. This means that it’s crucial to have truth-tracking practices when conducting risk assessment. We have three main suggestions for how to go about doing risk assessment with better epistemic practices:
Suggestion 1: Beware of selection pressure.
We think it’s essential to avoid selection bias when deciding which evals to run or include in a report. That means experiments should be included even if the results were unfavorable. For instance, GDM mentions:
“Note that Guan et al. (2025) introduced a set of evaluations that can also be used to assess how informative the CoT of a model is by default. We ran a subset of them but found them to be too simplistic, and also could not exclude the risk of training data contamination. Hence, we decided to not report results for them and instead measure the default informativeness via our autorater approach.” (p. 137).
It’s commendable that GDM disclosed the omission. However, a cynical reader might suspect that GDM chose to omit these monitorability evaluations because they reflected unfavorably on Argon’s monitorability. If GDM truly had good reasons to suspect this eval wasn’t going to be informative, it would have been safer to come to that conclusion and throw out the eval before knowing what the results would be.
The report also doesn’t say how GDM decided which monitorability evals to run; for instance, the report doesn’t say whether the decision to include the Honeypots setting was made before or after seeing the poor monitorability scores on LinuxArena. Since monitors might have different levels of effectiveness at different tasks, if you resample the setting when you get a poor result, you would predictably find a setting that is particularly easy to monitor for. However, a misaligned model would in practice choose to misbehave in the least monitorable conditions.
In order for a safety report to give compelling evidence of safety, we need to understand what selection pressures went into deciding which results to present.
Suggestion 2: Make it more clear how GDM’s actions depend on their results.
GDM assures us that Argon is sufficiently monitorable to release despite the fact that it can evade realistic monitors on LinuxArena around half the time, and the majority of the time on a control setting closer to GDM’s real infrastructure. It’s hard to tell whether the bottom line would have been any different had monitor evasion in those settings instead been 90%, or 99%. If arbitrarily concerning results on those benchmarks could have been excused by arguments about their lack of realism, then they can’t be used to argue that risk is low.
Arguments about the risk of deploying AI models would be substantially stronger if AI developers were more clear about how they’d act differently given different benchmark results, so that readers can be more confident the conclusion wasn’t predetermined in advance of running any experiments.
Suggestion 3: Don’t count “we never observed it” as evidence without demonstrating that the detector would have caught it.
Several of GDM’s reassurances take the form “we looked for X and didn’t find it”, where the method they used to search for X could be unreliable if X was actually occurring. Some concrete examples:
First, GDM reports “no verbalized evidence” that Argon sandbagged on their control evals (p. 131). But a model which can control its CoT well enough to frequently evade CoT monitors might be able to underperform without saying so, in which case the evasion rates above could be underestimates. This isn’t a purely theoretical concern; OpenAI recently identified covert underperformance in Astra.
Second, GDM found no attempts to tamper with external monitoring or logging, but that search covered only the RL rollouts that monitors had already flagged (p. 98). Tampering that worked would not have been flagged.
Third, GDM says that without prompting Argon “rarely reasons about whether its chain of thought is being monitored”, and that its spontaneous attempts to modify its thoughts are “much weaker than those elicited in our dedicated monitorability stress tests” (p. 84). But these observations come from reading the CoT, so they can only surface attempts clumsy enough to be visible there.
One defense of these claims is that subtlety is continuous: if Argon sometimes evaded detection skillfully, we might also expect to see attempts of intermediate skill, which the monitors would catch. If that were true, and all of the attempts that GDM caught were substantially more clumsy than the most skillful attempt they’d expect monitors to catch, then the lack of any intermediate attempts is evidence against highly subtle ones.
However, this argument might fail if Argon’s subtlety was bimodal. GDM’s own results show a large gap in how well Argon hides its reasoning depending on whether it is trying to: prompted to evade a combined monitor, it succeeds about half the time on LinuxArena, but when prompted to avoid only an action-only monitor, a CoT monitor catches about 90% of its successful attacks (Figure 3.25).
In general, a claim that some behavior was never observed must be defended with evidence that the detector would have caught the behavior had it been present. GDM did this for their alignment-faking autorater, splicing disguised examples into real rollouts to check that the judge flagged them (p. 101). We’d like to see the same for the claims above: plant subtle monitor evasion, silent sandbagging and log tampering in real transcripts, and report how often the monitors and audits catch them.
Conclusion
We’ve argued that the Gemini 4 safety reports don’t provide strong evidence that Argon or Carbon would be unable to severely subvert GDM’s control measures if they tried. We think that given how capable frontier models are becoming and how poor the current state of alignment science is, making a strong safety case would require implementing very robust control measures and red-teaming them substantially more carefully than these reports describe. Here are some things GDM could do to make future safety reports more informative:
Be more transparent about what methodology was used to decide which experiments to run and which results to include in the report.
Make it more clear what results would have changed GDM’s conclusion about the overall level of risk; make it harder to explain away arbitrarily concerning results.
Provide more details about to what extent their training practices damage CoT informativeness, for example by running and reporting more careful experiments studying the effect of accidental training against CoT.
Provide detailed reports of misalignment incidents, like OpenAI does. This would show what kinds of misaligned actions GDM detects its models taking in practice, and how often4.
Work out how the premises of the control argument combine into an overall assessment of risk, ideally with end-to-end red-teaming: for each outcome GDM is worried about, how many defenses an attack must get past, how likely each is to fail, and what GDM does after catching a failed attempt.
Evaluate whether models can convince monitors to not flag misaligned actions with rationalization, and provide stronger evidence that monitors are not prone to collusion.
Run control evals against production monitors, or report how the false positive rates compare, and report how evasion rates change as side tasks get harder.
Back up claims that a behavior was never observed by planting examples of it and measuring whether the monitors and audits would have caught them.
Rerun monitorability/control evals for all new models that will be extensively deployed.
Following this list of suggestions isn’t on its own sufficient to make a good safety case – for one, we expect that a more careful analysis of the existing safety measures wouldn’t actually conclude that risk is negligible.
GDM asserts that they have performed “rigorous safety and security evaluations designed to ensure the model is safe”. If arguments like this are used to justify developing or deploying substantially more powerful models in the (near) future, we think developers would be incurring substantial loss-of-control risk.The Carbon report gives monitor alert rates for RL training and internal coding deployment, along with a few high-level descriptions of the flagged behavior (Carbon pp. 35-36), but more detailed incident descriptions would be ideal.
Quotes and page numbers refer to the Argon report unless marked “Carbon”. GDM assessed that Carbon showed no material capability change relative to Argon, so it “did not repeat that holistic assessment in full” (Carbon p. 29) and carried forward Argon’s monitor-evasion and CoT-controllability results (Carbon p. 42).
In particular, since models typically aren’t explicitly trained to modify their CoTs when prompted to do so, it can be difficult to assess whether they would be able to intentionally control them (e.g. to hide sneaky misbehavior) if they were actually motivated to do so. Since Argon apparently does control its CoT pretty competently when prompted to do so, severe under-elicitation is less of a concern, though it would be hard to rule out intentional sandbagging.
This is almost certainly referring to the same mechanism used in Argon’s training, although GDM technically doesn’t confirm this.
The Carbon report gives monitor alert rates for RL training and internal coding deployment, along with a few high-level descriptions of the flagged behavior (Carbon pp. 35-36), but more detailed incident descriptions would be ideal.
来源:Redwood Research:Blog · blog.redwoodresearch.org