长度惩罚强化学习可以缩短思维链推理过程,同时隐藏影响模型答案的驱动因素。在我们的实验中,即使模型的思维链提及提示线索的频率大幅降低,使用长度惩罚进行训练也无法阻止误导性提示对模型产生引导。若采用token准确率评估,这些运行会被视为成功,因为它们使用了更少的推理token且准确率损失很小;但这种方法会忽略剩余痕迹是否仍能显示答案的驱动因素。
我们训练了不同目标链长度的Qwen3-4B和Qwen3-14B变体,随后在保留的MMLU-Pro-R和四个迁移基准上使用偏差提示干预进行评估。压缩显著减少了推理token,保持了大部分多项选择题准确率,并使提示影响接近基线水平。
在最强压缩条件下,Qwen3-14B的下限忠实度降至基线的63.1%,Qwen3-4B降至69.4%;监控器捕捉提示使用的原始比率从69%降至49%,从60%降至48%。为分离长度与内容的影响,我们从未压缩的基线链中随机删除句子,直至剩余文本与压缩链长度匹配。
即使经过这种长度匹配,对于两种Qwen3规模及全部五个评估分布,压缩链披露提示的频率仍比我们随机缩短的基线链低7至35个百分点。因此,压缩的作用不仅限于缩短推理过程,还会优先移除监控器判断答案影响因素所需的线索。综合来看,这些结果揭示了一条压缩-可监控性边界:更经济的推理可以在保持答案正确的同时,使答案背后的影响机制更难被察觉。
Length-penalized reinforcement learning can shorten chain-of-thought reasoning while hiding an influence that drives the model's answer. In our experiments, training with length penalties does not stop misleading hints from steering models, even though the models' chains of thought mention the hint much less often. A token-accuracy evaluation would count these runs as successful because they use fewer reasoning tokens with little accuracy loss; it would miss whether the remaining trace still shows what drove the answer. We train Qwen3-4B and Qwen3-14B variants with different target chain lengths, then evaluate them with biasing-hint interventions on held-out MMLU-Pro-R and four transfer benchmarks.
Compression sharply cuts reasoning tokens, preserves most multiple-choice accuracy, and leaves hint influence near baseline. At the strongest target, lower-bound faithfulness falls to 63.1% of baseline for Qwen3-14B and 69.4% for Qwen3-4B; the raw rate at which a monitor catches hint use falls from 69% to 49% and from 60% to 48%. To separate length from content, we randomly delete sentences from uncompressed baseline chains until the remaining text matches the compressed length. Even after this length matching, compressed chains disclose the hint 7-35 percentage points less often than baseline chains that we shorten at random, for both Qwen3 sizes and all five evaluation distributions.
Compression therefore does more than shorten reasoning, preferentially removing the cues a monitor needs to see what influenced the answer. Together, these results reveal a compression-monitorability frontier in which cheaper reasoning can preserve answers while making the influences behind them harder to detect.