# PPO 评论家学习再思考：价值平坦化成因与 SP3O 缓解方法

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-16 08:00
- AIHOT 分数：35
- AIHOT 链接：https://aihot.news/items/cmu56wpkg03hcro4oc21uptno
- 原文链接：https://arxiv.org/abs/2609.18708

## AI 摘要

研究揭示 PPO 评论家存在"价值平坦化"失效模式：中间状态的真实价值随 Monte Carlo 延续剧烈变化，评论家预测却相对平坦，且随状态空间增大而加剧。为此提出 SP³O，每条回复仅对少量间隔良好的状态施加价值损失。在 Qwen3-Base 上，每条回复仅监督三个状态即可缓解价值平坦化，并在不同模型规模和评测集上稳定提升策略表现。

## 正文

In reinforcement learning for large language models, Proximal Policy Optimization (PPO) commonly uses a critic to estimate state values and reduce the variance of policy updates. However, we uncover a systematic failure mode in PPO critics, which we call Value Flattening: state values, estimated from multiple Monte Carlo continuations, change sharply across intermediate states while critic predictions remain comparatively flat. We further observe this phenomenon in a controlled FrozenLake environment and find that it becomes more pronounced as the state space grows.

Our theoretical and empirical analyses relate Value Flattening to an implicit variance penalty in the critic loss and redundant updates from temporally correlated states with similar gradients. Motivated by these findings, we introduce SParse Proximal Policy Optimization (SP^3O), which applies the value loss to only a few well-separated states in each response to mitigate both effects. Experiments on Qwen3-Base show that SP^3O with only three states supervised per response can mitigate Value Flattening and consistently improve the learned policy across model sizes and evaluation suites.

Together, our results identify Value Flattening as an important yet overlooked failure mode of critic learning in standard PPO and show that a simple sparse supervision strategy can mitigate it.