# 面向 LLM 的全流程 FP8 强化学习：Calibrated Clipping 消除训练不稳定

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-19 08:00
- AIHOT 分数：39
- AIHOT 链接：https://aihot.news/items/cmud6bt0v03u7rorafrlmtel2
- 原文链接：https://arxiv.org/abs/2609.22870

## AI 摘要

针对全流程 FP8 强化学习仍存在训练不稳定、中期熵激增和输出乱码的问题，研究揭示其根源是 FP8 量化噪声叠加扭曲重要性比值，使负优势 token 被错误推出信任域并梯度归零。

## 正文

Reinforcement learning (RL) has become a key technique for improving the reasoning and agentic abilities of large language models (LLMs). Although FP8 quantization can accelerate RL training, maintaining stability throughout an FP8 RL pipeline remains challenging. While previous works have focused on resolving train-inference mismatches using correction techniques like TIS, we reveal that full-pipeline FP8 RL still suffers from severe training instability, manifesting as anomalous mid-training entropy surges and garbled outputs. We trace this instability to a previously overlooked cause: compounded FP8 quantization noise distorts the importance ratio, disproportionately pushing negative-advantage tokens outside the trust region and erroneously zeroing out their gradients.

As a result, pathological outputs are not properly penalized and accumulate over the course of training. To address this, we propose Calibrated Clipping, a dynamic method that aligns the FP8 clipping bounds with high-precision BF16 distributions by matching the lower-bound clipping quantile and rebalancing the upper bound accordingly. Extensive experiments across GRPO and DAPO algorithms, model scales from 8B to 32B, and multiple FP8 scaling granularities demonstrate that our approach successfully eliminates entropy surges and restores performance comparable to the BF16 baseline.