# 1% 的 Token 就够了：论在线策略蒸馏中的梯度估计

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-21 08:00
- AIHOT 分数：38
- AIHOT 链接：https://aihot.news/items/cmuca34990j2erots17aba8r0
- 原文链接：https://arxiv.org/abs/2609.24432

## AI 摘要

研究发现在线策略蒸馏（OPD）中，仅需 0.1%–1% 的 token 预算即可匹配甚至超越全量 OPD。作者从信息几何角度提出信息效率比（IER），基于信噪比分解刻画最优标量基线下的相对梯度估计误差，并据此进行 token 选择。在数学与医学推理任务上，将 IER 与现有有用性评分结合可改进多种设置下的选择器，同时保留采样的反向 KL 训练目标。

## 正文

Sparse on-policy distillation (OPD) allocates teacher supervision to a small subset of tokens in student-generated trajectories. However, useful teacher guidance can yield a noisy update when its gradient is estimated from a sampled next token. We study this estimation problem at a fixed prefix in information geometry and propose an information-efficiency ratio (IER) based on a signal-to-noise decomposition. IER characterizes relative gradient estimation error under an optimal scalar baseline. A candidate-set approximation enables token selection based on IER and its combination with existing usefulness scores, while retaining the sampled reverse-KL training objective. On mathematical and medical reasoning tasks, adding IER improves existing selectors in multiple settings, with sparse configurations matching or exceeding full OPD without token selection at small token budgets of 0.1%--1%. These results support accounting for both usefulness and gradient-estimation reliability when allocating sparse supervision. Our code is available at https://github.com/BruceSheng1202/IER-OPD.