# RetireOPD：面向智能体强化学习的自退役同策略蒸馏

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-17 08:00
- AIHOT 分数：40
- AIHOT 链接：https://aihot.news/items/cmu6ds2t00kq6rofj6jszgyfh
- 原文链接：https://arxiv.org/abs/2609.20784

## AI 摘要

RetireOPD 提出自适应退役机制的同策略蒸馏方法：先用环境奖励训练技能条件教师，再让无技能学生联合 RL 与 OPD 训练，当师生差距不再缩小且学生达到教师成功率目标比例时自动弃用教师、仅用 RL 继续训练。

## 正文

Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD.

Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.