# MInTRL：离线策略干预如何提升在线策略强化学习

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-11 08:00
- AIHOT 分数：37
- AIHOT 链接：https://aihot.news/items/cmu2e2or806wrro25vuyxusso
- 原文链接：https://arxiv.org/abs/2609.12419

## AI 摘要

研究者提出 Minimal Intervention Reinforcement Learning（MInTRL），在原本 on-policy 的 rollout 中通过稀疏、局部干预扩展探索边界：生成时由 judge-intervention 策略周期性审查当前策略输出，用简短修正替换错误后缀后立即交还控制权。

## 正文

Reinforcement learning with verifiable rewards is typically performed on-policy, keeping training data close to the current policy but limiting learning to trajectories that the policy can discover itself. Off-policy methods such as supervised fine-tuning, on the other hand, can leverage external knowledge beyond the base model's capabilities, but may suffer from large distribution shift. The key challenge is thus to expand exploration without sacrificing learnability. In this work, we introduce Minimal Intervention Reinforcement Learning (MInTRL), which expands the exploration frontier through sparse, local interventions in otherwise on-policy rollouts.

During generation, a judge-intervention policy periodically reviews the current policy's output, replaces erroneous suffixes with short corrections, and immediately returns control to the policy. During training, MInTRL adopts a sequence-level advantage-regression objective that eliminates the need for importance sampling. We show that sparse, local interventions can substantially improve coverage beyond finite-budget on-policy sampling while preserving the overall on-policy nature of the resulting trajectories. Across math and code benchmarks, MInTRL consistently outperforms standard on-policy and off-policy baselines.

Ablations show that MInTRL remains effective with self-intervention and across different judge policies, while performance peaks at moderate intervention intensity, highlighting the importance of intervening minimally. These results establish minimal intervention as an effective paradigm for enhancing on-policy RL.