# 并非所有提示词都等价：面向多模态强化学习后训练的探索引导提示词脚手架

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-14 08:00
- AIHOT 分数：43
- AIHOT 链接：https://aihot.news/items/cmu23cu0407efrow29kbt6eil
- 原文链接：https://arxiv.org/abs/2609.15051

## AI 摘要

研究者提出探索引导的提示词脚手架框架，用于多模态大模型（MLLM）的在线强化学习后训练，其核心 Exploration Potential Score（EPS）可直接由 on-policy rollout 统计计算，无需额外开销。

## 正文

Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs). Central to our approach is the Exploration Potential Score (EPS), a lightweight rollout-based proxy for prompt utility derived from KL-regularized policy improvement theory, computable directly from on-policy rollout statistics without additional overhead.

Rather than discarding low-utility prompts, we use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated with GRPO on Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7% relative improvement in-domain and gains of 11.5% on MathVision and 11.1% on MMMU-Pro.