# DATPO：难度自适应树结构策略优化，提升 RLVR 推理覆盖

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-09-08 08:00
- AIHOT 分数：39
- AIHOT 链接：https://aihot.news/items/cmtuxuyrr0igerorpi0h454ie
- 原文链接：https://arxiv.org/abs/2609.08650

## AI 摘要

针对 RLVR 训练中 pass@k 难以提升的问题，研究者提出 DATPO（难度自适应、句子熵引导的树结构策略优化），将难度自适应树搜索与兄弟多样性优势项结合，显式提升语义多样性。在数学推理基准上，DATPO 在 pass@k 上优于基线，并直接转化为更好的测试时扩展性能。

## 正文

Reinforcement Learning with Verifiable Rewards (RLVR) has been central to the recent success of Large Reasoning Models. However, while RLVR significantly improves single-sample accuracy, it often fails to expand the model's intrinsic reasoning coverage (pass@k) due to limited exploration during training. To address this, we optimize the structural design of train-time rollouts to enhance pass@k. Our analysis identifies three key design principles: (1) difficulty-adaptive rollout can play an important role in expanding pass@k, beyond serving as an efficiency heuristic; (2) tree-based rollout outperforms parallel sampling in discovering correct answers; and (3) sentence-entropy-guided forking overcomes the localization phenomenon of token-level branching to maximize semantic diversity. Building on these insights, we propose DATPO (Difficulty-Adaptive Sentence-entropy-guided Tree-structured Policy Optimization). DATPO integrates difficulty-adaptive tree search with a sibling-diversity advantage term, explicitly promoting semantic diversity to expand reasoning coverage during training. Experiments on mathematical reasoning benchmarks demonstrate that DATPO outperforms baselines especially in pass@k, which directly translates to superior test-time scaling performance.