OpenBMB· @OpenBMB · X·· 2 小时前AI 评分46
AI 导读
面壁智能(OpenBMB)发布长上下文性能的新经验缩放定律,对 log(LongPPL) 建模为 N、Ds、Dl 的加性幂律,在留出规模上外推 R²=0.97。配套的 Constant-Last4 协议可在单条可扩展训练轨迹上稳定监控,收益可从 32K 迁移到 128K,并在 SFT 后及长上下文与智能体任务中保持。
正文 · 原文
Long-context training: don't stop too early 📈
New empirical scaling law on long-context performance:
- Scaing Law on log(LongPPL) ≈ f(N, Ds, Dl): additive power law, extrapolates to held-out scales (R²=0.97)
- Constant-Last4 protocol: stable monitoring on one extensible training trajectory
- Gains transfer: 32K → 128K, after SFT, and in long-context and agentic tasks
📄 arxiv.org/abs/2610.29352
💻 github.com/thunlp/dont-stop-l…
🤗 huggingface.co/collections/Ed…
#LLM #LongContext #ScalingLaw #ContinualPretraining #AI #Agentic
来源:OpenBMB · x.com