跳到正文
OpenBMB· @OpenBMB · X·· 2 小时前AI 评分46
AI 导读

面壁智能(OpenBMB)发布长上下文性能的新经验缩放定律,对 log(LongPPL) 建模为 N、Ds、Dl 的加性幂律,在留出规模上外推 R²=0.97。配套的 Constant-Last4 协议可在单条可扩展训练轨迹上稳定监控,收益可从 32K 迁移到 128K,并在 SFT 后及长上下文与智能体任务中保持。

正文 · 原文

Long-context training: don't stop too early 📈

New empirical scaling law on long-context performance:

  • Scaing Law on log(LongPPL) ≈ f(N, Ds, Dl): additive power law, extrapolates to held-out scales (R²=0.97)
  • Constant-Last4 protocol: stable monitoring on one extensible training trajectory
  • Gains transfer: 32K → 128K, after SFT, and in long-context and agentic tasks

📄 arxiv.org/abs/2610.29352
💻 github.com/thunlp/dont-stop-l…
🤗 huggingface.co/collections/Ed…

#LLM #LongContext #ScalingLaw #ContinualPretraining #AI #Agentic

来源:OpenBMB · x.com