RecSA 在 KV cache 上构建树结构并让每个查询逐层搜索,使单次查询成本降至 O(log L),仅在 32K 训练即可外推到 2M token。基于 Qwen3-4B、用 0.62B token 在 32K 下训练后,它在 RULER 的 131K 至 2M 区间全面超越基线,1M 处得分 60.5,最强基线 Dense + DCA 为 40.0。
RecSA: Million-Token Extrapolation at Logarithmic Cost 🚀
RecSA builds a tree over the KV cache and lets each query search it level by level, so the cost per query grows as O(log L) and a model trained only at 32K extrapolates to 2M tokens.
Initialized from Qwen3-4B and trained on 0.62B tokens, all at 32K, RecSA outperforms all baselines on RULER from 131K to 2M, scoring 60.5 at 1M against 40.0 for the strongest baseline (Dense + DCA). On one H200, it speeds up prefill by 3.2× and decoding by 3.8× at 524K.
Blog: haoailab.com/blogs/recsa/
Paper: arxiv.org/abs/2611.07232
Code: github.com/skywong14/RecSA
来源:Hao AI Lab · x.com
