Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.
Gated DeltaNet-2:解耦线性注意力中的擦除与写入
AI 导读
线性注意力通过固定循环状态替代无界缓存,但面临精确编辑压缩记忆的挑战。现有模型如Delta-rule与KDA使用单一标量门同时控制“擦除”与“写入”两个操作。本文提出Gated DeltaNet-2,引入独立的通道级擦除门和写入门,实现了这两个操作的解耦,从而泛化并改进了前代模型。该模型在1.3B参数规模、100B tokens训练下,在语言建模、常识推理等任务中表现优异,尤其在长上下文RULER多键检索基准上优势显著。
HuggingFace Daily Papers(社区热门论文)
精选
70
AI 编辑部评分,满分 100Gated DeltaNet-2:解耦线性注意力中的擦除与写入
线性注意力通过固定循环状态替代无界缓存,但面临精确编辑压缩记忆的挑战。现有模型如Delta-rule与KDA使用单一标量门同时控制“擦除”与“写入”两个操作。本文提出Gated DeltaNet-2,引入独立的通道级擦除门和写入门,实现了这两个操作的解耦,从而泛化并改进了前代模型。该模型在1.3B参数规模、100B tokens训练下,在语言建模、常识推理等任务中表现优异,尤其在长上下文RULER多键检索基准上优势显著。
推荐理由
把线性注意力里擦除和写入的解耦,像给模型装了独立刹车和油门,在长上下文检索上直接拉开差距,做架构的值得细读。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org