Recently, linear attention layers have been increasingly adopted to replace softmax attention at scale for long-context modeling. However, existing context extension approaches typically apply continued pretraining directly without modifying these layers, overlooking the spectral properties of linear attention state dynamics. In this work, we study long-context extension of Gated DeltaNet (GDN) from a spectral perspective of transition matrix and identify two essential factors governing long-range information retrieval: (1) a sufficiently broad slow spectral band aligned with the target dependency length, and (2) the preservation of fast-decaying modes for state clearing and context switching. Based on this observation, we propose SpectralShift, a spectral reparameterization approach for long-context continual pretraining of GDNs. Specifically, SpectralShift reparameterizes the alpha projections initialization to reshape the decay spectrum by enhancing slow propagation capacity, and further introduces a learning-rate scaling for alpha projections to facilitate long-context training. Experiments show that SpectralShift consistently improves long-context capabilities over training, providing an effective and efficient solution for extending context windows of linear attention models. The code has been open-sourced at https://github.com/RUCAIBox/GDN-SpectralShift.
SpectralShift:通过谱重参数化扩展 Gated DeltaNet 有效上下文窗口
AI 导读
SpectralShift 通过谱重参数化扩展 Gated DeltaNet(GDN)的有效上下文窗口,用于长上下文持续预训练。该方法重新参数化 alpha 投影初始化以重塑衰减谱、增强慢速传播能力,并引入 alpha 投影的学习率缩放。实验显示其持续提升长上下文能力,代码已开源。
HuggingFace Daily Papers(社区热门论文)
41
AI 编辑部评分,满分 100SpectralShift:通过谱重参数化扩展 Gated DeltaNet 有效上下文窗口
SpectralShift 通过谱重参数化扩展 Gated DeltaNet(GDN)的有效上下文窗口,用于长上下文持续预训练。该方法重新参数化 alpha 投影初始化以重塑衰减谱、增强慢速传播能力,并引入 alpha 投影的学习率缩放。实验显示其持续提升长上下文能力,代码已开源。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org