While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions. We term this the Superposition Linearity Hypothesis. We provide evidence that superposition is an intrinsic property of the Transformer architecture rather than an emergent consequence of training; in fact, we observe that it tends to diminish as pretraining progresses. However, we demonstrate that linearity can be substantially restored through lightweight fine-tuning, significantly reducing the divergence between the predicted next-token distribution and the average of the individual next-token distributions. Finally, we introduce a guided decoding procedure that disentangles superposed outputs, enabling the simultaneous generation of two coherent continuations from a single forward pass.
Transformer 能同时容纳两种思路:LLM 中线性叠加的证据
AI 导读
研究提出「叠加线性假设」,证明 Transformer 对来自不同文本流的输入做线性组合时,输出是各自下一 token 分布的叠加,且该性质是架构固有而非训练涌现,并随预训练推进而减弱。轻量微调可显著恢复这种线性,缩小预测分布与各分布均值的偏差。团队还提出引导式解码,从单次前向传播中解耦叠加输出,同时生成两条连贯续写。
HuggingFace Daily Papers(社区热门论文)
44
AI 编辑部评分,满分 100Transformer 能同时容纳两种思路:LLM 中线性叠加的证据
研究提出「叠加线性假设」,证明 Transformer 对来自不同文本流的输入做线性组合时,输出是各自下一 token 分布的叠加,且该性质是架构固有而非训练涌现,并随预训练推进而减弱。轻量微调可显著恢复这种线性,缩小预测分布与各分布均值的偏差。团队还提出引导式解码,从单次前向传播中解耦叠加输出,同时生成两条连贯续写。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org