Transformer 能同时容纳两种思路:LLM 中线性叠加的证据

HuggingFace Daily Papers(社区热门论文)·2026-09-24 08:00·1天前
AI 导读

研究提出「叠加线性假设」,证明 Transformer 对来自不同文本流的输入做线性组合时,输出是各自下一 token 分布的叠加,且该性质是架构固有而非训练涌现,并随预训练推进而减弱。轻量微调可显著恢复这种线性,缩小预测分布与各分布均值的偏差。团队还提出引导式解码,从单次前向传播中解耦叠加输出,同时生成两条连贯续写。

HuggingFace Daily Papers(社区热门论文)
44AI 编辑部评分,满分 100

Transformer 能同时容纳两种思路:LLM 中线性叠加的证据

2026-09-24 08:00· 1天前
AI 导读

研究提出「叠加线性假设」,证明 Transformer 对来自不同文本流的输入做线性组合时,输出是各自下一 token 分布的叠加,且该性质是架构固有而非训练涌现,并随预训练推进而减弱。轻量微调可显著恢复这种线性,缩小预测分布与各分布均值的偏差。团队还提出引导式解码,从单次前向传播中解耦叠加输出,同时生成两条连贯续写。

While Large Language Models (LLMs) rely on highly non-linear components, in this work we demonstrate that they exhibit fundamental linearity: when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions. We term this the Superposition Linearity Hypothesis. We provide evidence that superposition is an intrinsic property of the Transformer architecture rather than an emergent consequence of training; in fact, we observe that it tends to diminish as pretraining progresses. However, we demonstrate that linearity can be substantially restored through lightweight fine-tuning, significantly reducing the divergence between the predicted next-token distribution and the average of the individual next-token distributions. Finally, we introduce a guided decoding procedure that disentangles superposed outputs, enabling the simultaneous generation of two coherent continuations from a single forward pass.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org