可解释性常被视为能力的“代价”：语言模型被训练成不透明系统，事后再用可靠性难以验证的方法加以解释。在这项工作中，我们挑战了这一前提。我们不去逆向解析模型，而是把可解释性作为训练流程中的一项约束，与语言建模目标一同优化。在跨越三个数量级的算力规模上，无论是自回归还是扩散语言模型，可解释性都随能力同步提升，而非与之相悖。
令人意外的是，模型表征随规模扩大而变得更加解耦，并与人类可理解的概念更加对齐。我们用 Steerling-8B 实例化了这一训练时方案——这是一个带有因果注意力掩码的扩散语言模型。对于任意一组生成 token，Steerling-8B 都能将输出归因到相关输入 token、人类可理解的概念以及训练数据。
这实现了闭环干预：通过概念或特征归因诊断输出，检索相似训练数据，并通过概念引导修正行为，而无需重新训练。Steerling-8B 在竞争力上与使用 2-16 倍算力训练的开源同类模型相当，这暗示了一种不同的扩展范式：可解释性可以被设计进训练过程，并随规模扩大而持续改善。
Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it.
Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining.
Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale.