Ling-2.6-1T-base 是 Ling-2.6-1T 和 Ring-2.6-1T 背后的基础检查点。它是一个万亿参数的混合专家（MoE）语言模型，由 Ling-2.0-1T-base 改造而来，采用了混合线性注意力设计，并经过持续预训练和长上下文中期训练。
本次发布面向研究、持续预训练、知识蒸馏以及监督式或基于偏好的微调。它并非对话对齐的助手模型。如果你想要开箱即用的指令模型或推理模型，请改用相应的 Ling-2.6 或 Ring-2.6 后训练检查点。
Ling-2.6-1T-base 的设计目标是在保留 Ling-2.0 万亿级主干能力的同时，让长上下文训练和推理显著更高效。核心升级是混合注意力改造，将 Lightning Attention 与 MLA 以 7:1 的比例结合，并配套一条从原始基于 GQA 架构平滑迁移的流水线。
根据技术报告，该模型在迁移预训练、持续预训练和中期训练中总共训练了约 9.6T tokens，上下文长度分阶段从 4K 扩展到 256K。同一个基础检查点随后被专门化为：
Ling-2.6，用于即时、token 高效的响应
Ring-2.6，用于更深层推理和长周期智能体工作流
混合线性注意力架构，将 Lightning Attention 与 MLA 以 7:1 的比例结合
万亿参数 MoE 主干网络从 Ling-2.0-1T-base 升级而来，而非从头重新训练
长上下文训练流程在中期训练阶段扩展至 256K 上下文
持续预训练数据混合覆盖智能体数据、长上下文数据、知识密集型网络数据、数学、代码及多语言语料
基础模型在知识、数学、代码、推理及长上下文理解等基准测试中均展现出强劲质量
- 模型概要
项目 值 架构 细粒度 MoE 搭配混合线性注意力 参数规模 总计约 1T，激活约 63B Transformer 层数 80 注意力头数 64 隐藏层维度 8192 每 MoE 层路由专家数 256 每 MoE 层共享专家数 1 每 token 激活路由专家数 8 稠密 FFN 层 前 4 个 Transformer 块 专家中间层维度 2048 稠密中间层维度 18432 词表大小 157,184 位置编码 部分 RoPE 注意力设计 Lightning Attention + MLA，比例 7:1 训练方案 迁移预训练 + 持续预训练 + 中期训练 总训练 token 数 约 9.6T 上下文训练计划 4K -> 32K -> 256K
- 训练亮点
架构迁移
该模型以 Ling-2.0-1T-base 为起点，通过多阶段迁移流程转换为 Ling-2.6-1T 架构，该流程包括：
Lightning Attention 转换
线性预热
MLA 转换
MLA 预热
完整继续预训练
该改造方案旨在保留预训练能力，同时降低长上下文计算成本和 KV-cache 压力。
数据混合
继续预训练和中期训练阶段包括：
由工具使用和编程环境构建的智能体语料库
涵盖数学、网页解析、摘要、检索和多跳推理的长上下文语料库
带有针对性 STEM 和事实增强的通用网页知识数据
数学和代码语料库
覆盖 21 种语言的多语言数据
以下数据选自技术报告，反映的是基础模型评测结果，而非聊天对齐或指令微调后的表现。
基准测试 Ling-2.0-1T-base Ling-2.6-1T-base MMLU 86.03 86.82 MMLU-Pro 67.91 67.79 GPQA 41.92 45.45 SimpleQA 20.87 38.26 C-SimpleQA 64.53 76.83 MMMLU 68.68 71.53 GSM8K 89.31 93.93 OmniMath 33.60 38.70 HumanEval-Plus 83.54 85.98 LiveCodeBench 40.09 44.27 BIRD-SQL 42.70 44.59 BBH 86.88 89.73 AutoLogic 65.76 67.43 LEval 72.30 76.21 LongBenchv2 30.02 43.54
在技术报告中，Ling-2.6-1T-base 相较 Ling-2.0-1T-base 展现出广泛提升，尤其是在事实知识、多语言知识覆盖、长上下文理解和推理导向评测方面，同时保持或提升了强劲的数学与代码能力。在这一精选子集中，一个值得注意的例外是 MMLU-Pro，Ling-2.0-1T-base 在该项上仍略高。
Ling-2.6-1T-base is the base checkpoint behind the Ling-2.6-1T and Ring-2.6-1T. It is a trillion-parameter Mixture-of-Experts language model retrofitted from Ling-2.0-1T-base with a hybrid linear attention design, continued pre-training, and long-context mid-training.
This release is intended for research, continued pre-training, distillation, and supervised or preference-based fine-tuning. It is not a chat-aligned assistant model. If you want an out-of-the-box instruction or reasoning model, use the corresponding Ling-2.6 or Ring-2.6 post-trained checkpoints instead.
Ling-2.6-1T-base is designed to preserve the capability of the Ling-2.0 trillion-scale backbone while making long-context training and inference materially more efficient. The core upgrade is a hybrid attention retrofit that combines Lightning Attention with MLA in a 7:1 ratio, together with a smooth migration pipeline from the original GQA-based architecture.
According to the technical report, the model is trained through approximately 9.6T tokens across migration pre-training, continued pre-training, and mid-training, with staged context extension from 4K to 256K. The same base checkpoint is later specialized into:
Ling-2.6 for instant, token-efficient response
Ring-2.6 for deeper reasoning and long-horizon agentic workflows
Hybrid linear attention architecture combining Lightning Attention and MLA in a 7:1 ratio
Trillion-parameter MoE backbone upgraded from Ling-2.0-1T-base instead of retraining from scratch
Long-context training pipeline extended to 256K context during mid-training
Continued pre-training mixture covering agentic data, long-context data, knowledge-rich web data, math, code, and multilingual corpora
Strong base-model quality across knowledge, math, code, reasoning, and long-context understanding benchmarks
- Model Summary
Item Value Architecture Fine-grained MoE with hybrid linear attention Parameter Scale Totoal ~1T, Activated ~63B Transformer layers 80 Attention heads 64 Hidden size 8192 Routed experts per MoE layer 256 Shared experts per MoE layer 1 Active routed experts per token 8 Dense FFN layers First 4 transformer blocks Expert intermediate size 2048 Dense intermediate size 18432 Vocabulary size 157,184 Positional encoding Partial RoPE Attention design Lightning Attention + MLA, 7:1 ratio Training recipe Migration pre-training + continued pre-training + mid-training Total training tokens ~9.6T Context training schedule 4K -> 32K -> 256K
- Training Highlights
Architecture Migration
The model starts from Ling-2.0-1T-base and is converted into the Ling-2.6-1T architecture through a multi-stage migration pipeline that includes:
Lightning Attention conversion
Linear warmup
MLA conversion
MLA warmup
Full continued pre-training
This retrofit is designed to preserve pre-trained capability while reducing long-context compute cost and KV-cache pressure.
Data Mixture
The continued pre-training and mid-training stages include:
Agentic corpus built from tool-use and coding environments
Long-context corpus covering mathematics, web parsing, summarization, retrieval, and multi-hop reasoning
General web knowledge data with targeted STEM and factual augmentation
Math and code corpora
Multilingual data spanning 21 languages
The following numbers are selected from the technical report and reflect base-model evaluation rather than chat-aligned or instruction-tuned performance.
Benchmark Ling-2.0-1T-base Ling-2.6-1T-base MMLU 86.03 86.82 MMLU-Pro 67.91 67.79 GPQA 41.92 45.45 SimpleQA 20.87 38.26 C-SimpleQA 64.53 76.83 MMMLU 68.68 71.53 GSM8K 89.31 93.93 OmniMath 33.60 38.70 HumanEval-Plus 83.54 85.98 LiveCodeBench 40.09 44.27 BIRD-SQL 42.70 44.59 BBH 86.88 89.73 AutoLogic 65.76 67.43 LEval 72.30 76.21 LongBenchv2 30.02 43.54
In the technical report, Ling-2.6-1T-base shows broad gains over Ling-2.0-1T-base, especially on factual knowledge, multilingual knowledge coverage, long-context understanding, and reasoning-oriented evaluations, while preserving or improving strong math and code capability. One notable exception in this selected subset is MMLU-Pro, where Ling-2.0-1T-base remains slightly higher.