简介
我们已开源 Ling-3.0 系列，这是我们迄今为止最高效的语言基础模型家族。为支持研究与社区驱动的创新，我们发布训练过程中的一系列检查点，如下所示：
| 模型 | 预训练 | 中训练 | 合并（即 WSM） |
|---|---|---|---|
| Ling-3.0-tiny | Ling-3.0-tiny-base-30T | Ling-3.0-tiny-base-midtrain | Ling-3.0-tiny-base |
| Ling-3.0-flash | Ling-3.0-flash-base-30T | Ling-3.0-flash-base-midtrain | Ling-3.0-flash-base |
这些检查点对应训练过程的不同阶段：
- 预训练检查点已完成大规模预训练，但尚未经过中期训练、WSM 合并（或学习率衰减）或后训练。
- 中期训练检查点已完成中期训练，但尚未经过 WSM 合并（或学习率衰减）或后训练。
- 合并检查点已在中期训练检查点的基础上经过 WSM 合并（或学习率衰减），但尚未经过后训练。
发布这些检查点是为了支持持续预训练、微调以及进一步研究。关于后训练模型，请参见 Ling-3.0-tiny 和 Ling-3.0-flash。
模型概览
核心特性
- 高度稀疏（1/64）MoE 架构：512 个路由专家，每个 token 仅激活 8 个路由专家和 1 个共享专家。这使其具备广泛的模型能力，同时每个 token 仅激活 5.1B（非嵌入）参数；
- 原生混合线性注意力： Ling-3.0 系列从预训练一开始就采用原生混合线性注意力架构，将 KDA 与 Gated MLA 相结合，从而实现对长上下文输入的高效处理。
- 预热-稳定与合并： 我们以加权检查点合并取代了传统的学习率衰减。通过消除衰减阶段，我们的 Base Model 更适合持续预训练和动态数据扩展，同时还能离线探索不同的衰减曲线，而无需为每种策略重新运行昂贵的实验。
- 无缝扩展： Ling-3.0-tiny-base 与 Ling-3.0-flash-base 共享相同的训练配方，使社区能够先在 Ling-3.0-tiny-base 上进行实验，然后将经过验证的训练策略扩展到更大的 Ling-3.0-flash-base。
| 模型类型 | Base（中期训练的最终检查点） |
|---|---|
| 架构 | 混合线性 MoE |
| 参数规模 | 总计 124B，激活 5.1B（非嵌入） |
| Transformer 层数 | 35 KDA + 7 Gated MLA（5:1） |
| 稠密层数量 | 2 |
| 路由专家数量 | 512 |
| 共享专家数量 | 1 |
| 激活专家数量 | 8 |
| 注意力头数 | 32 |
| 隐藏层维度 | 2560 |
| 专家中间层维度 | 768 |
| 稠密中间层维度 | 6144 |
| 词表大小 | 157,184 |
基础模型评估
为了系统性地评估基础模型的能力，我们使用了一套全面的基准测试套件，覆盖数学、编程、推理、多语言理解和长上下文理解等多个关键领域。预训练基础检查点（即 Ling-3.0-flash-base）的性能对比如下：
预期用途
推荐用例：
- 持续预训练
- 中期训练
- 面向领域适配的监督微调
- 偏好优化与 RL 后训练 知识蒸馏研究
- 长上下文与 MoE 系统研究
不推荐直接用于：
- 直接面向终端用户的对话部署
- 未经额外对齐和评估的安全关键型应用
- 未经后训练和任务特定验证的生产环境使用
使用方法
关于微调示例，请参考我们的 ling-cookbook。
常见问题
如有任何问题，欢迎随时发起讨论。
许可证
本模型基于 MIT License 发布。
包含 inclusionAI/Ling-3.0-flash-base-midtrain 的合集
inclusionAI/Ling-3.0-flash-base-midtrain 的论文
Introduction
We have open-sourced the Ling-3.0 series, our most efficient language foundation model family to date. To support research and community-driven innovation, we are releasing a collection of checkpoints during the training process as following:
| Model | Pre-trained | Mid-trained | Merged (i.e., WSM) |
|---|---|---|---|
| Ling-3.0-tiny | Ling-3.0-tiny-base-30T | Ling-3.0-tiny-base-midtrain | Ling-3.0-tiny-base |
| Ling-3.0-flash | Ling-3.0-flash-base-30T | Ling-3.0-flash-base-midtrain | Ling-3.0-flash-base |
These checkpoints correspond to different stages of the training process:
- Pretrained checkpoint have completed large-scale pretraining but have not undergone mid-training, WSM merging (or learning-rate decay), or post-training.
- Mid-trained checkpoint have completed mid-training but have not undergone WSM merging (or learning-rate decay) or post-training.
- Merged checkpoints have undergone WSM merging (or learning-rate decay) based on the mid-training checkpoints but have not undergone post-training.
These checkpoints are released to support continued pretraining, fine-tuning, and further research. For the post-trained model, please see and see Ling-3.0-tiny and Ling-3.0-flash.
Model Overview
Key features
- Highly sparse (1/64) MoE architecture: 512 routed experts, with only 8 routed experts and 1 shared expert activated per token. This enables broad model capabilities while activating just 5.1B (Non-emb) parameters per token;
- Native hybrid linear attention: Ling-3.0 series adopt a native hybrid linear attention architecture from the very start of pretraining by combining KDA with Gated MLA to enable efficient processing of long-context inputs.
- Warmup-Stable and Merge: We replace conventional learning-rate decay with weighted checkpoint merging. By eliminating the decay phase, our Base Model is better suited for continual pretraining and dynamic data expansion, while enabling offline exploration of different decay profiles without rerunning costly experiments for each strategy.
- Scale Seamlessly: Ling-3.0-tiny-base and Ling-3.0-flash-base share the same training recipe, enabling community to experiment on the Ling-3.0-tiny-base first and then scale validated training strategies to the larger Ling-3.0-flash-base.
| Model Type | Base (final checkpoint of mid-training) |
|---|---|
| Architecture | Hybrid-linear MoE |
| Parameter Scale | Totoal 124B, Activated 5.1B (Non-emb) |
| Transformer Layers | 35 KDA + 7 Gated MLA (5:1) |
| Number of Dense Layers | 2 |
| Number of Routed Experts | 512 |
| Number of Shared Experts | 1 |
| Number of Activated Experts | 8 |
| Attention Heads | 32 |
| Hidden Size | 2560 |
| Expert Intermediate Size | 768 |
| Dense Intermediate Size | 6144 |
| Vocabulary Size | 157,184 |
Base Model Evaluation
To systematically assess the capabilities of the base model, we use a comprehensive benchmark suite covering several key domains, including mathematics, coding, reasoning, multilingual understanding, and long-context comprehension. The performance of the pretrained base checkpoint, i.e., Ling-3.0-flash-base, is compared below:
Intended Use
Recommended use cases:
- Continued pre-training
- Mid-training
- Supervised fine-tuning for domain adaptation
- Preference optimization and RL post-training Distillation research
- Long-context and MoE systems research
Not recommended as-is for:
- Direct end-user chat deployment
- Safety-critical applications without additional alignment and evaluation
- Production use without post-training and task-specific validation
Usage
For fine-tuning examples, please refer to our ling-cookbook.
FAQ
If you have any question, please feel free to add a discussion.
License
This model is released under the MIT License.
Collection including inclusionAI/Ling-3.0-flash-base-midtrain
Ling 3.0 •13 items•Updated 17 minutes ago• 15