蚂蚁 inclusionAI 开源 Ling-3.1-flash-base 基座模型，560B 参数激活 25B
inclusionAI/Ling-3.1-flash-base
蚂蚁 inclusionAI 在 Hugging Face 发布 Ling-3.1-flash-base，即 Ling-3.1-flash 的基座检查点，总参数 560B、每 token 激活 25B。
🤗 Hugging Face | 🤖 ModelScope
Introduction
We introduce Ling-3.1-flash-base, the base checkpoint of Ling-3.1-flash, released to support research and community-driven innovation. With 560B total parameters and 25B active parameters per token, the model provides a foundation for continued pretraining, domain adaptation, and post-training research.
Ling-3.1-flash-base shares the same hybrid-linear Mixture-of-Experts (MoE) architecture as Ling-3.1-flash, combining KDA with Gated MLA. As a base checkpoint, it is intended for further training and adaptation.
The model was trained using FP8. This release provides the BF16 base checkpoint produced after Warmup-Stable and Merge (WSM) checkpoint merging. BF16 refers to the precision of the released weights, while FP8 refers to the precision used during training.
Model Overview
Key Features
- Sparse MoE architecture: 512 routed experts, with 8 routed experts and 1 shared expert activated per token, for a total of 25B active parameters per token.
- Hybrid linear attention: 63 KDA layers and 9 Gated MLA layers combine linear attention with Gated MLA at a 7:1 ratio.
Model Specifications
| Specification | Value |
|---|---|
| Model Type | Base checkpoint |
| Architecture | Hybrid-linear MoE |
| Parameter Scale | Total 560B, Activated 25B |
| Transformer Layers | 63 KDA + 9 Gated MLA (7:1) |
| Number of Dense Layers | 4 |
| Number of Routed Experts | 512 |
| Number of Shared Experts | 1 |
| Number of Activated Routed Experts | 8 |
| Attention Heads | 64 |
| Hidden Size | 4096 |
| Expert Intermediate Size | 1280 |
| Dense Intermediate Size | 10240 |
| Vocabulary Size | 157184 |
Architecture Diagram
Base Model Evaluation
We evaluate Ling-3.1-flash-base on 47 benchmarks across eight domains: language understanding, knowledge, code, mathematics, reasoning, multilingual capabilities, medical knowledge, and long-context understanding. The results below compare the model with DeepSeek-V4-Pro-Base and MiMo-V2.5-Pro-Base under the shot configurations shown.
Bold indicates the highest score in each row; underlining indicates the second-highest distinct score. Tied scores receive the same formatting. EM denotes exact match, and Acc denotes accuracy. Parameter scales are shown as total parameters and active parameters per token (A).
| Domain | Benchmark | Shot Config | Ling-3.1-flash-base | DeepSeek-V4-Pro-Base | MiMo-V2.5-Pro-Base |
|---|---|---|---|---|---|
| 560B A25B | 1.6T A49B | 1.02T A42B | |||
| Language Understanding | Race-High(EM) | 0-shot | 93.88 | 93.37 | 93.60 |
| SquadV2(Acc) | 1-shot | 92.79 | 87.70 | 88.93 | |
| Belebele(EM) | 0-shot | 94.67 | 93.83 | 92.72 | |
| C3(EM) | 0-shot | 83.34 | 81.64 | 81.53 | |
| CCPM(EM) | 0-shot | 91.88 | 93.31 | 90.18 | |
| Knowledge | ARC-c(EM) | 0-shot | 97.97 | 96.61 | 96.95 |
| AGIEval(Acc) | few-shot(0~5) | 82.59 | 81.26 | 79.90 | |
| SimpleQA-Verified(Acc) | 5-shot | 52.40 | 52.10 | 32.60 | |
| C-SimpleQA(Acc) | 5-shot | 77.43 | 77.60 | 70.93 | |
| AA-Omniscience(Acc) | 3-shot | 37.67 | 48.16 | 36.45 | |
| MMLU-Pro(EM) | 5-shot | 76.71 | 75.28 | 71.26 | |
| CEval(EM) | 5-shot | 92.36 | 93.76 | 92.08 | |
| CMMLU(EM) | 5-shot | 91.60 | 90.52 | 88.33 | |
| GPQA-Diamond(EM) | 0-shot | 56.57 | 52.02 | 54.04 | |
| SuperGPQA(EM) | 5-shot | 55.84 | 53.19 | 48.65 | |
| Code | HumanEval-Plus(Pass@1) | 0-shot | 79.88 | 57.93 | 82.93 |
| CruxEval(Pass@1) | 1-shot | 80.50 | 69.06 | 71.25 | |
| MultiPL-E(Pass@1) | 0-shot | 70.55 | 55.94 | 64.74 | |
| MBPP-Plus(Pass@1) | 3-shot | 76.19 | 73.81 | 77.51 | |
| BigCodeBench(Pass@1) | 0-shot | 53.60 | 51.49 | 66.75 | |
| LCBench1(Pass@1) | 3-shot | 68.48 | 59.14 | 67.58 | |
| LCBench-cn1(Pass@1) | 3-shot | 60.83 | 57.35 | 63.36 | |
| Math | MATH500(Acc) | 4-shot | 79.40 | 68.20 | 57.20 |
| CollegeMath(Acc) | 4-shot | 73.41 | 63.15 | 66.61 | |
| FinanceReasoning(Acc) | 3-shot | 81.76 | 82.16 | 80.89 | |
| OlympiadBench(Acc) | 3-shot | 60.24 | 46.39 | 48.04 | |
| TheoremQA(Acc) | 5-shot | 63.56 | 61.92 | 60.27 | |
| OmniMath(Acc) | 3-shot | 65.31 | 34.89 | 40.13 | |
| Reasoning | CommonSenseQA(EM) | 5-shot | 92.22 | 86.49 | 88.53 |
| BBH(EM) | 3-shot | 90.25 | 89.98 | 89.80 | |
| KorBench(Acc) | 3-shot | 54.96 | 54.56 | 53.20 | |
| AutoLogi(Acc) | 3-shot | 67.13 | 64.81 | 66.23 | |
| ZebraLogic(Acc) | 1-shot | 30.80 | 20.30 | 28.90 | |
| Multilingual | mARC(EM) | 0-shot | 92.98 | 89.10 | 88.55 |
| MMMLU(EM) | 0-shot | 77.56 | 68.81 | 67.15 | |
| MultiGSM(Acc) | 4-shot | 87.66 | 85.87 | 86.33 | |
| HumanEvalXL(Pass@1) | 0-shot | 83.25 | 70.31 | 87.34 | |
| Med | DiagnosisArena(EM) | 0-shot | 62.20 | 55.50 | 52.70 |
| medmcqa(EM) | 5-shot | 78.08 | 80.33 | 77.50 | |
| medqa(EM) | 5-shot | 87.82 | 87.75 | 85.55 | |
| Long-context | LongBench(Acc) | 0-shot | 51.90 | 17.70 | 15.66 |
| LEval(Acc) | 0-shot | 85.09 | 58.34 | 61.49 | |
| ruler_16k(Acc) | 1-shot | 96.90 | 97.00 | 90.61 | |
| ruler_32k(Acc) | 1-shot | 95.87 | 95.57 | 89.10 | |
| ruler_64k(Acc) | 1-shot | 93.81 | 95.56 | 87.19 | |
| ruler_128k(Acc) | 1-shot | 88.64 | 95.06 | 85.79 | |
| ruler_256k(Acc) | 1-shot | 86.64 | 94.37 | 86.00 | |
|
Note: 1 LCBench and LCBench-cn: January 2023–February 2025 (2301-2502). |
|||||
Intended Use
Recommended use cases:
- Continued pretraining.
- Mid-training.
- Supervised fine-tuning for domain adaptation.
- Preference optimization and reinforcement-learning post-training.
- Distillation research.
- Long-context and MoE systems research.
Not recommended as-is for:
- Direct end-user chat deployment.
- Safety-critical applications without additional alignment and evaluation.
- Production use without post-training and task-specific validation.
Usage
For fine-tuning examples, please refer to our ling-cookbook.
FAQ
For questions or feedback, please open a discussion in the model repository.
License
This model is released under the MIT License.
来源：蚂蚁 inclusionAI：HuggingFace 新模型 · huggingface.co