蚂蚁 inclusionAI 开源 Ming-3.0-tts 语音合成模型

Multilingual speech synthesis with free-form and paralinguistic control.

🤗 Hugging Face｜ 🤖 ModelScope ｜ 📑 Blog & Demos ｜ 💬 DingTalk(钉钉)

Ming-3.0-tts provides two speech-generation models: 0.5b, a dense model, and tiny, a MoE model built on Ling V3 (7.9B total · 1.3B active).

🏆 Artificial Analysis Leaderboards

On the independent Artificial Analysis text-to-speech leaderboards (October 2026), Ming-3.0-tts ranks 1st among open-weights models on Controlled Voice, 1st overall on Pronunciation Robustness, and 3rd among open-weights models on Provider Voice. Ranks change as new models are added.

Within Pronunciation Robustness, Ming-3.0-tts also ranks 1st on Expanding Shorthand (text normalization, 96.5%) and Preserving Exact Sequences (sequence fidelity, 90.6%).

Featured Abilities

• Voice creation: clone a reference voice, design one from text, or choose from 100+ Chinese and five English built-in voices.
• Free-form control: guide speech delivery with natural-language instructions.
• Paralinguistic control: insert 14 inline tags for laughter, sighs, and other vocal expressions in Chinese or English.
• Languages and dialects: synthesize Chinese, English, German, Spanish, French, Korean and Japanese, plus Sichuanese and Cantonese.
• Emotion and attributes: control emotion, or adjust speed, pitch and volume.
• Multi-speaker dialogue: generate a conversation from a script and speaker references (tiny only).

Capabilities use separate task modes, not arbitrary combinations. See the Cookbook for supported inputs, Performance for benchmarks, and Blog & Demos for audio examples.

Demo Video

tts11_under10mb.mp4

Architecture

Text and task-specific control tokens condition a language model with a DiT patch head and an audio tokenizer. Both models share this generation framework.

News

• 🔥 [2026.10] Ming-3.0-tts-0.5b and Ming-3.0-tts-tiny are released, with inference code, a task cookbook and the evaluation tables below.

Roadmap

• Ming-3.0-tts-0.5b weights and inference code
• Ming-3.0-tts-tiny weights and inference code
• Ming-3.0-tts-tiny benchmarks
• vLLM-Omni accelerated inference
• Technical report

Contents

• Demo Video
• Models
• Quick Start
• Cookbook
• Built-in voices
• Performance
• Limitations
• Usage Disclaimer

Models

Each release checkpoint bundles the backbone, DiT head, audio tokenizer, speaker encoder and text tokenizer. The 0.5b release layout is shown below; tiny uses sharded weights with model.safetensors.index.json.

Ming-3.0-tts-0.5b/
├── config.json              # model_type: dense, auto_map -> BailingMMConfig
├── configuration.json       # ModelScope task descriptor
├── model.safetensors        # backbone + audio VAE + DiT head + aggregator
├── campplus.onnx            # speaker embedding extractor
├── tokenizer.json           # plus vocab.json / merges.txt / added_tokens.json
├── tokenizer_config.json    #      special_tokens_map.json / chat_template.jinja
└── README.md

Quick Start 🚀

Installation

git clone https://github.com/inclusionAI/Ming-3.0-tts.git
cd Ming-3.0-tts
pip install -r requirements.txt        # Ming-3.0-tts-0.5b

Or build the 0.5b Docker image (torch 2.6, CUDA 12.4, transformers==4.46.3):

docker build -f docker/Dockerfile -t ming-tts .

The 0.5b environment has been tested on NVIDIA H800-80GB and H20-96G. For 0.5b, flash-attn is optional; inference falls back to PyTorch SDPA.

Tiny uses a separate reference inference environment: Python 3.10.15, PyTorch and torchaudio 2.7.1+cu118 (CUDA 11.8 wheels), and transformers==4.46.3. Install the pinned dependencies from requirements-tiny.txt in a dedicated environment, as other dependency versions differ from 0.5b. We recommend using NVIDIA H800 for inference.

conda create -n ming-tiny python=3.10.15 -y && conda activate ming-tiny
pip install -r requirements-tiny.txt

Model Download

pip install "huggingface_hub[cli]"
hf download inclusionAI/Ming-3.0-tts-0.5b --local-dir Ming-3.0-tts-0.5b

In mainland China, ModelScope is usually faster:

pip install modelscope
modelscope download --model inclusionAI/Ming-3.0-tts-0.5b --local_dir Ming-3.0-tts-0.5b

Replace 0.5b with tiny for the other variant.

Text Normalization

Numbers, dates, units and symbols are normalized with FST grammars, which need pynini. It has no universal wheel:

conda install -c conda-forge pynini==2.1.6   # or: pip install pynini==2.1.6

It is loaded lazily; pass --no_text_normalization to skip it.

Ming-3.0-tts-0.5b

python infer.py \
  --model_path inclusionAI/Ming-3.0-tts-0.5b \
  --task TTS \
  --language_id Chinese \
  --text "傍晚的风从山口吹过来，远处的云慢慢染成了金色。" \
  --prompt_wav_path data/voice_presets/voice_audio/lingguang.wav \
  --prompt_text "诶，你今天有空吗？我们能不能聊聊这件事啊？" \
  --use_spk_emb \
  --output output/tts.wav

Or from Python:

from ming_tts import MingTTS

model = MingTTS("inclusionAI/Ming-3.0-tts-0.5b")
model.synthesize(
    task="TTS",
    language_id="Chinese",             # English for English text
    text="傍晚的风从山口吹过来，远处的云慢慢染成了金色。",
    prompt_wav_path="data/voice_presets/voice_audio/lingguang.wav",
    prompt_text="诶，你今天有空吗？我们能不能聊聊这件事啊？",
    use_spk_emb=True,
    output_path="output/tts.wav",
)

Ming-3.0-tts-tiny

Every task has a one-line script in examples/one_click/; variables such as MODEL, TEXT and OUT can be overridden from the environment:

bash examples/one_click/run_seed.sh
MODEL=/path/to/Ming-3.0-tts-tiny OUT=outputs/yue.wav bash examples/one_click/run_dialect.sh

The same call from Python. Loading configures the release runtime (bf16, portable MoE loop, fp32 recurrent state) and checks the checkpoint against it; generate() defaults to cfg_scale=2.0, temperature=0, flow_steps=10, sigma=0.25, and seed 1234 (1235 for podcast):

from ming_tiny import MingTinyTTS

tts = MingTinyTTS.from_pretrained("inclusionAI/Ming-3.0-tts-tiny", device="cuda:0")

tts.generate(
    task="seed",
    text="傍晚的风从山口吹过来，远处的云慢慢染成了金色。",
    prompt_wav="data/voice_presets/voice_audio/lingguang.wav",
    prompt_text="诶，你今天有空吗？我们能不能聊聊这件事啊？",
    target_language="Chinese",          # English for English text
    output_path="outputs/seed.wav",
)

tts.generate(
    task="podcast",
    text="speaker_1: 你最近是不是又去那家面馆了？\nspeaker_2: 对啊，排了半个小时的队，不过真的值得。",
    prompt_wav=["data/voice_presets/voice_audio/lingguang.wav", "data/voice_presets/voice_audio/db30.wav"],
    prompt_text=["诶，你今天有空吗？我们能不能聊聊这件事啊？", "感谢你的认可。"],
    output_path="outputs/podcast.wav",
)

Each call selects one task and accepts the inputs listed below:

python infer_tiny.py --help lists the same arguments for the command line; repeat --prompt_wav and --prompt_text for podcast.

Verify Your Setup

Check the checkpoint and tokenizer without loading weights:

python tools/preflight.py --model_path /path/to/Ming-3.0-tts-0.5b
python tools/preflight_tiny.py --model_path /path/to/Ming-3.0-tts-tiny

Cookbook

Ordered the way a line gets made: who speaks, how it is said, what goes beyond the words, and where it is heard. Each example is one task; controls from another task are rejected. Texts and controls are the ones on the demo page, so you can compare against what you hear there. 0.5b API; full value lists in English / 简体中文, every task end to end in cookbooks/cookbook.ipynb.

Who speaks

Zero-shot Cloning: the TTS call in Quick Start clones the voice of prompt_wav_path.

Voice Design: describe the voice; no reference audio.

model.synthesize(
    task="VOICE_DESIGN",
    text="冰川在漫长岁月中缓缓移动，也悄悄记录着气候变化的痕迹。",
    controls={"style": "成熟沉稳的男性纪录片旁白，吐字清晰，节奏舒缓，语气理性而有厚度。"},
    use_zero_spk_emb=True,
)

Built-in Voices: recommended US_James, US_Stella, 灵小柔 and 灵楷. See the full voice list.

model.synthesize(
    task="IP",
    text="The garden will remain open until sunset, provided the weather stays clear.",
    controls={"ip": "US_James"},
    use_zero_spk_emb=True,
)

How it is said

Freeform: describe the voice and delivery with one natural-language instruction. Both models default to generation without reference audio; adding a reference is not recommended. Not combined with voice design, a built-in voice, emotion, dialect or attribute controls.

model.synthesize(
    task="FREEFORM",
    text="The old radio crackled softly as the storm moved across the valley.",
    instruction="Speak in a warm, intimate storytelling voice, with a gentle pace and a hint of nostalgia.",
    use_zero_spk_emb=True,
)

For Tiny, omit prompt_wav and prompt_text:

tts.generate(
    task="freeform",
    text="The old radio crackled softly as the storm moved across the valley.",
    instruction="Speak in a warm, intimate storytelling voice, with a gentle pace and a hint of nostalgia.",
)

Emotion: 高兴 / 悲伤 / 愤怒 / 恐惧 / 厌恶 / 惊讶 / 中性.

model.synthesize(
    task="EMOTION",
    text="所有测试都顺利通过了，我们终于可以把这个好消息告诉大家！",
    controls={"emotion": "高兴"},
    prompt_wav_path="data/wavs/emotion_prompt.wav",
    use_spk_emb=True,
)

Speed, Pitch and Volume: speed 慢速 / 中速 / 快速, pitch and volume 低 / 中 / 高.

model.synthesize(
    task="BASIC",
    text="今晚的观测将在云层散开后开始，请耐心等待下一次通知。",
    controls={"speed": "慢速", "pitch": "中", "volume": "中"},
    spk_emb_path="/path/to/speaker_embedding.pt",
    use_spk_emb=True,
)

Beyond the words

Paralinguistic Tags: inline, anywhere in the sentence, in Chinese or English. [Breathing], [Cough], [Sigh], [Laughter], [Surprise-ah], [Surprise-oh], [Dissatisfaction-hnn], [Uhm], [Confirmation-en], [Question-ei], [Question-ah], [Question-en], [Question-oh], [Question-huh].

model.synthesize(
    task="TTS",
    text="[Sigh]雨还是没有停，[Cough]我们恐怕只能把活动改到室内了。",
    prompt_wav_path="data/voice_presets/voice_audio/lingguang.wav",
    prompt_text="诶，你今天有空吗？我们能不能聊聊这件事啊？",
    use_spk_emb=True,
)

Where it is heard

Multilingual: language_id is required for the five new languages; short codes and full names both work.

model.synthesize(
    task="MULTILINGUAL",
    text="週末、駅前の小さな喫茶店で、季節限定の抹茶パフェを食べました",
    language_id="Japanese",     # ja / ko / de / fr / es
    prompt_wav_path="/path/to/japanese_prompt.wav",
    prompt_text="参考音频对应的日语文本。",
    use_spk_emb=True,
)

Dialects: chuanyu dialect or cantonese; this task takes no language_id.

model.synthesize(
    task="DIALECT",
    text="今日天气几好，我哋食完饭一齐去海边行下啦。",
    controls={"dialect": "cantonese"},
    prompt_wav_path="data/wavs/yue_prompt.wav",
    use_spk_emb=True,
)

Built-in voices

Recommended voices

Over 100 Chinese voices and five English voices are available. See the voice catalog for the full list, including emotion-specific variants.

Performance

Open benchmarks are below; independent arena results are in Artificial Analysis Leaderboards.

Below, tiny is Ming-3.0-tts-tiny and 0.5b is Ming-3.0-tts-0.5b. Ranks and marks are computed per metric among the listed open-source models; Seed-TTS and NVbench-GT are reference rows and are not ranked. Where a historical Ming-omni-tts result beats a bolded Ming-3.0-tts value, it is noted below the table and no SOTA is claimed. Bold marks the best value and underline the second-best value in each column; ties share a mark. Other systems' results use their reported inference settings.

Summary

1. Paralinguistic: together, the two models reach SOTA on all six NVBench single-tag metrics; tiny reaches SOTA on all three English multi-tag metrics.
2. Dialect: 0.5b reaches SOTA on Cantonese Base CER and ACC; tiny reaches SOTA on Sichuanese Hard CER and Easy ACC.
3. Multilingual: tiny reaches SOTA on German and Spanish WER, ranks 2nd on French WER, and ties for 2nd on Japanese CER.
4. Emotion: tiny ties for SOTA on CV3 text-related Happy and reaches SOTA on both CV3N Sad subsets; 0.5b reaches SOTA on CV3 text-related Angry. Tiny's CV3 average ranks 2nd.
5. Attribute control: 0.5b leads the current comparison on average ACC and pitch, while tiny leads on WER and SIM. Historical Ming-omni-tts results still hold the corresponding overall records.
6. Podcast: tiny achieves CER 2.03%, cpSIM 0.561 and UTMOS 2.12, ranking 1st, 2nd and 2nd; historical Ming-omni-tts retains lower CER.
7. Voice Design: tiny ranks 2nd on average ACC (75.37%), APS (85.10%) and DSD (79.90%).
8. Chinese and English: 0.5b's Chinese WER ranks 3rd; tiny reaches SOTA on Chinese and English SIM.
9. Next: improve multilingual and Cantonese speaker similarity, tiny's Korean accuracy and pitch control, text-unrelated emotion, role-play instruction following, and predicted podcast audio quality.

Chinese and English (seed-tts-eval)

Tiny achieves SOTA speaker similarity in Chinese and English. 0.5b has lower WER in both languages and ranks 3rd on Chinese WER.

Multilingual (MiniMaxAI TTS-Multilingual-Test-Set)

_{Each open-source model was evaluated using its recommended inference settings under a unified ASR-based WER/CER evaluation protocol.}

Tiny achieves SOTA WER in German and Spanish, ranks 2nd in French WER and ties for 2nd in Japanese CER. 0.5b is more accurate in Korean; speaker similarity remains a gap for both models.

Paralinguistic control (NVBench)

CER, PCER (paralinguistic CER) and OCER are reported. NVbench-GT is human-labelled reference, shown for context and not ranked.

Together, the two models achieve SOTA on all six Chinese and English single-tag metrics; tiny also leads all three English multi-tag metrics. Chinese multi-tag control remains behind NV-CV3.

Dialect

Tiny is stronger in Sichuanese control accuracy and achieves SOTA Sichuanese Hard CER and Easy ACC, while 0.5b achieves SOTA Cantonese Base CER and ACC. Cantonese speaker similarity remains below CosyVoice2-Yue. Historical Ming-omni-tts results remain stronger on Sichuanese Easy CER (2.25%), Hard SIM (0.750) and Hard ACC (90.45%), and on Cantonese Coverage CER (7.87%).

Emotion control (CV3)

Tiny and 0.5b rank 2nd and 3rd in average ACC. Tiny is stronger on text-related Happy and Sad, while 0.5b is stronger on Angry; text-unrelated emotion remains a shared limitation. Historical Ming-omni-tts results remain higher on text-related Sad (86%) and text-unrelated Angry (82%).

Emotion control (CV3N, hard emotion control)

0.5b and tiny rank 2nd and 3rd in average ACC. Tiny achieves SOTA on both Sad subsets, while 0.5b has higher overall accuracy. Historical Ming-omni-tts results remain higher on text-related Happy (78%) and text-unrelated Happy (30%).

Voice Design (InstructTTSEval-ZH)

Tiny and 0.5b rank 2nd and 3rd in average ACC, behind Qwen3-TTS. Role-play instruction following remains the main gap for both models.

Basic attribute control

0.5b has higher control accuracy, while tiny has lower WER and higher speaker similarity. Historical Ming-omni-tts models retain the overall records for average ACC, pitch, WER and SIM.

Podcast

Tiny ranks 1st in CER and 2nd in cpSIM and UTMOS. Historical Ming-omni-tts retains lower CER and higher UTMOS.

Limitations

• Speaker similarity. Tiny ranks third on German SIM (0.791), but neither model reaches the top three in the other four new languages. French SIM is 0.675 for 0.5b and 0.733 for tiny; both also trail CosyVoice2-Yue on Cantonese SIM.
• Several events in one line. NVBench Multi works, but trails Single; stability with many tags per sentence is still improving.
• Emotion from delivery alone. When the words carry no emotional signal (CV3-Eval Text-Unrelated, and most of the hard set), accuracy drops well below the Text-Related half.
• Pitch on tiny. Pitch is the least accurate attribute for both models. Tiny's 78.70% ranks 2nd, 7.30 percentage points below 0.5b's 86.00%; the historical Ming-omni-tts-0.5B score is 91.33%.
• Role-play voice design. RP is the lowest voice design sub-score, ranking 4th for tiny (61.10%) and 5th for 0.5b (61.06%).
• Korean on tiny. Tiny's Korean CER (6.000%) ranks 6th, well behind 0.5b's 1.728% (3rd).
• Podcast audio quality. Tiny's UTMOS of 2.12 ranks 2nd, below ZipVoice-Dia's 2.24 and the historical Ming-omni-tts-0.5B score of 2.25.

Usage Disclaimer

• Voice cloning and built-in character voices are provided for research and legitimate creative use. Do not use them to impersonate a real person without consent, to deceive, or for any illegal purpose.
• Label synthetic speech as synthetic where it could be mistaken for a real recording.
• The authors accept no liability for misuse. If you find abuse of this project, please report it to us.

Acknowledgements

• CAM++ for speaker embedding extraction
• Ming-omni-tts for the audio tokenizer

License

Released under the MIT license.