蚂蚁 inclusionAI 开源 Ming-3.0-audio-flash 音频理解模型

Ming-3.0-audio.mp4

🧩 Model Variants

• Ming-3.0-audio-flash — A 126B-parameter Mixture-of-Experts model with 6.5B activated parameters. This is the model used by the examples and benchmark results below.
• Ming-3.0-audio-tiny — A compact 9.3B-parameter Mixture-of-Experts model with 2.7B activated parameters. See its model card for checkpoint details and usage guidance.

✨ Highlights

• 🎙️ Complex speech recognition — Recognizes Chinese and English speech across dialects, accents, noisy environments, and multi-speaker meetings.
• ⏱️ Long-audio understanding — Comprehends and analyzes long-form recordings.
• 🎧 Multi-dimensional perception — Understands fine-grained speech, non-speech sounds, acoustic scenes, and music.
• 💡 Audio reasoning — Analyzes and reasons directly over audio content.
• 🗣️ Audio interaction — Supports end-to-end spoken dialogue with optional synthesized speech output.

Ming-3.0-audio-flash closes the loop from hearing the sound, to understanding the information, to getting the task done.

🏗️ Architecture

📊 Benchmarks

All scores are reported as percentages. ↑ means higher is better, while ↓ means lower is better. Ming-3.0-audio-flash is evaluated in non-thinking mode. Bold indicates the best result, underlined indicates the second-best result, and an asterisk (*) indicates results obtained in our own evaluation.

Speech Recognition

Audio Intelligence

Text Intelligence

Note

Results for comparison models are reproduced under our evaluation setup and may differ from their official reports due to differences in prompts, decoding settings, model versions, and evaluation pipelines.

🧭 Inference Options

🚀 Quick Start

1️⃣ Download the Model

pip install -U huggingface_hub
hf download inclusionAI/Ming-3.0-audio-flash \
  --local-dir inclusionAI/Ming-3.0-audio-flash

export MODEL_PATH="$PWD/inclusionAI/Ming-3.0-audio-flash"

Download from ModelScope

pip install -U modelscope
modelscope download --model inclusionAI/Ming-3.0-audio-flash \
  --local_dir inclusionAI/Ming-3.0-audio-flash \
  --revision master

export MODEL_PATH="$PWD/inclusionAI/Ming-3.0-audio-flash"

The model directory contains the tokenizer, processor, model implementation, checkpoint shards, and Talker assets. Large model files are distributed through Hugging Face and ModelScope rather than this GitHub repository.

2️⃣ Start a Server

vLLM-Omni

Install the Ming-3.0-audio-flash branch of inclusionAI/vllm-omni:

python3 -m venv .venv-vllm-omni
source .venv-vllm-omni/bin/activate
pip install -r requirements-vllm-omni.txt

This branch contains the bailing_moe_v3_audio pipeline and the current Ming Talker request interface. It supports IP and registered voice IDs through Chat Completions audio.voice, and voice cloning through ref_audio plus prompt_text.

Start text-output serving for audio understanding and reasoning:

MING_OUTPUT_MODE=text ./scripts/serve_vllm_omni.sh "$MODEL_PATH"

Use the ASR profile when reproducing recognition evaluations with 10-gram blocking support:

MING_OUTPUT_MODE=asr ./scripts/serve_vllm_omni.sh "$MODEL_PATH"

Or enable both text and synthesized speech output:

MING_OUTPUT_MODE=audio ./scripts/serve_vllm_omni.sh "$MODEL_PATH"

For Ming-3.0-audio-tiny, point MODEL_PATH to the Tiny checkpoint and select the Tiny deployment profiles:

export MODEL_PATH="$PWD/inclusionAI/Ming-3.0-audio-tiny"
MING_MODEL_VARIANT=tiny MING_OUTPUT_MODE=text ./scripts/serve_vllm_omni.sh "$MODEL_PATH"

scripts/serve_vllm_omni.sh also detects Tiny automatically when the model directory name contains tiny. Set MING_MODEL_VARIANT explicitly when using a custom directory name.

Note

The default Flash profiles use tensor parallelism across eight GPUs. This is required for the default 128K context length on 80 GB H800 GPUs because vLLM reserves KV-cache memory from the configured context length at startup. In the full-audio profile, the Talker reuses GPU 7 with a separate memory budget. The Tiny full-audio profile uses one GPU per stage. Adjust devices, context and sequence limits, and gpu_memory_utilization in deploy/*.yaml for your hardware.

SGLang

Install the audio branch of inclusionAI/sglang_ling_v3 in a separate environment:

python3 -m venv .venv-sglang
source .venv-sglang/bin/activate
pip install -r requirements-sglang.txt

When the sglang_ling_v3 source checkout is available next to this repository, install that exact checkout in editable mode instead:

pip install -e ../sglang_ling_v3/python

Start Ming-3.0-audio-flash with the validated four-GPU profile:

./scripts/serve_sglang.sh "$MODEL_PATH"

For Ming-3.0-audio-tiny, use its checkpoint and the single-GPU profile:

export MODEL_PATH="$PWD/inclusionAI/Ming-3.0-audio-tiny"
MING_MODEL_VARIANT=tiny CUDA_VISIBLE_DEVICES=0 \
  ./scripts/serve_sglang.sh "$MODEL_PATH"

The SGLang example currently covers audio input with text output. Use vLLM-Omni or direct Transformers inference when synthesized speech output is required.

3️⃣ Send a Request

vLLM-Omni

pip install -r requirements-client.txt

Speech recognition

python examples/vllm_omni_audio_chat.py examples/audio/introduce_ant_group.wav \
  --task asr \
  --prompt 'Please recognize the language of this speech and transcribe it, placing a `\t` between the identified language and the transcribe.' \
  --max-tokens 256 \
  --no-repeat-ngram-size 10

Spoken response

python examples/vllm_omni_audio_chat.py examples/audio/introduce_ant_group.wav \
  --audio-response \
  --output-audio outputs/answer.wav

By default, vLLM-Omni uses the built-in IP voice 灵小柔. Pass --voice to select another IP voice from the complete built-in voice list, or a registered voice ID returned by GET /v1/audio/voices. The clients send this as Chat Completions audio.voice: {"id": "..."}. Voice selection cannot be combined with --ref-audio; omitting it uses the server default.

The model directory also provides two preset reference voices, 灵楷 and US_James, under voice_presets/voice_audio/. Their matching transcripts are recorded in voice_presets/voice_prompts.json. Use these presets, or another reference WAV with its exact transcript, through --ref-audio and --prompt-text:

python examples/vllm_omni_audio_chat.py examples/audio/introduce_ant_group.wav \
  --audio-response \
  --ref-audio path/to/reference.wav \
  --prompt-text "<exact transcript of the reference audio>" \
  --output-audio outputs/cloned_answer.wav

Streaming spoken response with latency metrics

python examples/vllm_omni_audio_streaming.py examples/audio/introduce_ant_group.wav \
  --output-audio outputs/stream_answer.wav

The clients use http://localhost:8091/v1 by default and automatically select the first model returned by /v1/models. Override the connection with --base-url, --api-key, and --model, or the corresponding MING_* environment variables.

SGLang

The SGLang client sends raw base64 audio using the branch's input_audio message format. Run ASR against the default endpoint at http://localhost:8008/v1:

python examples/sglang_audio_chat.py examples/audio/introduce_ant_group.wav \
  --task asr \
  --max-tokens 256

For spoken question answering, use the validated default system prompt:

python examples/sglang_audio_chat.py examples/audio/introduce_ant_group.wav

Override the endpoint or served model with --base-url and --model, or with MING_SGLANG_BASE_URL and MING_SGLANG_MODEL.

🤗 Transformers Inference

Use an isolated environment for direct inference:

python3 -m venv .venv-transformers
source .venv-transformers/bin/activate
pip install -r requirements-transformers.txt

Speech recognition

python examples/transformers_audio.py examples/audio/introduce_ant_group.wav \
  --model "$MODEL_PATH" \
  --task asr \
  --prompt 'Please recognize the language of this speech and transcribe it, placing a `\t` between the identified language and the transcribe.' \
  --max-new-tokens 256 \
  --no-repeat-ngram-size 10

Spoken question answering

python examples/transformers_audio.py examples/audio/introduce_ant_group.wav \
  --model "$MODEL_PATH"

Thinker + native Talker

Use the Talker implementation bundled with the model to generate speech without starting a vLLM-Omni server:

python examples/transformers_thinker_talker.py \
  --model "$MODEL_PATH" \
  --audio examples/audio/introduce_ant_group.wav \
  --voice US_Stella \
  --thinker-max-memory 72GiB \
  --output outputs/answer.wav

Reference-audio voice cloning is also supported:

python examples/transformers_thinker_talker.py \
  --model "$MODEL_PATH" \
  --audio examples/audio/introduce_ant_group.wav \
  --ref-audio path/to/reference.wav \
  --prompt-text "<exact transcript of the reference audio>" \
  --output outputs/cloned_answer.wav

Important

Direct inference loads the model's registered AutoClasses with trust_remote_code=True. Review downloaded model code before enabling remote code in your environment. Native speech output additionally imports the Talker implementation bundled in the model directory.

📚 More Examples

• Explore task-specific prompts in examples/README.md.
• Run the interactive walkthrough in cookbook.ipynb.
• Inspect the bundled sample and metadata in examples/audio/.

🗂️ Repository Layout

Ming-3.0-audio-flash/
├── cookbook.ipynb                  # Interactive walkthrough
├── deploy/                         # vLLM-Omni deployment profiles
├── examples/
│   ├── audio/                      # Sample audio and metadata
│   ├── sglang_audio_chat.py         # SGLang text-output client
│   ├── vllm_omni_audio_chat.py      # Text and non-streaming speech client
│   ├── vllm_omni_audio_streaming.py # Streaming speech and latency metrics
│   ├── transformers_thinker_talker.py # Direct Thinker-to-Talker inference
│   └── transformers_audio.py       # Direct Transformers inference
├── scripts/
│   ├── serve_vllm_omni.sh          # vLLM-Omni launcher
│   └── serve_sglang.sh             # SGLang launcher
├── requirements-client.txt         # Lightweight API client dependencies
├── requirements-sglang.txt         # Compatible SGLang audio branch
├── requirements-vllm-omni.txt      # Compatible vLLM-Omni server branch
└── requirements-transformers.txt   # Direct inference dependencies

🔌 API Format

The vLLM-Omni endpoint accepts audio as a base64 data URL:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this audio."},
            {
                "type": "audio_url",
                "audio_url": {"url": "data:audio/wav;base64,..."},
            },
        ],
    }
]

Use modalities=["text"] for text output or modalities=["text", "audio"] for text and synthesized speech.

The SGLang endpoint uses raw base64 audio in an input_audio content item:

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "input_audio",
                "input_audio": {"data": audio_base64, "format": "wav"},
            },
            {"type": "text", "text": "Describe this audio."},
        ],
    }
]

Chat Completions speech output selects an IP or registered voice through audio.voice:

audio = {
    "format": "wav",  # The streaming client uses vLLM-Omni's raw PCM format.
    "voice": {"id": "灵小柔"},
}

The server checks its IP voice list before looking up registered reference voices. Omit voice to use the default built-in IP voice, 灵小柔.

Reference-audio voice cloning uses raw base64 audio and its transcript:

additional_information = {
    "voice_mode": "reference",
    "ref_audio": reference_audio_base64,
    "prompt_text": reference_audio_transcript,
}

📄 License

This project is released under the MIT License. See LEGAL.md for the legal disclaimer.