StepAudio 3 Gen 技术报告:统一音频生成模型

HuggingFace Daily Papers(社区热门论文)·2026-09-11 08:00·3天前
AI 导读

阶跃星辰发布通用音频生成模型 StepAudio 3 Gen,在统一框架内支持零样本 TTS、音色设计、人声生成、音效、音乐、vibe speech 及多种音频类型混合。

HuggingFace Daily Papers(社区热门论文)
50AI 编辑部评分,满分 100

StepAudio 3 Gen 技术报告:统一音频生成模型

2026-09-11 08:00· 3天前
AI 导读

阶跃星辰发布通用音频生成模型 StepAudio 3 Gen,在统一框架内支持零样本 TTS、音色设计、人声生成、音效、音乐、vibe speech 及多种音频类型混合。

We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared 16 times 2048 residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org