We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.
AuK 技术报告:开源语音生成与编辑基础模型
AI 导读
AuK 是一个开源基础模型,通过自然语言指令和音频上下文统一语音生成与编辑。模型基于约 30.3 亿条指令-音频实例和 195 万小时有效监督训练,覆盖语音生成、内容编辑、增强与分离、副语言编辑及声学编辑五类任务。蒸馏版 AuK-Flash 支持 4 步推理且无需 classifier-free guidance,在匹配条件下相比完整模型实现 4.5 倍墙钟加速,代码与权重已开源。
HuggingFace Daily Papers(社区热门论文)
48
AI 编辑部评分,满分 100AuK 技术报告:开源语音生成与编辑基础模型
AuK 是一个开源基础模型,通过自然语言指令和音频上下文统一语音生成与编辑。模型基于约 30.3 亿条指令-音频实例和 195 万小时有效监督训练,覆盖语音生成、内容编辑、增强与分离、副语言编辑及声学编辑五类任务。蒸馏版 AuK-Flash 支持 4 步推理且无需 classifier-free guidance,在匹配条件下相比完整模型实现 4.5 倍墙钟加速,代码与权重已开源。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org