QWEN-LIVE HARNESS
QWEN-MM-PLUGINS
QWEN3.8-OMNI-FLASH API
QWEN3.8-OMNI-FLASH-REALTIME API
引言#
今天，我们正式推出Qwen3.8-Omni-Flash，这是我们下一代原生全模态模型。其核心目标是强化智能体在真实生产力场景中的能力，推动全模态模型从“理解全模态内容”迈向“规划任务、调用工具并完成创作工作”。在编码、文本类知识工作和 GUI 操作等通用智能体能力的基础上，Qwen3.8-Omni-Flash 进一步拓展了以音频和视频为核心的智能体应用，在视频剪辑、音乐视频创作、影片制作与解说、音视频摘要以及实时对话等工作流中均取得了出色表现。
图 1. Qwen3.8-Omni-Flash 及其在生产中的应用。
- Qwen3.8-Omni-Flash — now available on the Qianwen AI Platform:
- 支持文本、图像、音频和视频输入，具备1M token 上下文窗口。
Qwen3.8-Omni-Flash 支持 1M token 的上下文窗口，同时保持与同等规模纯文本模型相当的文本性能，并在全模态能力上实现显著提升。在 29 项评测1中，其平均得分较 Qwen3.5-Omni-Plus 提升超过 25%；每小时音频输入的 API 价格下降超过 98%，每小时音视频输入的价格下降超过 93%2。在音视频智能体、编程和长时程任务方面，该模型在 WildClawBench-MM 上提升 36.5 分，在 AgenticVBench 上提升 22.3 分，同时在 UniClawBench 上取得 69.6 的强劲成绩。其核心能力在长音频与音视频理解、音视频推理、音视频描述以及多说话人识别方面也有显著提升。例如，它在 LongAudioSpan 上提升 8.3 分，在 OmniVideoBench 上提升 9.6 分；其 OmniCap-IF 的 CSR 和 ISR 分别提升 8.5 分和 14.1 分；其 AliMeeting 的 DER 和 cpWER 从 88.11 / 89.61 降至 3.35 / 17.18。通过扩展数据、上下文和智能体环境，Qwen3.8-Omni-Flash 实现了接近 Gemini 3.8 Flash 的音视频性能，以及超越 Gemini 3.8 Flash 的整体音频性能。这些进展也意味着，音频和视频正从感知输入演变为智能体理解环境、进行推理和执行任务的核心媒介。
1. 范围涵盖音频推理基准：AliMeeting-test、AISHELL-4、MagicData-RAMC、MLC-SLM (en)、WenetSpeech (Net | Meeting)、FLEURS-60 ASR、FLEURS-60 S2TT、SpotSoundBench、MMAU、MMAR、MMSU、MuchoMusic-RUL、HumMusQA、MusTBench、Audio-MultiChallenge、WildSpeech 和 VoiceBench；音视频推理基准：DailyOmni、WorldSense、AVUT、JointAVBench、OmniCloze、OmniCap-IF、QIVD、OmniVideoBench 和 StreamingBench；以及音视频智能体基准：WildClawBench-MM、UniClawBench 和 OmniGAIA。
2. 定价方法：每小时音频或音视频输入价格按两分钟源素材输入成本的 30 倍估算；音视频输入使用 720p、1 fps。Gemini 3.8 Flash 使用 media_resolution=high，Seed 2.0 Lite 使用 max_frame_tokens=384，其他所有 API 参数均使用默认值。文本输入和输出价格以每 1M tokens 的人民币计价；Gemini 和 Muse 的价格按 1 USD = 6.7191 CNY 的汇率从美元换算。
音频和视频是将智能体带入真实世界生产力场景的重要媒介，但它们也带来了一系列新的系统级挑战。长音频和长视频在多轮推理中的存储、传输和处理成本高昂；现有的智能体 harness 框架缺乏对这些模态的原生支持；而将全模态理解与端到端任务执行相连接的工作流仍处于早期阶段。要应对这些挑战，需要模型、harness 工具和运行时环境协同演进。
为应对这些挑战，我们使用 Qwen3.8-Omni-Flash 来探索如何将源内容理解、任务规划、工具执行与结果交付串联成一条完整的流水线。它支持端到端、长时程的工作流，例如视频剪辑、翻译、影视解说和内容创作，推动 Omni 从音视频理解迈向自主行动与任务完成。
为此，我们进一步扩展了 Qwen-MM-Plugins，为其加入面向长音频和长视频的按需感知、工具调用与工作流执行能力。我们还开源了 Qwen-Live Harness，作为面向持续、实时全模态交互的原生运行时。二者共同应对长时程工作流与实时交互，同时与模型一道持续拓展全模态智能体的能力。
长音频与长视频理解#
Qwen3.8-Omni-Flash 为长音频与长视频理解带来了重大升级——从可控描述与智能体式证据收集，到理解会议并推进后续任务，最终到生成以视频为核心的深度研究报告。它不仅仅是处理更长的内容，而是更精准地找到相关证据、更深入地进行推理、更高效地采取行动。
可控音视频描述#
对于一段视频应该如何描述，并不存在唯一的答案。内容创作优先考虑叙事，素材检索聚焦于特定片段，而资产管理则依赖于结构。不同的应用需要不同的视频描述。在 Qwen3.8-Omni-Flash 中，我们将视频描述从回答"模型看到了什么"升级为理解"用户想知道什么"。用户可以自由指定主体、时间范围、详细程度和输出格式。对于同一段视频，模型可以提供概览、定位关键片段，或者深入分析人物动作、镜头、灯光和声音，并按需生成结构化结果。你定义看什么、看多细，以及如何呈现。
智能体式长时音视频理解#
对于长达数小时的视频，传统方法要求模型从头到尾处理整个录像，即便答案只出现在其中几分钟的画面里。原生 Qwen3.8-Omni-Flash 智能体从问题出发，自主决定观看和收听的内容，并通过多轮由粗到细的证据收集来定位关键信息。无需处理每一帧，它就能将有限的计算和 token 预算集中在相关片段上，从而实现更高效的长视频理解。在 OmniVideoBench 上，智能体理解将准确率从 63.4 提升至 67.8，同时将 token 消耗从 145,736 降低至 79,117，降幅约为 45.7%。下表对比了静态理解与智能体理解在 OmniVideoBench 上的准确率和 token 消耗：
| 静态理解 | 智能体理解 | |
|---|---|---|
| 准确率（↑） | 63.4 | 67.8 |
| 每次查询的 token 数（↓） | 145,736 | 79,117 |
长会议：从会议纪要到行动#
多参与者会议是最复杂的音视频理解场景之一：发言者轮流发言并相互重叠，而身份、指代和讨论主题不断变化。Qwen3.8-Omni-Flash 跨音频和视频联合识别发言者，并原生支持长达一小时的音视频输入。它能够端到端地执行发言者分割、内容转写和身份对齐。给定一段完整的会议视频和一项请求，该模型可以梳理参与者关系、生成会议纪要、识别行动事项并分析项目风险，利用视觉信息来消解音频中的指代和实体歧义。结合智能体与工具使用，它还能根据会议需求发送邮件、整理任务，甚至开始编写代码——从理解会议迈向对会议采取行动。
利用音频和视频开展深度研究#
当用户带着某个具体问题观看视频时，答案往往超出视频本身。Qwen3.8-Omni-Flash 将用户的需求与视频内容相结合，识别出值得深入探究的问题，整理关键素材，并在全网搜索多模态来源——包括图像、视频和文档。随后，它会生成一份以视频为中心、图文并茂的研究报告，帮助用户理解内容并解决实际问题。
例如，当用户遇到 Photoshop 头发抠图边缘出现色边的问题时，该模型可以拆解教程步骤，研究 Multiply 和 Screen 混合模式背后的原理，比较其他边缘修复技术，并说明哪种方法最适合用户的具体情况。
音视频制作与编辑#
Qwen3.8-Omni-Flash 正将音视频智能体带入一个新阶段：从理解声音和图像，到自主规划、调用工具并交付成片，将全模态智能带入专业音视频内容制作工作流。
Music2MV#
在音乐视频（MV）创作方面，Qwen3.8-Omni-Flash 能够细致理解用户所提供歌曲的结构、节奏、情绪、人声以及器乐变化，为角色、场景和镜头的设计提供依据。它还能输出带时间戳的逐行歌词，以对齐演唱、字幕和画面。结合 Qwen-MM-Plugins 等创意工具，该模型支持从音乐理解、创意策划到最终质量审核的完整工作流，展现出强大的音视频理解、推理与创作能力。
工作流
下一步
最终结果 1
下一步
最终结果 2
下一步
最终结果 3
下一步
短剧翻译#
传统视频翻译往往需要在转录、翻译、配音和剪辑平台之间反复切换。这使 API 调用和工作流协调变得复杂，也难以在角色音色、对话时长和视觉节奏之间保持一致性。借助基于 Qwen3.8-Omni-Flash 构建的智能体，用户只需用一句话描述需求，即可完成说话人感知的对话识别、对话式翻译、角色音色克隆与配音、音频混音以及最终质量审核。这些原本碎片化的本地化步骤由此成为一个完整的工作流，能够为国际受众自动交付短剧。
工作流
下一步
最终结果 1
下一步
源文
翻译结果
最终结果 2
下一步
源文
翻译结果
最终结果 3
下一步
源文
翻译结果
长片影评解说#
为两三个小时的全长电影制作解说视频，往往需要反复观看影片并梳理剧情，随后还要进行镜头挑选、脚本撰写、配音、配乐和剪辑。这是一个复杂且耗时的过程。借助基于 Qwen3.8-Omni-Flash 构建的智能体，用户只需提供一部电影并用一句话描述创作需求，即可完成长视频理解、关键剧情提取、解说策划、配音与配乐制作、剪辑、渲染以及最终质量审核。
该智能体还能智能地将原片对白与解说交错穿插，自动调整语速和音量，使旁白、原声、背景音乐与画面自然融合，打造出更真实、更具沉浸感、更有电影质感的解说视频。
工作流程
下一步
最终结果 1
下一步
最终结果 2
下一步
从使用模型到优化模型#
真实世界的多模态应用往往复杂且对成本敏感，要求模型在质量、延迟、算力与部署成本之间取得平衡。
因此，针对特定场景定制更小的模型，是规模化部署应用的一条重要路径。然而传统工作流涉及数据构建、问题诊断、多轮训练与评估，耗时且高度依赖人工专业知识。这一次，我们将 Qwen3.8-Omni-Flash 延伸到模型开发本身，探索一种新方法：由大模型驱动研发，而小模型服务于业务需求。
我们给 Qwen3.8-Omni-Flash 布置了一项任务：在 12 小时内提升 Qwen2.5-Omni-3B 的四川方言语音识别能力，并交付一个可用的模型。它独立选定了 WenetSpeech-Chuan 评测集，确定了评测标准，并建立了基线。随后它直接聆听音频样本，利用识别结果诊断问题，并构建有针对性的训练数据。在连续四轮实验中，该智能体创建了 3,413 条训练样本，根据评估反馈调整策略，保留有效的改进，并回滚不成功的尝试。Qwen2.5-Omni-3B 在同一评测集上的字符错误率最终从 25.79% 降至 15.30%，相对降低约 40.7%。
这一实验展示了模型演进的另一种可能：通用多模态模型理解数据、规划实验并推动迭代，而更小的模型则获得面向特定应用的专业能力。智能体可以超越"用模型帮助解决实际业务问题"这一层面。
音视频信息压缩#
音频和视频承载着丰富的信息，但其线性、非结构化的形式使得检索和复用变得困难。Qwen3.8-Omni-Flash 能够理解声音、视觉和时间线中的内容，通过智能体工作流提取信息、重组其结构并验证结果。它将长视频中的核心知识和实践经验转化为更密集、更易于消费和复用的信息资产。
Video2Note#
为了将视频知识转化为结构化资源，我们在 Qwen-MM-Plugins 中开源了 Video2Note。借助 Qwen3.8-Omni-Flash 对语音、视觉和流程的联合理解能力，它能够自动组织知识、拆解关键步骤、选取代表性帧，并生成图文对应的 PDF 笔记。自动化审核与迭代修正进一步将数小时的视频浓缩为清晰易读、便于回顾的文档。
工作流
下一步
最终结果 1
下一步
示例视频
最终 PDF
最终结果 2
下一步
示例视频
最终 PDF
最终结果 3
下一步
示例视频
最终 PDF
最终结果 4
下一步
示例视频
最终 PDF
Omni Skill Creator#
视频不仅记录了"如何做某件事"，还记录了专家积累的实践经验。基于这一考虑，我们在 Qwen-MM-Plugins 中引入了 Omni Skill Creator 这一全新的开源能力。它可以从演示中提取标准操作流程（SOP），以执行可复用的自动化工作，或从专家指导中学习工具使用方法、决策标准和关键洞见。一次演示即可转化为经过验证和评估的智能体技能，从而基于全模态内容构建可复用、可分享的技能。
实时音视频交互#
Qwen3.8-Omni-Flash 专为对完整音视频内容的深度理解与创作而设计。针对连续、低延迟的交互场景，我们进一步推出 Qwen3.8-Omni-Flash-Realtime。它在接收实时音视频流的同时进行感知与响应，并利用实时上下文调用工具、执行任务，将全模态能力从“理解一段内容”推进到“参与一场交互”。
实时口语练习#
口语没有标准输入。口音、元音与辅音的替换以及声调偏差，都可能导致逐字转写的结果偏离原本要表达的含义。Qwen3.8-Omni-Flash-Realtime 对发音与语义进行联合建模，能够理解受口音影响的非标准表达，将其与正确的词对齐，并实时生成标准发音示范。在多次练习轮次中，模型会结合新的音频更新其判断，在纠正影响理解的错误的同时，保留自然的节奏、语调与情感。
全模态空间音频感知#
在真实空间中，声音提供了视觉之外的另一条坐标轴。Qwen3.8-Omni-Flash-Realtime 将空间声音与视觉信息相结合，在感知障碍物、可通行区域以及场景变化的同时，持续判断声源的方向与距离，使其成为首个能够通过声音定位目标的全模态模型。
对于“过来这里”或“去看看是什么发出了那个声音”这类指令，模型能够从环境噪声中分离出人声和目标声音，将其含义锚定在周围空间中，并调用工具完成定位、搜索、路径规划和导航——从听到目标到抵达目标。
用于音视频交互的外部知识#
实时交互不仅需要低延迟，还需要能够为每个应用动态加载知识和行为。Qwen3.8-Omni-Flash-Realtime 支持通过技能注入身份设定、表达风格、业务知识和交互规则，而工具调用则将这些能力延伸到任务执行中。
在客户服务等场景中，模型可以实时加载品牌话术和服务流程，理解用户的语音、视觉信息和上下文，生成符合业务要求的回复，并执行操作。因此，同一个实时模型可以承担不同的知识、角色和行为方式。
基准测试结果#
Omni#
| Qwen3.8-Omni-Flash | Qwen3.5-Omni-Plus | Gemini 3.8 Flash | Seed 2.0 Lite | Muse Spark 1.2 | |
|---|---|---|---|---|---|
| 智能体全模态智能 | |||||
WildClawBench-MM 多模态工具使用 | 71.0 | 34.5 | 58.9 | 41.9 | -- |
UniClawBench 多模态工具使用 | 69.6 | 67.1 | 69.0 | 61.2 | -- |
AgenticVBench 多模态工具使用 | 36.8 | 14.5 | 45.0 | 10.0 | -- |
OmniGAIA 网页搜索 | 74.0 | 57.2 | 78.6 | 64.4 | -- |
| 通用音视频能力 | |||||
DailyOmni 音视频理解 | 85.1 | 85.1 | 84.0 | 81.4 | 79.6 |
WorldSense 音视频理解 | 68.5 | 63.9 | 69.6 | 67.3 | 65.0 |
AVUT 音视频理解 | 86.6 | 85.9 | 88.0 | 81.5 | 82.4 |
JoinAVBench 音视频理解 | 75.9 | 74.1 | 70.4 | 70.6 | 71.8 |
OmniVideoBench 音视频推理 | 63.4 | 53.8 | 65.2 | 58.5 | 62.2 |
Video-MME-v2 音视频推理 | 65.0 | 47.9 | 71.0 | 64.9 | -- |
LVOmniBench 长视频推理 | 63.3 | 53.2 | 70.7 | -- | -- |
OmniCloze 音视频描述 | 63.2 | 64.2 | 60.9 | 56.3 | 65.3 |
OmniCap-IF 音视频描述 | CSR 80.6 ISR 28.2 | CSR 72.1 ISR 14.1 | CSR 81.9 ISR 28.3 | CSR 74.6 ISR 18.1 | CSR 77.9 ISR 26.8 |
QIVD 音视频交互 | 69.6 | 65.6 | 69.1 | 62.0 | 62.0 |
StreamingBench 音视频交互 | 80.8 | 57.1 | 79.9 | 77.2 | 77.8 |
| 通用音频能力 | |||||
AliMeeting 测试 多说话人 ASR（DER | cpWER，↓） | 3.4 | 17.2 | 88.1 | 89.6 | 72.6 | 53.1 | 75.1 | 76.1 | 93.7 | 92.7 |
AISHELL-4 多说话人 ASR（DER | cpWER，↓） | 2.8 | 11.2 | 100.0 | 100.0 | 66.4 | 56.9 | 64.8 | 64.2 | 91.3 | 86.0 |
MagicData-RAMC 多说话人 ASR（DER | cpWER，↓） | 5.7 | 14.1 | 98.4 | 97.1 | 67.9 | 33.8 | 43.4 | 35.1 | 82.1 | 75.3 |
MLC-SLM（en） 多说话人 ASR（DER | cpWER，↓） | 4.0 | 14.2 | 68.6 | 63.9 | 60.8 | 26.6 | 40.4 | 45.5 | 74.3 | 52.9 |
WenetSpeech（Net） ASR（WER，↓） | 4.8 | 3.7 | 14.2 | 4.3 | 68.2 |
WenetSpeech（会议） ASR（WER，↓） | 4.6 | 4.8 | 16.7 | 4.7 | 42.6 |
FLEURS-ASR 多语言 ASR（WER，↓） | 9.3 | 7.2 | 7.9 | 32.1 | 23.6 |
FLEURS-S2TT 多语言语音到文本翻译（BLEU） | 31.8 | 32.2 | 33.0 | 24.8 | 28.8 |
SpotSoundBench 音频定位 | 67.2 | 64.2 | 39.7 | 59.6 | 16.9 |
MMAU 音频理解 | 81.8 | 81.9 | 76.9 | 77.2 | 63.5 |
MMAR 音频理解 | 79.8 | 79.8 | 78.5 | 77.7 | 67.3 |
MMSU 音频理解 | 82.1 | 83.0 | 83.3 | 80.2 | 59.9 |
LongAudioSpan 长音频推理 | 准确率 82.7 评分标准 71.8 链式 48.2 | 准确率 74.4 评分标准 49.8 链式 45.1 | 准确率 79.3 评分标准 65.5 链式 64.6 | -- | -- |
MuchoMusic-RUL 音乐理解 | 72.6 | 71.6 | 53.7 | 61.7 | 40.1 |
HumMusQA 音乐理解 | 75.8 | 75.5 | 71.2 | 66.0 | 63.3 |
MusTBench 音乐理解 | 50.6 | 49.1 | 40.3 | 44.0 | 29.4 |
Audio MultiChallenge 音频交互 | 71.5 | 57.6 | 71.9 | 63.4 | 57.9 |
WildSpeech 音频交互 | 74.3 | 75.7 | 76.4 | 74.5 | 73.4 |
VoiceBench 音频交互 | 91.6 | 92.9 | 92.3 | 84.1 | 79.8 |
1. 面向智能体全模态智能的评测框架：WildClawBench-MM 和 AgenticVBench 使用 Claude Code，UniClawBench 使用 OpenClaw，而 OmniGAIA 不使用任何评测框架。对于 WildClawBench-MM，我们仅评估 WildClawBench 中涉及图像、视频或音频的多模态任务。
2. FLEURS：ASR 和 S2TT 评测结果均覆盖以下 60 种语言：中文（普通话）、英语、粤语、阿拉伯语、德语、法语、西班牙语、葡萄牙语、印尼语、意大利语、韩语、俄语、泰语、越南语、日语、土耳其语、印地语、马来语、荷兰语、乌尔都语、挪威语、瑞典语、丹麦语、希伯来语、芬兰语、波兰语、冰岛语、捷克语、菲律宾语、波斯语、希腊语、南非荷兰语、阿斯图里亚斯语、白俄罗斯语、保加利亚语、孟加拉语、波斯尼亚语、加泰罗尼亚语、宿务语、爱沙尼亚语、加利西亚语、古吉拉特语、克罗地亚语、匈牙利语、爪哇语、哈萨克语、卡纳达语、吉尔吉斯语、拉脱维亚语、马其顿语、马拉雅拉姆语、马拉地语、旁遮普语、罗马尼亚语、斯洛伐克语、斯洛文尼亚语、斯瓦希里语、塔吉克语、阿塞拜疆语和乌克兰语。
3. 空白单元格（--）：分数尚不可用或不适用。
智能体全模态理解#
长音频和视频录制中的关键信息往往分散在不同片段中，而复杂问题需要跨声音和图像进行多步推理。智能体全模态理解使模型能够从问题出发，规划方法、调用工具，并逐步定位和验证证据。通过将计算集中在相关内容上，它提升了长时音视频理解的准确性和效率。为评估这一能力，我们在 OmniVideoBench、Video-MME-v2 和 LVOmniBench 上，在两种设置下对比了 Qwen3.8-Omni-Flash 和 Gemini 3.8 Flash：静态模式，即模型直接解读输入；以及使用 Qwen Code 的智能体模式。这一对比展示了引入智能体工作流如何影响各模型的表现。
| Qwen3.8-Omni-Flash （静态） | Qwen3.8-Omni-Flash （Qwen Code） | Gemini 3.8 Flash （静态） | Gemini 3.8 Flash （Qwen Code） | |
|---|---|---|---|---|
OmniVideoBench 音视频推理 | 63.4 | 67.8 | 65.2 | 70.1 |
Video-MME-v2 音视频推理 | 65.0 | 71.3 | 71.0 | 72.7 |
LVOmniBench 长视频推理 | 63.3 | 73.6 | 70.7 | 70.7 |
Text#
| Qwen3.8-Omni-Flash | Qwen3.8- Flash | Qwen3.8- 27B | Qwen3.7- Plus | DeepSeek-V4-Flash-0731 | Claude-Opus-4.6 (Max) | |
|---|---|---|---|---|---|---|
| 编程与智能体 | ||||||
DeepSWE 1.1 长时程软件工程 | 57.8 | 58.7 | 42.2 | 16.5 | 54.4 | -- |
SWE-bench Pro 长时程软件工程 | 63.3 | 62.5 | 61.7 | 55.8 | 56.0 | 53.4 |
SWE-bench Multilingual 多语言软件工程 | 80.5 | 81.0 | 73.8 | 75.8 | -- | 77.5 |
NL2Repo-Bench 仓库级代码生成 | 48.9 | 48.1 | 42.3 | 41.1 | 54.2 | 47.6 |
CoWorkBench 长周期办公工作 | 75.3 | 73.9 | 70.7 | 65.1 | 45.1 | 68.2 |
| 通用文本能力 | ||||||
IFBench 指令遵循 | 81.5 | 81.3 | 79.5 | 79.1 | 79.2 | 62.5 |
GPQA Diamond 科学推理 | 91.0 | 91.7 | 89.2 | 90.3 | 90.8 | 91.3 |
HLE 多学科推理 | 36.5 | 35.9 | 30.8 | 34.7 | 33.8 | 40.0 |
LiveCodeBench v6 竞赛编程 | 92.6 | 91.9 | 90.3 | 89.6 | 90.6 | 88.8 |
1. DeepSWE 1.1：使用 Claude Code 和 mini-SWE-agent 测试框架进行评估，temp=1.0，top_p=0.95，256K 上下文窗口。我们报告两个测试框架中的最高分；值得注意的是，Qwen3.8-Flash-Next 在 mini-SWE-agent 上表现最佳。
2. SWE-bench Pro：除 Claude-Opus-4.6 (Max) 我们报告其官方公布的分数外，所有模型均使用 Claude Code 测试框架进行评估，temp=1.0，top_p=0.95，256K 上下文窗口。有问题的任务已被修正，所有基线模型均在改进后的基准上重新评估。
3. SWE-bench Multilingual：使用 mini-SWE-agent 测试框架进行评估，temp=1.0，top_p=0.95，256K 上下文窗口。
4. NL2Repo-Bench：使用 Claude Code 测试框架进行评估。为防止奖励黑客行为，我们禁用了试图访问特定仓库的 Bash 命令，例如 pip download、pip install 和 git clone。
5. CoWorkBench：一个内部协作基准，用于评估跨计算机科学、金融、法律、医疗及其他生产力领域的长期办公和生产力智能体任务。
6. HLE：由 GPT-4o 评判。
7. 空白单元格（--）：分数尚不可用或不适用。
视觉#
| Qwen3.8-Omni-Flash | Qwen3.8- Flash | Qwen3.8- 27B | Qwen3.7- Plus | Claude-Opus-4.6 (Max) | |
|---|---|---|---|---|---|
| 智能体视觉智能 | |||||
ClawEval-MM 多模态工具使用 | Pass@3 60.4 平均 61.9 | Pass@3 64.4 平均 60.4 | Pass@3 57.4 平均 56.9 | Pass@3 57.4 平均 60.1 | Pass@3 52.5 平均 54.7 |
AndroidWorld 移动端使用 | 87.1 | 84.5 | 81.9 | 81.0 | 62.0 |
Vision2Web 可视化网页开发 | 62.9 | 64.0 | 62.9 | 42.1 | -- |
| 通用视觉能力 | |||||
ERQA 具身智能 | 71.0 | 72.3 | 65.5 | 69.8 | 40.8 |
LVBench 长视频理解 | 76.9 | 76.6 | 72.4 | 76.2 | 63.0 |
RealWorldQA 真实世界感知 | 87.7 | 88.5 | 85.9 | 86.9 | 73.9 |
MathVision 视觉数学问题求解 | 无 CI 91.8 有 CI 96.2 | 无 CI 90.6 有 CI 95.7 | 无 CI 90.0 有 CI 94.6 | 无 CI 90.3 有 CI 88.4 | 无 CI 65.5 有 CI -- |
CharXiv（RQ） 科学图表分析 | 无 CI 83.5 有 CI 91.4 | 无 CI 84.6 有 CI 90.6 | 无 CI 83.7 有 CI 90.2 | 无 CI 85.8 有 CI 85.9 | 无 CI 66.0 启用 CI -- |
1. ClawEval-MM：分数以“pass@3 / 平均分”形式报告。Pass@3 衡量在三次试验中至少一次通过的百分比，平均分是三次试验得分的均值。
2. RecreationBench：内部构建的长时程应用复现基准，用于评估跨五个平台的混合智能体能力——桌面端（Ubuntu、macOS、Windows）、移动端（Android）和 Web 端。
3. OSWorld 2.0：分数以“二值 / 部分分”形式报告。二值分数是获得完整任务奖励的任务百分比，部分分则汇总所有任务中获得的各部分奖励。
4. Vision2Web：分数报告为前端、网页和网站三个类别的平均值，使用 Claude Code harness，由 gpt-5.4-2026-03-05 评判。
5. MathVision、CharXiv（RQ）：分数以“无 CI / 有 CI”形式报告。MathVision 中少量错误的真值标注经人工核验后已修正。我们的模型分数使用固定提示词评估，例如“Please reason step by step, and put your final answer within \boxed{}.”。对于其他模型，我们报告使用与不使用 \boxed{} 格式的多次运行中较高的分数。
6. 空白单元格（--）表示分数尚不可用或不适用。
吞吐量与延迟#
以下结果展示了 Qwen3.8-Omni-Flash-Realtime API 在不同输入条件下的实测吞吐量与延迟，反映了用户在生产环境中所体验到的性能表现。
| 输入场景 | 文本输出 TPS（Tokens/s） | 首 Token 时间（ms） | 首个音频包时间（ms） | 音频生成 RTF |
|---|---|---|---|---|
| Realtime API 性能 | ||||
| 音频 6s | 84.87 | 591.26 | 978.36 | 0.1538 |
| 音频 12s | 83.18 | 604.80 | 982.74 | 0.1537 |
| 音频 20s | 81.06 | 617.98 | 1026.39 | 0.1538 |
| 音视频 6s | 84.89 | 837.96 | 1214.73 | 0.1524 |
| 音视频 12s | 84.33 | 911.85 | 1268.07 | 0.1527 |
| 音视频 20s | 83.00 | 981.01 | 1350.49 | 0.1528 |
支持语言#
| 能力 | 语言 | 中文方言 |
|---|---|---|
| 语音识别 | 74 种语言：南非荷兰语、阿拉伯语、阿斯图里亚斯语、阿塞拜疆语、巴斯克语、白俄罗斯语、孟加拉语、波斯尼亚语、保加利亚语、粤语、加泰罗尼亚语、宿务语、中文、克罗地亚语、捷克语、丹麦语、荷兰语、英语、世界语、爱沙尼亚语、菲律宾语、芬兰语、法语、加利西亚语、格鲁吉亚语、德语、希腊语、希伯来语、印地语、匈牙利语、冰岛语、印度尼西亚语、国际语、意大利语、日语、爪哇语、卡纳达语、哈萨克语、韩语、吉尔吉斯语、林加拉语、拉脱维亚语、立陶宛语、马其顿语、马来语、马拉雅拉姆语、马耳他语、毛利语、马拉地语、蒙古语、书面挪威语、新挪威语、奥里亚语、波斯语、波兰语、葡萄牙语、旁遮普语、罗马尼亚语、俄语、塞尔维亚语、斯洛伐克语、斯洛文尼亚语、西班牙语、斯瓦希里语、瑞典语、塔吉克语、泰米尔语、泰卢固语、泰语、土耳其语、乌克兰语、乌尔都语、维吾尔语和越南语 | 39 种方言：东北官话、贵州话、广东粤语、河南话、香港粤语、上海话、陕西话、天津话、台湾国语、云南话、安徽话、福建话、甘肃话、广东官话、湖北话、湖南话、江西话、山东话、山西话、四川话、广西话、海南话、重庆话、长沙话、杭州话、合肥话、银川话、郑州话、沈阳话、温州话、武汉话、昆明话、太原话、南昌话、济南话、兰州话、南京话、客家话和闽南语 |
| 语音生成 | 29 种语言：中文、英语、德语、意大利语、葡萄牙语、西班牙语、日语、韩语、法语、俄语、泰语、印尼语、阿拉伯语、越南语、土耳其语、芬兰语、波兰语、印地语、荷兰语、捷克语、乌尔都语、他加禄语、瑞典语、丹麦语、希伯来语、冰岛语、马来语、挪威语和波斯语 | 7 种方言：四川话、北京话、天津话、南京话、陕西话、粤语和闽南语 |
Qwen3.8-Omni-Flash 快速上手#
API 用法#
Qwen3.8-Omni-Flash 正式支持 reasoning_effort，用于调整推理深度并控制成本：
xhigh（默认）：适用于需要深入分析的复杂任务。medium：兼顾准确性与速度。low：针对速度和成本优化的高效推理。
此外，preserve_thinking 在所有场景下默认启用，以提供最佳的开箱即用体验。
Qwen3.8-Omni-Flash 支持行业标准协议，包括兼容 OpenAI 的 Chat Completions 和 Responses API。示例如下：
"""
Environment variables:
DASHSCOPE_API_KEY: Your API Key from https://platform.qianwenai.com/home
DASHSCOPE_BASE_URL: (optional) Base URL for compatible-mode API.
- Beijing: https://dashscope.aliyuncs.com/compatible-mode/v1
- Singapore: https://dashscope-intl.aliyuncs.com/compatible-mode/v1
"""
from openai import OpenAI
import os
api_key = os.environ.get("DASHSCOPE_API_KEY")
if not api_key:
raise ValueError(
"DASHSCOPE_API_KEY is required. "
"Set it via: export DASHSCOPE_API_KEY='your-api-key'"
)
client = OpenAI(
api_key=api_key,
base_url=os.environ.get(
"DASHSCOPE_BASE_URL",
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
),
)
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241022/emyrja/dog_and_girl.jpeg"
},
},
{
"type": "input_audio",
"input_audio": {
"data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
"format": "wav"
},
},
{"type": "text", "text": "Please describe the image and tell me what is being said in the audio."},
],
},
]
completion = client.chat.completions.create(
model="qwen3.8-omni-flash",
messages=messages,
extra_body={
"enable_thinking": True,
# "preserve_thinking": True,
},
reasoning_effort="xhigh", # supported levels are xhigh, medium, and low
stream=True,
)
reasoning_content = ""
answer_content = ""
is_answering = False
print("\n" + "=" * 20 + "Reasoning" + "=" * 20 + "\n")
for chunk in completion:
if not chunk.choices:
print("\nUsage:")
print(chunk.usage)
continue
delta = chunk.choices[0].delta
if hasattr(delta, "reasoning_content") and delta.reasoning_content is not None:
if not is_answering:
print(delta.reasoning_content, end="", flush=True)
reasoning_content += delta.reasoning_content
if hasattr(delta, "content") and delta.content:
if not is_answering:
print("\n" + "=" * 20 + "Answer" + "=" * 20 + "\n")
is_answering = True
print(delta.content, end="", flush=True)
answer_content += delta.content
可控音视频描述的最佳实践#
内容组织与详细程度：将视频描述、OCR 提取、音频描述和语音转录整理为多个部分，涵盖画面、语音、音乐、音效和环境音。保留原始文本、说话人身份及对应的时间范围，以便于检索和核验。在提示词中指定目标长度以控制详细程度，例如 Describe the video in approximately 2000–3000 words.，并根据视频的时长和信息密度进行调整。
提示词示例
Provide a detailed description of the video.
Make sure your description covers every one of the following dimensions:
Visual
- Subjects and characters: appearance, clothing, gender/age cues, identity, distinctive features
- Actions and events in chronological order, and how the scene evolves over time
- Setting and background: location, environment, time of day
- Spatial layout and relations between subjects/objects; counts and quantities
- On-screen text: captions, titles, subtitles, logos, UI — exact content and appearance
- Visual style: colors, lighting, camera shots, angles, and camera movement
Audio
- Speech: the exact spoken content, transcribed verbatim
- Speakers: who is speaking (mapped to the on-screen person or voice-over), with accent, tone, gender/age cues
- Speaking state: prosody, emotion, volume, and speaking style
- Music: presence, genre/mood, and lyrics if any
- Sound effects and ambient/background sounds
- Non-speech vocalizations: laughter, crying, applause, etc.
Audio-visual correspondence
- Which speech or sound aligns with which on-screen person or visual event
- The timing of each event, expressed with timestamps
It should explicitly include three sections:
1. A structured chronological storyline of **every noticeable audio and visual details**
2. A structured list of all visible text. For each text element, include start timestamp, end timestamp, the exact text content, the appearance characteristics. If no text appears, explicitly state so.
3. A structured speech-to-text transcription, include speaker (corresponding to the character or voice-over in Section 1, including their accent and tone), exact spoken content, start timestamp, end timestamp, and speaking state (prosody, emotion, and style). If no speech appears, explicitly state so.
Aside from these three required sections, you are free to organize any additional content in any way you find helpful. This additional content can include global information about the entire video or localized information about specific moments. You may choose the topic of this extra content freely.
Rules:
- Add as much descriptive detail as possible.
- Do not use Markdown bold formatting.
- Carefully look at frames and listen to the audio, making sure no detail is overlooked.
Output Format:
```
## Storyline
<xx:xx.xxx> - <xx:xx.xxx>
<an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.>
<xx:xx.xxx> - <xx:xx.xxx>
<an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.>
<xx:xx.xxx> - <xx:xx.xxx>
<an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.>
...
## Visible Text
<xx:xx.xxx> - <xx:xx.xxx>
“<element>”: <appearance>
“<element>”: <appearance>
<xx:xx.xxx> - <xx:xx.xxx>
“<element>”: <appearance>
“<element>”: <appearance>
“<element>”: <appearance>
<xx:xx.xxx> - <xx:xx.xxx>
“<element>”: <appearance>
...
## Speakers and Transcript
Speaker profiles:
<speaker> - <profile>
<speaker> - <profile>
<speaker> - <profile>
...
<xx:xx.xxx> - <xx:xx.xxx>
Speaker: <speaker>
State: <description>
Content: “<content>”
<xx:xx.xxx> - <xx:xx.xxx>
Speaker: <speaker>
State: <description>
Content: “<content>”
<xx:xx.xxx> - <xx:xx.xxx>
Speaker: <speaker>
State: <description>
Content: “<content>”
...
## <another section>
<paragraphs>
## <another section>
<paragraphs>
...
```
结构化输出与 schema 约束：在提示词中包含任务指令和完整的 JSON Schema，明确字段含义、类型、必填字段和约束条件，以支持程序化解析和下游使用。
提示词示例
Describe the audio and visual content in detail in English, organized into scenes and events, following the JSON Schema below.
All timestamps must be relative to the beginning of the video. End times must not precede start times or exceed the video duration. Each event must fall within the time range of its parent scene.
Include only information directly supported by the audio or video. Do not guess or invent details. Do not infer causality merely because a sound and an action occur at the same time.
Return only valid JSON, without Markdown fences or commentary.
JSON Schema:
{
"$defs": {
"Event": {
"additionalProperties": false,
"properties": {
"time_range": {
"$ref": "#/$defs/TimeRange",
"description": "Time range of the event"
},
"participants": {
"description": "People, animals, or objects involved, named by observable features; use consistent names for the same participant",
"items": {
"type": "string"
},
"title": "Participants",
"type": "array"
},
"action": {
"description": "Specific actions, interactions, and observable outcomes",
"title": "Action",
"type": "string"
},
"sounds": {
"description": "Sounds heard during the event; use an empty list if none are discernible",
"items": {
"type": "string"
},
"title": "Sounds",
"type": "array"
}
},
"required": [
"time_range",
"participants",
"action",
"sounds"
],
"title": "Event",
"type": "object"
},
"Scene": {
"additionalProperties": false,
"properties": {
"time_range": {
"$ref": "#/$defs/TimeRange",
"description": "Time range of the scene"
},
"setting": {
"description": "Environment, spatial layout, and main visual features",
"title": "Setting",
"type": "string"
},
"events": {
"description": "Events in chronological order; use an empty list if there are none",
"items": {
"$ref": "#/$defs/Event"
},
"title": "Events",
"type": "array"
}
},
"required": [
"time_range",
"setting",
"events"
],
"title": "Scene",
"type": "object"
},
"TimeRange": {
"additionalProperties": false,
"properties": {
"start_seconds": {
"description": "Start time in seconds relative to the beginning of the video",
"minimum": 0,
"title": "Start Seconds",
"type": "number"
},
"end_seconds": {
"description": "End time in seconds; must not precede the start time",
"minimum": 0,
"title": "End Seconds",
"type": "number"
}
},
"required": [
"start_seconds",
"end_seconds"
],
"title": "TimeRange",
"type": "object"
}
},
"additionalProperties": false,
"properties": {
"summary": {
"description": "An overview of the main content of the video",
"title": "Summary",
"type": "string"
},
"scenes": {
"description": "Scenes in chronological order; group continuous footage with a consistent setting into one scene",
"items": {
"$ref": "#/$defs/Scene"
},
"title": "Scenes",
"type": "array"
}
},
"required": [
"summary",
"scenes"
],
"title": "CaptionResult",
"type": "object"
}
在 Agent Harness 中安装 Qwen-MM-Plugins#
Qwen-MM-Plugins 是一套面向智能体运行框架的多模态插件套件。它让智能体能够理解图像、音频、视频和文档，还能为长视频维护记忆并创作内容。它支持包括 Codex、Claude Code、Qwen Code、Gemini CLI、Qoder、CodeBuddy 和 OpenClaw 在内的智能体运行框架。我们也欢迎社区开发者贡献代码。
你可以直接在你常用的办公智能体中输入以下请求：
Help me install the core, api, and omni-related plugins from https://github.com/QwenLM/Qwen-MM-Plugins.
或者，通过命令行安装：
# Run the official guided installer:
curl -fsSL https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh | bash
在菜单中，选择：
- 安装
- 你使用的智能体运行框架
- 你需要的 Omni 插件
| 插件 | 能力 |
|---|---|
core | 读取图像、视频帧、PDF、Office 文档、代码、数据和 3D 文件 |
api | 图像理解、OCR、目标定位、音视频转写、说话人分离、事件分析和图像分割 |
omni-chatcut | 制作 MV、长篇影视解说以及视频语音翻译 |
omni-video2note | 将视频教程转化为带有关键截图的 PDF 笔记 |
omni-skill-creator | 将教学视频、屏幕录制或操作演示转化为可复用的 Agent Skill.md 文件 |
omni-memory | 为长视频中的人物、对话、声音和事件构建记忆 |
安装完成后，重启 agent harness 或创建一个新任务。
@song.mp3 Generate a complete MV based on the song's rhythm and content.
@short_drama.mp4 Translate the video into English while preserving the original speakers' voice characteristics where possible.
@movie.mp4 Create a film commentary video with Chinese narration and subtitles.
@weekly_report_sop.mp4 Turn this screen recording of writing a weekly report into a Skill.md file.
@tutorial.mp4 Turn the tutorial into PDF notes with key screenshots, timestamps, and step-by-step instructions.
@documentary.mp4 Build audio-visual memory that records people, dialogue, sounds, and important events.
Qwen3.8-Omni-Flash-Realtime 快速上手#
API 用法#
Qwen3.8-Omni-Flash-Realtime 支持通过 WebSocket 和 WebRTC 连接。运行下面的基础示例会打开摄像头和麦克风，进行实时音视频对话。我们建议使用耳机。
# Run pip install websocket-client pyaudio dashscope opencv-python -U to install dependencies
"""
Environment variables:
DASHSCOPE_API_KEY: Your API Key from https://platform.qianwenai.com/home
DASHSCOPE_BASE_URL: (optional) Base URL for realtime API.
- Beijing: wss://dashscope.aliyuncs.com/api-ws/v1/realtime
- Singapore: wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime
"""
import os
import base64
import time
import pyaudio
import cv2
from dashscope.audio.qwen_omni import MultiModality, AudioFormat,OmniRealtimeCallback,OmniRealtimeConversation
import dashscope
url = f'wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime'
dashscope.api_key = os.getenv('DASHSCOPE_API_KEY')
# Determine the voice
voice = 'Tina'
# Determine the model
model = 'qwen3.8-omni-flash-realtime'
# Determine the model role
instructions = "You are Qwen-Omni, a helpful assistant."
video_fps = 1
video_size = (1280, 720)
class SimpleCallback(OmniRealtimeCallback):
def __init__(self, pya):
self.pya = pya
self.out = None
def on_open(self):
# Initialize audio output stream
self.out = self.pya.open(
format=pyaudio.paInt16,
channels=1,
rate=24000,
output=True
)
def on_event(self, response):
if response['type'] == 'response.audio.delta':
# Play audio
self.out.write(base64.b64decode(response['delta']))
elif response['type'] == 'conversation.item.input_audio_transcription.delta':
# Streaming preview: text is the confirmed prefix, stash is the confirmed suffix
preview = response.get('text', '') + response.get('stash', '')
print(f"\r[User] {preview}", end='', flush=True)
elif response['type'] == 'conversation.item.input_audio_transcription.completed':
# Transcription completed, print the final text and a new line
print(f"\r[User] {response['transcript']}")
elif response['type'] == 'response.audio_transcript.done':
# Print the assistant's response text
print(f"[LLM] {response['transcript']}")
# 1. Initialize audio device
pya = pyaudio.PyAudio()
# 2. Create callback function and session
callback = SimpleCallback(pya)
conv = OmniRealtimeConversation(model=model, callback=callback, url=url)
# 3. Establish connection and configure session
conv.connect()
conv.update_session(output_modalities=[MultiModality.AUDIO, MultiModality.TEXT], voice=voice, instructions=instructions)
# 4. Initialize audio input stream
mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
camera = cv2.VideoCapture(0)
next_frame_at = 0.0
# 5. Main loop to process audio and video input
try:
if not camera.isOpened():
raise RuntimeError("Cannot open camera 0.")
camera.set(cv2.CAP_PROP_FRAME_WIDTH, video_size[0])
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, video_size[1])
camera.set(cv2., 1)
print(f"Conversation started with camera ({video_fps} fps, {video_size[0]}x{video_size[1]}), speak into the microphone (Ctrl+C to exit)...")
while True:
audio_data = mic.read(3200, exception_on_overflow=False)
conv.append_audio(base64.b64encode(audio_data).decode())
success, frame = camera.read()
if not success:
raise RuntimeError("Cannot read a camera frame.")
if time.monotonic() >= next_frame_at:
frame = cv2.resize(frame, video_size)
success, image = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 80])
if not success:
raise RuntimeError("Cannot encode a camera frame.")
conv.append_video(base64.b64encode(image).decode())
next_frame_at = time.monotonic() + 1 / video_fps
time.sleep(0.01)
except KeyboardInterrupt:
pass
finally:
# Clean up resources
camera.release()
conv.close()
mic.close()
if callback.out:
callback.out.close()
pya.terminate()
print("\nConversation ended")
Qwen-Live Harness#
Qwen-Live Harness 是一个围绕 Qwen3.8-Omni-Flash-Realtime API 构建的综合性开源 harness。它可以通过一条命令完成安装，并集成到主流智能体工作流中。它支持任务委派、主动交互、长期记忆和上下文管理，并欢迎社区贡献。
图 2. Qwen-Live Harness 交互框架。
安装并开始使用：
npm install -g qwen-live-harness
qwen-live-harness init
qwen-live-harness
引用#
@misc{qwen38omniflash,
title = {Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery.},
url = {https://qwen.ai/blog?id=qwen3.8-omni-flash},
author = {{Qwen Team}},
month = {September},
year = {2026}
}
QWEN-LIVE HARNESS
QWEN-MM-PLUGINS
QWEN3.8-OMNI-FLASH API
QWEN3.8-OMNI-FLASH-REALTIME API
Introduction#
Today, we are launching Qwen3.8-Omni-Flash, our next-generation native omnimodal model. Its core objective is to strengthen agent capabilities in real-world productivity scenarios, advancing omnimodal models from “understanding omnimodal content” to “planning tasks, calling tools, and completing creative work.” Building on general agentic capabilities in coding, text-based knowledge work, and GUI operation, Qwen3.8-Omni-Flash further extends agentic applications centered on audio and video, delivering strong results across workflows such as video editing, music video creation, film production and commentary, audio-visual summarization, and real-time conversations.
Figure 1. Qwen3.8-Omni-Flash and its applications in production.
- Qwen3.8-Omni-Flash — now available on the Qianwen AI Platform:
- Text, image, audio, and video inputs with a 1M-token context window.
Qwen3.8-Omni-Flash supports a 1M-token context window while maintaining text performance comparable to a text-only model of the same size and delivering significant improvements in omnimodal capabilities. Across 29 evaluations1, its average score improves by more than 25% over Qwen3.5-Omni-Plus; the API price per hour of audio input decreases by more than 98%, and the price per hour of audio-visual input decreases by more than 93%2. For audio-visual agents, coding, and long-horizon tasks, the model improves by 36.5 points on WildClawBench-MM and 22.3 points on AgenticVBench, while scoring a strong 69.6 on UniClawBench. Its core capabilities also improve significantly in long-form audio and audio-visual understanding, audio-visual reasoning, audio-visual captioning, and multi-speaker recognition. For example, it gains 8.3 points on LongAudioSpan and 9.6 points on OmniVideoBench; its OmniCap-IF CSR and ISR improve by 8.5 and 14.1 points, respectively; and its AliMeeting DER and cpWER decrease from 88.11 / 89.61 to 3.35 / 17.18. By scaling data, context, and agentic environments, Qwen3.8-Omni-Flash achieves audio-visual performance close to Gemini 3.8 Flash and overall audio performance that exceeds Gemini 3.8 Flash. These advances also mean that audio and video are evolving from perceptual inputs into core media through which agents understand their environment, reason, and execute tasks.
1. The scope includes audio reasoning benchmarks: AliMeeting-test, AISHELL-4, MagicData-RAMC, MLC-SLM (en), WenetSpeech (Net | Meeting), FLEURS-60 ASR, FLEURS-60 S2TT, SpotSoundBench, MMAU, MMAR, MMSU, MuchoMusic-RUL, HumMusQA, MusTBench, Audio-MultiChallenge, WildSpeech, and VoiceBench; audio-visual reasoning benchmarks: DailyOmni, WorldSense, AVUT, JointAVBench, OmniCloze, OmniCap-IF, QIVD, OmniVideoBench, and StreamingBench; and audio-visual agent benchmarks: WildClawBench-MM, UniClawBench, and OmniGAIA.
2. Pricing methodology: hourly audio or audio-visual input prices are estimated as 30 times the input cost of two minutes of source material; audio-visual input uses 720p at 1 fps. Gemini 3.8 Flash uses media_resolution=high, Seed 2.0 Lite uses max_frame_tokens=384, and all other API parameters use their default values. Text input and output prices are in CNY per 1M tokens; Gemini and Muse prices are converted from USD at an exchange rate of 1 USD = 6.7191 CNY.
Audio and video are important media for bringing agents into real-world productivity scenarios, but they also introduce a new set of system-level challenges. Long-form audio and video are costly to store, transmit, and process across multiple rounds of inference; existing agent harness frameworks lack native support for these modalities; and workflows that connect omnimodal understanding with end-to-end task execution are still at an early stage. Addressing these challenges requires models, harness tools, and runtime environments to evolve together.
To address these challenges, we use Qwen3.8-Omni-Flash to explore how to connect source understanding, task planning, tool execution, and result delivery into a complete pipeline. It supports end-to-end, long-horizon workflows such as video editing, translation, film commentary, and content creation, advancing Omni from audio-visual understanding toward autonomous action and task completion.
To this end, we have further expanded Qwen-MM-Plugins with on-demand perception, tool use, and workflow execution for long-form audio and video. We have also open-sourced Qwen-Live Harness as a native runtime for continuous, real-time omnimodal interaction. Together, they address long-horizon workflows and real-time interaction while continuing to expand the capabilities of omnimodal agents alongside the model.
Long-Form Audio-Visual Understanding#
Qwen3.8-Omni-Flash brings a major upgrade to long-form audio-visual understanding—from controllable descriptions and agentic evidence gathering, to understanding meetings and advancing follow-up tasks, and finally to producing video-centered deep research reports. It does not merely process longer content, but finds relevant evidence more precisely, reasons more deeply, and acts more efficiently.
Controllable Audio-Visual Captioning#
There is no single answer to how a video should be described. Content creation prioritizes narrative, footage retrieval focuses on specific segments, and asset management depends on structure. Different applications need different video descriptions. In Qwen3.8-Omni-Flash, we have upgraded video captioning from answering "what the model saw" to understanding "what the user wants to know." Users can freely specify the subject, time range, level of detail, and output format. For the same video, the model can provide an overview, locate key segments, or analyze character actions, camera shots, lighting, and sound in depth, producing structured results as needed. You define what to look at, how closely to look, and how to present it.
Agentic Long-Form Audio-Visual Understanding#
For videos lasting several hours, conventional approaches require the model to process the entire recording from beginning to end, even when the answer appears in only a few minutes of footage. The native Qwen3.8-Omni-Flash agent starts from the question, independently decides what to watch and listen to, and locates key information through multiple rounds of coarse-to-fine evidence gathering. Without processing every frame, it can focus limited compute and token budgets on the relevant segments, enabling more efficient long-form video understanding. On OmniVideoBench, Agentic Understanding improves accuracy from 63.4 to 67.8 while reducing token consumption from 145,736 to 79,117, a reduction of approximately 45.7%. The table below compares the accuracy and token consumption of Static Understanding and Agentic Understanding on OmniVideoBench:
| Static Understanding | Agentic Understanding | |
|---|---|---|
| Accuracy (↑) | 63.4 | 67.8 |
| Tokens per query (↓) | 145,736 | 79,117 |
Long Meetings: From Minutes to Action#
Multi-participant meetings are among the most complex audio-visual understanding scenarios: speakers take turns and overlap, while identities, references, and discussion topics continuously change. Qwen3.8-Omni-Flash jointly recognizes speakers across audio and video and natively supports up to one hour of audio-visual input. It can perform speaker segmentation, content transcription, and identity alignment end to end. Given a complete meeting video and a request, the model can map participant relationships, generate meeting minutes, identify action items, and analyze project risks, using visual information to resolve references and entity ambiguity in the audio. Combined with agents and tool use, it can also send emails, organize tasks, and even begin coding in response to meeting requirements—moving from understanding a meeting to acting on it.
Conducting Deep Research with Audio and Video#
When users watch a video with a specific question in mind, the answer often extends beyond the video itself. Qwen3.8-Omni-Flash combines the user’s needs with the video content to identify questions worth deeper investigation, organize the key material, and search multimodal sources across the web—including images, videos, and documents. It then produces a video-centered, richly illustrated research report that helps users understand the content and solve practical problems. For example, when a user encounters color fringing around a Photoshop hair cutout, the model can break down the tutorial steps, study the principles behind Multiply and Screen blend modes, compare alternative edge-repair techniques, and explain which approach best fits the user’s situation.
Audio-Visual Production and Editing#
Qwen3.8-Omni-Flash is taking audio-visual agents into a new stage: from understanding sounds and images to independently planning, calling tools, and delivering finished videos, bringing omnimodal intelligence into professional audio-visual content production workflows.
Music2MV#
For music video (MV) creation, Qwen3.8-Omni-Flash can understand the structure, rhythm, mood, vocals, and instrumental changes of a user-provided song in fine detail, informing the design of characters, scenes, and shots. It can also output line-level lyrics with timestamps to align singing, subtitles, and visuals. Combined with creative tools such as Qwen-MM-Plugins, the model supports the complete workflow from music understanding and creative planning to final quality review, demonstrating strong audio-visual understanding, reasoning, and creation capabilities.
Workflow
Next
Final Result 1
Next
Final Result 2
Next
Final Result 3
Next
Short Drama Translation#
Traditional video translation often requires repeatedly switching between transcription, translation, dubbing, and editing platforms. This complicates API calls and workflow coordination and makes it difficult to maintain consistency across character voices, dialogue duration, and visual pacing. With an agent built on Qwen3.8-Omni-Flash, users can describe their needs in a single sentence to perform speaker-aware dialogue recognition, conversational translation, character voice cloning and dubbing, audio remixing, and final quality review. These otherwise fragmented localization steps become a complete workflow, enabling automated delivery of short dramas for international audiences.
Workflow
Next
Final Result 1
Next
Source
Translated Result
Final Result 2
Next
Source
Translated Result
Final Result 3
Next
Source
Translated Result
Long-Form Film Commentary#
Producing commentary videos for full-length films of two or three hours often requires repeatedly watching the film and reconstructing its plot, followed by shot selection, scriptwriting, voiceover, music, and editing. This is a complex and time-consuming process. With an agent built on Qwen3.8-Omni-Flash, users need only provide a film and describe their creative requirements in one sentence to perform long-form video understanding, key-plot extraction, commentary planning, voiceover and music production, editing, rendering, and final quality review. The agent can also intelligently interleave original dialogue with commentary, automatically adjusting speech rate and volume so that narration, original audio, background music, and visuals flow naturally together, creating a more authentic, immersive, and cinematic commentary video.
Workflow
Next
Final Result 1
Next
Final Result 2
Next
From Using Models to Optimizing Models#
Real-world multimodal applications are often complex and cost-sensitive, requiring models to balance quality, latency, compute, and deployment costs.
Customizing smaller models for specific scenarios is therefore an important path to deploying applications at scale. Yet traditional workflows involve data construction, problem diagnosis, multiple rounds of training, and evaluation, making them time-consuming and heavily dependent on human expertise. This time, we extend Qwen3.8-Omni-Flash into model development itself, exploring a new approach in which large models drive research and development while smaller models serve business needs.
We gave Qwen3.8-Omni-Flash a task: improve Qwen2.5-Omni-3B's Sichuan dialect speech recognition within 12 hours and deliver a usable model. It independently selected the WenetSpeech-Chuan evaluation set, fixed the evaluation criteria, and established a baseline. It then listened directly to audio samples, diagnosed problems using the recognition results, and constructed targeted training data. Across four consecutive rounds of experiments, the agent created 3,413 training examples, adjusted its approach based on evaluation feedback, retained effective improvements, and rolled back unsuccessful attempts. Qwen2.5-Omni-3B's character error rate on the same evaluation set ultimately fell from 25.79% to 15.30%, a relative reduction of approximately 40.7%.
This experiment demonstrates another possibility for model evolution: general-purpose multimodal models understand data, plan experiments, and drive iteration, while smaller models acquire specialized capabilities for specific applications. Agents can go beyond using models to help solve practical business problems.
Audio-Visual Information Compression#
Audio and video carry rich information, but their linear, unstructured form makes retrieval and reuse difficult. Qwen3.8-Omni-Flash understands content across sound, visuals, and timelines, using an agentic workflow to extract information, reorganize its structure, and verify results. It transforms the core knowledge and practical experience in long videos into denser information assets that are easier to consume and reuse.
Video2Note#
To turn video knowledge into structured resources, we have open-sourced Video2Note in Qwen-MM-Plugins. Drawing on Qwen3.8-Omni-Flash's joint understanding of speech, visuals, and procedures, it automatically organizes knowledge, breaks down key steps, selects representative frames, and generates PDF notes with corresponding text and images. Automated review and iterative correction further condense hours of video into clear, readable documents that are easy to revisit.
Workflow
Next
Final Result 1
Next
Example Video
Final PDF
Final Result 2
Next
Example Video
Final PDF
Final Result 3
Next
Example Video
Final PDF
Final Result 4
Next
Example Video
Final PDF
Omni Skill Creator#
Videos record not only "how to do something," but also the practical expertise accumulated by specialists. With this in mind, we introduce Omni Skill Creator as a new open-source capability in Qwen-MM-Plugins. It can extract standard operating procedures (SOPs) from demonstrations to perform reusable automated work, or learn tool usage, decision criteria, and key insights from expert instruction. A single demonstration becomes an agent skill that has been verified and evaluated, enabling reusable, shareable skills built from omnimodal content.
Real-Time Audio-Visual Interaction#
Qwen3.8-Omni-Flash is designed for deep understanding and creation with complete audio-visual content. For continuous, low-latency interaction, we further introduce Qwen3.8-Omni-Flash-Realtime. It perceives and responds while receiving live audio-visual streams, and uses real-time context to call tools and execute tasks, taking omnimodal capabilities from “understanding a piece of content” to “participating in an interaction.”
Real-Time Speaking Practice#
Spoken language has no standard input. Accents, vowel and consonant substitutions, and tonal deviations can cause word-for-word transcription to diverge from the intended meaning. Qwen3.8-Omni-Flash-Realtime jointly models pronunciation and semantics, understands nonstandard expressions affected by accents, aligns them with the correct words, and generates standard-pronunciation demonstrations in real time. Across multiple practice turns, the model updates its judgment with new audio, correcting errors that affect understanding while preserving natural rhythm, tone, and emotion.
Omnimodal Spatial Audio Perception#
In real-world spaces, sound provides another coordinate axis beyond vision. Qwen3.8-Omni-Flash-Realtime combines spatial sound with visual information to continuously determine the direction and distance of sound sources while perceiving obstacles, navigable areas, and changes in the scene, making it the first omnimodal model capable of locating targets by sound.
For instructions such as “come over here” or “go see what is making that sound,” the model can isolate voices and target sounds from environmental noise, ground their meaning in the surrounding space, and call tools to perform localization, search, path planning, and navigation—from hearing a target to reaching it.
External Knowledge for Audio-Visual Interaction#
Real-time interaction requires not only low latency, but also the ability to load knowledge and behavior dynamically for each application. Qwen3.8-Omni-Flash-Realtime supports injecting identity settings, expression styles, business knowledge, and interaction rules through Skills, while tool use extends these capabilities into task execution.
In scenarios such as customer service, the model can load brand language and service procedures in real time, understand the user's speech, visuals, and context, generate responses that follow business requirements, and execute actions. The same real-time model can therefore take on different knowledge, roles, and ways of acting.
Benchmark Results#
Omni#
| Qwen3.8-Omni-Flash | Qwen3.5-Omni-Plus | Gemini 3.8 Flash | Seed 2.0 Lite | Muse Spark 1.2 | |
|---|---|---|---|---|---|
| Agentic Omni Intelligence | |||||
WildClawBench-MM Multimodal tool use | 71.0 | 34.5 | 58.9 | 41.9 | -- |
UniClawBench Multimodal tool use | 69.6 | 67.1 | 69.0 | 61.2 | -- |
AgenticVBench Multimodal tool use | 36.8 | 14.5 | 45.0 | 10.0 | -- |
OmniGAIA Web Search | 74.0 | 57.2 | 78.6 | 64.4 | -- |
| General Audio-Visual Capabilities | |||||
DailyOmni Audio-Visual Understanding | 85.1 | 85.1 | 84.0 | 81.4 | 79.6 |
WorldSense Audio-Visual Understanding | 68.5 | 63.9 | 69.6 | 67.3 | 65.0 |
AVUT Audio-Visual Understanding | 86.6 | 85.9 | 88.0 | 81.5 | 82.4 |
JoinAVBench Audio-Visual Understanding | 75.9 | 74.1 | 70.4 | 70.6 | 71.8 |
OmniVideoBench Audio-Visual Reasoning | 63.4 | 53.8 | 65.2 | 58.5 | 62.2 |
Video-MME-v2 Audio-Visual Reasoning | 65.0 | 47.9 | 71.0 | 64.9 | -- |
LVOmniBench Long Video Reasoning | 63.3 | 53.2 | 70.7 | -- | -- |
OmniCloze Audio-Visual Caption | 63.2 | 64.2 | 60.9 | 56.3 | 65.3 |
OmniCap-IF Audio-Visual Caption | CSR 80.6 ISR 28.2 | CSR 72.1 ISR 14.1 | CSR 81.9 ISR 28.3 | CSR 74.6 ISR 18.1 | CSR 77.9 ISR 26.8 |
QIVD Audio-Visual Interaction | 69.6 | 65.6 | 69.1 | 62.0 | 62.0 |
StreamingBench Audio-Visual Interaction | 80.8 | 57.1 | 79.9 | 77.2 | 77.8 |
| General Audio Capabilities | |||||
AliMeeting Test Multi-Speaker ASR (DER | cpWER, ↓) | 3.4 | 17.2 | 88.1 | 89.6 | 72.6 | 53.1 | 75.1 | 76.1 | 93.7 | 92.7 |
AISHELL-4 Multi-Speaker ASR (DER | cpWER, ↓) | 2.8 | 11.2 | 100.0 | 100.0 | 66.4 | 56.9 | 64.8 | 64.2 | 91.3 | 86.0 |
MagicData-RAMC Multi-Speaker ASR (DER | cpWER, ↓) | 5.7 | 14.1 | 98.4 | 97.1 | 67.9 | 33.8 | 43.4 | 35.1 | 82.1 | 75.3 |
MLC-SLM (en) Multi-Speaker ASR (DER | cpWER, ↓) | 4.0 | 14.2 | 68.6 | 63.9 | 60.8 | 26.6 | 40.4 | 45.5 | 74.3 | 52.9 |
WenetSpeech (Net) ASR (WER, ↓) | 4.8 | 3.7 | 14.2 | 4.3 | 68.2 |
WenetSpeech (Meeting) ASR (WER, ↓) | 4.6 | 4.8 | 16.7 | 4.7 | 42.6 |
FLEURS-ASR Multilingual ASR (WER, ↓) | 9.3 | 7.2 | 7.9 | 32.1 | 23.6 |
FLEURS-S2TT Multilingual S2TT (BLEU) | 31.8 | 32.2 | 33.0 | 24.8 | 28.8 |
SpotSoundBench Audio Grounding | 67.2 | 64.2 | 39.7 | 59.6 | 16.9 |
MMAU Audio Understanding | 81.8 | 81.9 | 76.9 | 77.2 | 63.5 |
MMAR Audio Understanding | 79.8 | 79.8 | 78.5 | 77.7 | 67.3 |
MMSU Audio Understanding | 82.1 | 83.0 | 83.3 | 80.2 | 59.9 |
LongAudioSpan Long Audio Reasoning | Accuracy 82.7 Rubric 71.8 Chain 48.2 | Accuracy 74.4 Rubric 49.8 Chain 45.1 | Accuracy 79.3 Rubric 65.5 Chain 64.6 | -- | -- |
MuchoMusic-RUL Music Understanding | 72.6 | 71.6 | 53.7 | 61.7 | 40.1 |
HumMusQA Music Understanding | 75.8 | 75.5 | 71.2 | 66.0 | 63.3 |
MusTBench Music Understanding | 50.6 | 49.1 | 40.3 | 44.0 | 29.4 |
Audio MultiChallenge Audio Interaction | 71.5 | 57.6 | 71.9 | 63.4 | 57.9 |
WildSpeech Audio Interaction | 74.3 | 75.7 | 76.4 | 74.5 | 73.4 |
VoiceBench Audio Interaction | 91.6 | 92.9 | 92.3 | 84.1 | 79.8 |
1. Evaluation harnesses for Agentic Omni Intelligence: WildClawBench-MM and AgenticVBench use Claude Code, UniClawBench uses OpenClaw, and OmniGAIA uses no harness. For WildClawBench-MM, we evaluate only the multimodal tasks in WildClawBench that involve images, video, or audio.
2. FLEURS: ASR and S2TT evaluation results both cover the following 60 languages: Chinese (Mandarin), English, Cantonese, Arabic, German, French, Spanish, Portuguese, Indonesian, Italian, Korean, Russian, Thai, Vietnamese, Japanese, Turkish, Hindi, Malay, Dutch, Urdu, Norwegian, Swedish, Danish, Hebrew, Finnish, Polish, Icelandic, Czech, Filipino, Persian, Greek, Afrikaans, Asturian, Belarusian, Bulgarian, Bengali, Bosnian, Catalan, Cebuano, Estonian, Galician, Gujarati, Croatian, Hungarian, Javanese, Kazakh, Kannada, Kyrgyz, Latvian, Macedonian, Malayalam, Marathi, Punjabi, Romanian, Slovak, Slovenian, Swahili, Tajik, Azerbaijani, and Ukrainian.
3. Empty cells (--): scores are not yet available or are not applicable.
Agentic Omni Understanding#
Key information in long audio and video recordings is often scattered across different segments, while complex questions require multiple steps of reasoning across sound and images. Agentic Omni Understanding enables the model to start from the question, plan its approach, call tools, and progressively locate and verify evidence. By focusing computation on relevant content, it improves the accuracy and efficiency of long-form audio-visual understanding. To evaluate this capability, we compare Qwen3.8-Omni-Flash and Gemini 3.8 Flash on OmniVideoBench, Video-MME-v2, and LVOmniBench under two settings: Static, where the model directly interprets the input, and an agent mode using Qwen Code. This comparison shows how introducing agent workflows affects each model’s performance.
| Qwen3.8-Omni-Flash (Static) | Qwen3.8-Omni-Flash (Qwen Code) | Gemini 3.8 Flash (Static) | Gemini 3.8 Flash (Qwen Code) | |
|---|---|---|---|---|
OmniVideoBench Audio-Visual Reasoning | 63.4 | 67.8 | 65.2 | 70.1 |
Video-MME-v2 Audio-Visual Reasoning | 65.0 | 71.3 | 71.0 | 72.7 |
LVOmniBench Long Video Reasoning | 63.3 | 73.6 | 70.7 | 70.7 |
Text#
| Qwen3.8-Omni-Flash | Qwen3.8- Flash | Qwen3.8- 27B | Qwen3.7- Plus | DeepSeek-V4-Flash-0731 | Claude-Opus-4.6 (Max) | |
|---|---|---|---|---|---|---|
| Coding and Agent | ||||||
DeepSWE 1.1 Long-horizon software engineering | 57.8 | 58.7 | 42.2 | 16.5 | 54.4 | -- |
SWE-bench Pro Long-horizon software engineering | 63.3 | 62.5 | 61.7 | 55.8 | 56.0 | 53.4 |
SWE-bench Multilingual Multilingual software engineering | 80.5 | 81.0 | 73.8 | 75.8 | -- | 77.5 |
NL2Repo-Bench Repo-level code generation | 48.9 | 48.1 | 42.3 | 41.1 | 54.2 | 47.6 |
CoWorkBench Long-horizon office work | 75.3 | 73.9 | 70.7 | 65.1 | 45.1 | 68.2 |
| General Text Capabilities | ||||||
IFBench Instruction following | 81.5 | 81.3 | 79.5 | 79.1 | 79.2 | 62.5 |
GPQA Diamond Scientific reasoning | 91.0 | 91.7 | 89.2 | 90.3 | 90.8 | 91.3 |
HLE Multidisciplinary reasoning | 36.5 | 35.9 | 30.8 | 34.7 | 33.8 | 40.0 |
LiveCodeBench v6 Competitive coding | 92.6 | 91.9 | 90.3 | 89.6 | 90.6 | 88.8 |
1. DeepSWE 1.1: evaluated with the Claude Code and mini-SWE-agent harnesses, temp=1.0, top_p=0.95, 256K context window. We report the highest score across the two harnesses; notably, Qwen3.8-Flash-Next performs best on mini-SWE-agent.
2. SWE-bench Pro: except for Claude-Opus-4.6 (Max), for which we report the officially published score, all models are evaluated with the Claude Code harness, temp=1.0, top_p=0.95, 256K context window. Problematic tasks were corrected and all baseline models were re-evaluated on the refined benchmark.
3. SWE-bench Multilingual: evaluated with the mini-SWE-agent harness, temp=1.0, top_p=0.95, 256K context window.
4. NL2Repo-Bench: evaluated with the Claude Code harness. To prevent reward hacking, we disable Bash commands that attempt to access the specific repository, such as pip download, pip install and git clone.
5. CoWorkBench: an in-house cowork benchmark for evaluating long-horizon office and productivity agent tasks across computer science, finance, law, medical and other productivity domains.
6. HLE: judged by GPT-4o.
7. Empty cells (--): scores are not yet available or are not applicable.
Vision#
| Qwen3.8-Omni-Flash | Qwen3.8- Flash | Qwen3.8- 27B | Qwen3.7- Plus | Claude-Opus-4.6 (Max) | |
|---|---|---|---|---|---|
| Agentic Vision Intelligence | |||||
ClawEval-MM Multimodal tool use | Pass@3 60.4 Average 61.9 | Pass@3 64.4 Average 60.4 | Pass@3 57.4 Average 56.9 | Pass@3 57.4 Average 60.1 | Pass@3 52.5 Average 54.7 |
AndroidWorld Mobile use | 87.1 | 84.5 | 81.9 | 81.0 | 62.0 |
Vision2Web Visual web development | 62.9 | 64.0 | 62.9 | 42.1 | -- |
| General Vision Capabilities | |||||
ERQA Embodied intelligence | 71.0 | 72.3 | 65.5 | 69.8 | 40.8 |
LVBench Long video understanding | 76.9 | 76.6 | 72.4 | 76.2 | 63.0 |
RealWorldQA Real-world perception | 87.7 | 88.5 | 85.9 | 86.9 | 73.9 |
MathVision Visual math problem solving | Without CI 91.8 With CI 96.2 | Without CI 90.6 With CI 95.7 | Without CI 90.0 With CI 94.6 | Without CI 90.3 With CI 88.4 | Without CI 65.5 With CI -- |
CharXiv (RQ) Scientific chart analysis | Without CI 83.5 With CI 91.4 | Without CI 84.6 With CI 90.6 | Without CI 83.7 With CI 90.2 | Without CI 85.8 With CI 85.9 | Without CI 66.0 With CI -- |
1. ClawEval-MM: scores are reported as "pass@3 / average score". Pass@3 measures the percentage passed in at least one of three trials, and the average score is the mean score across the three trials.
2. RecreationBench: an in-house long-horizon application-recreation benchmark for evaluating hybrid-agent abilities spanning five platforms — desktop (Ubuntu, macOS, Windows), mobile (Android) and web.
3. OSWorld 2.0: scores are reported as "binary / partial". The binary score is the percentage of tasks that receive the full task reward, while the partial score aggregates the partial rewards obtained across all tasks.
4. Vision2Web: scores are reported as the average over the frontend, webpage and website categories, using the Claude Code harness and judged by gpt-5.4-2026-03-05.
5. MathVision, CharXiv (RQ): scores are reported as "without CI / with CI". A small number of incorrect ground-truth annotations in MathVision were corrected after manual verification. Our model's score is evaluated using a fixed prompt, e.g. "Please reason step by step, and put your final answer within \boxed{}." For other models, we report the higher score between runs with and without the \boxed{} formatting.
6. Empty cells (--) indicate scores not yet available or not applicable.
Throughput & Latency#
The following results show the observed throughput and latency of the Qwen3.8-Omni-Flash-Realtime API under different input conditions, reflecting the performance users experience in production environments.
| Input Scenario | Text Output TPS (Tokens/s) | Time to First Token (ms) | Time to First Audio Packet (ms) | Audio Generation RTF |
|---|---|---|---|---|
| Realtime API Performance | ||||
| Audio 6s | 84.87 | 591.26 | 978.36 | 0.1538 |
| Audio 12s | 83.18 | 604.80 | 982.74 | 0.1537 |
| Audio 20s | 81.06 | 617.98 | 1026.39 | 0.1538 |
| Audio-Visual 6s | 84.89 | 837.96 | 1214.73 | 0.1524 |
| Audio-Visual 12s | 84.33 | 911.85 | 1268.07 | 0.1527 |
| Audio-Visual 20s | 83.00 | 981.01 | 1350.49 | 0.1528 |
Supported Languages#
| Capability | Languages | Chinese Dialects |
|---|---|---|
| Speech Recognition | 74 languages: Afrikaans, Arabic, Asturian, Azerbaijani, Basque, Belarusian, Bengali, Bosnian, Bulgarian, Cantonese, Catalan, Cebuano, Chinese, Croatian, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Interlingua, Italian, Japanese, Javanese, Kannada, Kazakh, Korean, Kyrgyz, Lingala, Latvian, Lithuanian, Macedonian, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Norwegian Bokmål, Norwegian Nynorsk, Oriya, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swahili, Swedish, Tajiki, Tamil, Telugu, Thai, Turkish, Ukrainian, Urdu, Uyghur, and Vietnamese | 39 dialects: Northeastern Mandarin, Guizhou dialect, Guangdong Cantonese, Henan dialect, Hong Kong Cantonese, Shanghainese, Shaanxi dialect, Tianjin dialect, Taiwanese Mandarin, Yunnan dialect, Anhui dialect, Fujian dialect, Gansu dialect, Guangdong Mandarin, Hubei dialect, Hunan dialect, Jiangxi dialect, Shandong dialect, Shanxi dialect, Sichuanese, Guangxi dialect, Hainan dialect, Chongqing dialect, Changsha dialect, Hangzhou dialect, Hefei dialect, Yinchuan dialect, Zhengzhou dialect, Shenyang dialect, Wenzhou dialect, Wuhan dialect, Kunming dialect, Taiyuan dialect, Nanchang dialect, Jinan dialect, Lanzhou dialect, Nanjing dialect, Hakka, and Southern Min |
| Speech Generation | 29 languages: Chinese, English, German, Italian, Portuguese, Spanish, Japanese, Korean, French, Russian, Thai, Indonesian, Arabic, Vietnamese, Turkish, Finnish, Polish, Hindi, Dutch, Czech, Urdu, Tagalog, Swedish, Danish, Hebrew, Icelandic, Malay, Norwegian, and Persian | 7 dialects: Sichuanese, Beijing dialect, Tianjin dialect, Nanjing dialect, Shaanxi dialect, Cantonese, and Southern Min |
Getting Started with Qwen3.8-Omni-Flash#
API Usage#
Qwen3.8-Omni-Flash officially supports reasoning_effort to adjust reasoning depth and control costs:
xhigh(default): for complex tasks that require in-depth analysis.medium: balances accuracy and speed.low: efficient reasoning optimized for speed and cost.
In addition, preserve_thinking is enabled by default across all scenarios for the best out-of-the-box experience.
Qwen3.8-Omni-Flash supports industry-standard protocols, including OpenAI-compatible Chat Completions and Responses APIs. Examples follow:
"""
Environment variables:
DASHSCOPE_API_KEY: Your API Key from https://platform.qianwenai.com/home
DASHSCOPE_BASE_URL: (optional) Base URL for compatible-mode API.
- Beijing: https://dashscope.aliyuncs.com/compatible-mode/v1
- Singapore: https://dashscope-intl.aliyuncs.com/compatible-mode/v1
"""
from openai import OpenAI
import os
api_key = os.environ.get("DASHSCOPE_API_KEY")
if not api_key:
raise ValueError(
"DASHSCOPE_API_KEY is required. "
"Set it via: export DASHSCOPE_API_KEY='your-api-key'"
)
client = OpenAI(
api_key=api_key,
base_url=os.environ.get(
"DASHSCOPE_BASE_URL",
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
),
)
messages=[
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241022/emyrja/dog_and_girl.jpeg"
},
},
{
"type": "input_audio",
"input_audio": {
"data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
"format": "wav"
},
},
{"type": "text", "text": "Please describe the image and tell me what is being said in the audio."},
],
},
]
completion = client.chat.completions.create(
model="qwen3.8-omni-flash",
messages=messages,
extra_body={
"enable_thinking": True,
# "preserve_thinking": True,
},
reasoning_effort="xhigh", # supported levels are xhigh, medium, and low
stream=True,
)
reasoning_content = ""
answer_content = ""
is_answering = False
print("\n" + "=" * 20 + "Reasoning" + "=" * 20 + "\n")
for chunk in completion:
if not chunk.choices:
print("\nUsage:")
print(chunk.usage)
continue
delta = chunk.choices[0].delta
if hasattr(delta, "reasoning_content") and delta.reasoning_content is not None:
if not is_answering:
print(delta.reasoning_content, end="", flush=True)
reasoning_content += delta.reasoning_content
if hasattr(delta, "content") and delta.content:
if not is_answering:
print("\n" + "=" * 20 + "Answer" + "=" * 20 + "\n")
is_answering = True
print(delta.content, end="", flush=True)
answer_content += delta.content
Best Practices for Controllable Audio-Visual Captioning#
Content organization and level of detail: Organize video descriptions, OCR extraction, audio descriptions, and speech transcripts into sections covering visuals, speech, music, sound effects, and ambient sounds. Preserve the original text, speaker identities, and corresponding time ranges to make retrieval and verification easier. Specify a target length in the prompt to control detail, for example, Describe the video in approximately 2000–3000 words., and adjust it to the video’s duration and information density.
Prompt Example
Provide a detailed description of the video.
Make sure your description covers every one of the following dimensions:
Visual
- Subjects and characters: appearance, clothing, gender/age cues, identity, distinctive features
- Actions and events in chronological order, and how the scene evolves over time
- Setting and background: location, environment, time of day
- Spatial layout and relations between subjects/objects; counts and quantities
- On-screen text: captions, titles, subtitles, logos, UI — exact content and appearance
- Visual style: colors, lighting, camera shots, angles, and camera movement
Audio
- Speech: the exact spoken content, transcribed verbatim
- Speakers: who is speaking (mapped to the on-screen person or voice-over), with accent, tone, gender/age cues
- Speaking state: prosody, emotion, volume, and speaking style
- Music: presence, genre/mood, and lyrics if any
- Sound effects and ambient/background sounds
- Non-speech vocalizations: laughter, crying, applause, etc.
Audio-visual correspondence
- Which speech or sound aligns with which on-screen person or visual event
- The timing of each event, expressed with timestamps
It should explicitly include three sections:
1. A structured chronological storyline of **every noticeable audio and visual details**
2. A structured list of all visible text. For each text element, include start timestamp, end timestamp, the exact text content, the appearance characteristics. If no text appears, explicitly state so.
3. A structured speech-to-text transcription, include speaker (corresponding to the character or voice-over in Section 1, including their accent and tone), exact spoken content, start timestamp, end timestamp, and speaking state (prosody, emotion, and style). If no speech appears, explicitly state so.
Aside from these three required sections, you are free to organize any additional content in any way you find helpful. This additional content can include global information about the entire video or localized information about specific moments. You may choose the topic of this extra content freely.
Rules:
- Add as much descriptive detail as possible.
- Do not use Markdown bold formatting.
- Carefully look at frames and listen to the audio, making sure no detail is overlooked.
Output Format:
```
## Storyline
<xx:xx.xxx> - <xx:xx.xxx>
<an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.>
<xx:xx.xxx> - <xx:xx.xxx>
<an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.>
<xx:xx.xxx> - <xx:xx.xxx>
<an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.>
...
## Visible Text
<xx:xx.xxx> - <xx:xx.xxx>
“<element>”: <appearance>
“<element>”: <appearance>
<xx:xx.xxx> - <xx:xx.xxx>
“<element>”: <appearance>
“<element>”: <appearance>
“<element>”: <appearance>
<xx:xx.xxx> - <xx:xx.xxx>
“<element>”: <appearance>
...
## Speakers and Transcript
Speaker profiles:
<speaker> - <profile>
<speaker> - <profile>
<speaker> - <profile>
...
<xx:xx.xxx> - <xx:xx.xxx>
Speaker: <speaker>
State: <description>
Content: “<content>”
<xx:xx.xxx> - <xx:xx.xxx>
Speaker: <speaker>
State: <description>
Content: “<content>”
<xx:xx.xxx> - <xx:xx.xxx>
Speaker: <speaker>
State: <description>
Content: “<content>”
...
## <another section>
<paragraphs>
## <another section>
<paragraphs>
...
```
Structured output and schema constraints: Include task instructions and a complete JSON Schema in the prompt, specifying field meanings, types, required fields, and constraints to support programmatic parsing and downstream use.
Prompt Example
Describe the audio and visual content in detail in English, organized into scenes and events, following the JSON Schema below.
All timestamps must be relative to the beginning of the video. End times must not precede start times or exceed the video duration. Each event must fall within the time range of its parent scene.
Include only information directly supported by the audio or video. Do not guess or invent details. Do not infer causality merely because a sound and an action occur at the same time.
Return only valid JSON, without Markdown fences or commentary.
JSON Schema:
{
"$defs": {
"Event": {
"additionalProperties": false,
"properties": {
"time_range": {
"$ref": "#/$defs/TimeRange",
"description": "Time range of the event"
},
"participants": {
"description": "People, animals, or objects involved, named by observable features; use consistent names for the same participant",
"items": {
"type": "string"
},
"title": "Participants",
"type": "array"
},
"action": {
"description": "Specific actions, interactions, and observable outcomes",
"title": "Action",
"type": "string"
},
"sounds": {
"description": "Sounds heard during the event; use an empty list if none are discernible",
"items": {
"type": "string"
},
"title": "Sounds",
"type": "array"
}
},
"required": [
"time_range",
"participants",
"action",
"sounds"
],
"title": "Event",
"type": "object"
},
"Scene": {
"additionalProperties": false,
"properties": {
"time_range": {
"$ref": "#/$defs/TimeRange",
"description": "Time range of the scene"
},
"setting": {
"description": "Environment, spatial layout, and main visual features",
"title": "Setting",
"type": "string"
},
"events": {
"description": "Events in chronological order; use an empty list if there are none",
"items": {
"$ref": "#/$defs/Event"
},
"title": "Events",
"type": "array"
}
},
"required": [
"time_range",
"setting",
"events"
],
"title": "Scene",
"type": "object"
},
"TimeRange": {
"additionalProperties": false,
"properties": {
"start_seconds": {
"description": "Start time in seconds relative to the beginning of the video",
"minimum": 0,
"title": "Start Seconds",
"type": "number"
},
"end_seconds": {
"description": "End time in seconds; must not precede the start time",
"minimum": 0,
"title": "End Seconds",
"type": "number"
}
},
"required": [
"start_seconds",
"end_seconds"
],
"title": "TimeRange",
"type": "object"
}
},
"additionalProperties": false,
"properties": {
"summary": {
"description": "An overview of the main content of the video",
"title": "Summary",
"type": "string"
},
"scenes": {
"description": "Scenes in chronological order; group continuous footage with a consistent setting into one scene",
"items": {
"$ref": "#/$defs/Scene"
},
"title": "Scenes",
"type": "array"
}
},
"required": [
"summary",
"scenes"
],
"title": "CaptionResult",
"type": "object"
}
Installing Qwen-MM-Plugins in an Agent Harness#
Qwen-MM-Plugins is a multimodal plugin suite for agent harnesses. It gives agents the ability to understand images, audio, video, and documents, as well as maintain memory for long-form video and create content. It supports agent harnesses including Codex, Claude Code, Qwen Code, Gemini CLI, Qoder, CodeBuddy, and OpenClaw. We also welcome contributions from community developers.
You can enter the following request directly in your usual office agent:
Help me install the core, api, and omni-related plugins from https://github.com/QwenLM/Qwen-MM-Plugins.
Alternatively, install from the command line:
# Run the official guided installer:
curl -fsSL https://raw.githubusercontent.com/QwenLM/Qwen-MM-Plugins/main/install.sh | bash
In the menu, select:
- Install
- The agent harness you use
- The Omni plugins you need
| Plugin | Capability |
|---|---|
core | Read images, video frames, PDFs, Office documents, code, data, and 3D files |
api | Image understanding, OCR, object localization, audio-visual transcription, speaker diarization, event analysis, and image segmentation |
omni-chatcut | Create MVs, long-form film commentary, and video speech translations |
omni-video2note | Turn video tutorials into PDF notes with key screenshots |
omni-skill-creator | Turn instructional videos, screen recordings, or operation demonstrations into reusable Agent Skill.md files |
omni-memory | Build memory for people, dialogue, sounds, and events in long-form videos |
After installation, restart the agent harness or create a new task.
@song.mp3 Generate a complete MV based on the song's rhythm and content.
@short_drama.mp4 Translate the video into English while preserving the original speakers' voice characteristics where possible.
@movie.mp4 Create a film commentary video with Chinese narration and subtitles.
@weekly_report_sop.mp4 Turn this screen recording of writing a weekly report into a Skill.md file.
@tutorial.mp4 Turn the tutorial into PDF notes with key screenshots, timestamps, and step-by-step instructions.
@documentary.mp4 Build audio-visual memory that records people, dialogue, sounds, and important events.
Getting Started with Qwen3.8-Omni-Flash-Realtime#
API Usage#
Qwen3.8-Omni-Flash-Realtime supports connections over WebSocket and WebRTC. Running the basic examples below opens the camera and microphone for a real-time audio-visual conversation. We recommend using headphones.
# Run pip install websocket-client pyaudio dashscope opencv-python -U to install dependencies
"""
Environment variables:
DASHSCOPE_API_KEY: Your API Key from https://platform.qianwenai.com/home
DASHSCOPE_BASE_URL: (optional) Base URL for realtime API.
- Beijing: wss://dashscope.aliyuncs.com/api-ws/v1/realtime
- Singapore: wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime
"""
import os
import base64
import time
import pyaudio
import cv2
from dashscope.audio.qwen_omni import MultiModality, AudioFormat,OmniRealtimeCallback,OmniRealtimeConversation
import dashscope
url = f'wss://dashscope-intl.aliyuncs.com/api-ws/v1/realtime'
dashscope.api_key = os.getenv('DASHSCOPE_API_KEY')
# Determine the voice
voice = 'Tina'
# Determine the model
model = 'qwen3.8-omni-flash-realtime'
# Determine the model role
instructions = "You are Qwen-Omni, a helpful assistant."
video_fps = 1
video_size = (1280, 720)
class SimpleCallback(OmniRealtimeCallback):
def __init__(self, pya):
self.pya = pya
self.out = None
def on_open(self):
# Initialize audio output stream
self.out = self.pya.open(
format=pyaudio.paInt16,
channels=1,
rate=24000,
output=True
)
def on_event(self, response):
if response['type'] == 'response.audio.delta':
# Play audio
self.out.write(base64.b64decode(response['delta']))
elif response['type'] == 'conversation.item.input_audio_transcription.delta':
# Streaming preview: text is the confirmed prefix, stash is the confirmed suffix
preview = response.get('text', '') + response.get('stash', '')
print(f"\r[User] {preview}", end='', flush=True)
elif response['type'] == 'conversation.item.input_audio_transcription.completed':
# Transcription completed, print the final text and a new line
print(f"\r[User] {response['transcript']}")
elif response['type'] == 'response.audio_transcript.done':
# Print the assistant's response text
print(f"[LLM] {response['transcript']}")
# 1. Initialize audio device
pya = pyaudio.PyAudio()
# 2. Create callback function and session
callback = SimpleCallback(pya)
conv = OmniRealtimeConversation(model=model, callback=callback, url=url)
# 3. Establish connection and configure session
conv.connect()
conv.update_session(output_modalities=[MultiModality.AUDIO, MultiModality.TEXT], voice=voice, instructions=instructions)
# 4. Initialize audio input stream
mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
camera = cv2.VideoCapture(0)
next_frame_at = 0.0
# 5. Main loop to process audio and video input
try:
if not camera.isOpened():
raise RuntimeError("Cannot open camera 0.")
camera.set(cv2.CAP_PROP_FRAME_WIDTH, video_size[0])
camera.set(cv2.CAP_PROP_FRAME_HEIGHT, video_size[1])
camera.set(cv2., 1)
print(f"Conversation started with camera ({video_fps} fps, {video_size[0]}x{video_size[1]}), speak into the microphone (Ctrl+C to exit)...")
while True:
audio_data = mic.read(3200, exception_on_overflow=False)
conv.append_audio(base64.b64encode(audio_data).decode())
success, frame = camera.read()
if not success:
raise RuntimeError("Cannot read a camera frame.")
if time.monotonic() >= next_frame_at:
frame = cv2.resize(frame, video_size)
success, image = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 80])
if not success:
raise RuntimeError("Cannot encode a camera frame.")
conv.append_video(base64.b64encode(image).decode())
next_frame_at = time.monotonic() + 1 / video_fps
time.sleep(0.01)
except KeyboardInterrupt:
pass
finally:
# Clean up resources
camera.release()
conv.close()
mic.close()
if callback.out:
callback.out.close()
pya.terminate()
print("\nConversation ended")
Qwen-Live Harness#
Qwen-Live Harness is a comprehensive open-source harness designed around the Qwen3.8-Omni-Flash-Realtime API. It can be installed with a single command and integrated into mainstream agent workflows. It supports task delegation, proactive interaction, long-term memory, and context management, and welcomes contributions from the community.
Figure 2. Qwen-Live Harness Interaction Framework.
Install and get started:
npm install -g qwen-live-harness
qwen-live-harness init
qwen-live-harness
Citation#
@misc{qwen38omniflash,
title = {Qwen3.8-Omni-Flash: Omni Senses. Agentic Delivery.},
url = {https://qwen.ai/blog?id=qwen3.8-omni-flash},
author = {{Qwen Team}},
month = {September},
year = {2026}
}