OmniVChat:原生音视频对话的合成、评测与训练

HuggingFace Daily Papers(社区热门论文)·2026-09-18 08:00·3天前
AI 导读

研究提出 OmniVChat 任务,让 omni 模型直接同时接收用户音频与视频并返回文本,无需独立文本提问、外部描述或语音识别。为此构建多智能体数据引擎 OmniVChat-Studio 合成单轮与多轮音视频对话,并推出覆盖五类能力的评测基准 OmniVChat-Bench。

HuggingFace Daily Papers(社区热门论文)
43AI 编辑部评分,满分 100

OmniVChat:原生音视频对话的合成、评测与训练

2026-09-18 08:00· 3天前
AI 导读

研究提出 OmniVChat 任务,让 omni 模型直接同时接收用户音频与视频并返回文本,无需独立文本提问、外部描述或语音识别。为此构建多智能体数据引擎 OmniVChat-Studio 合成单轮与多轮音视频对话,并推出覆盖五类能力的评测基准 OmniVChat-Bench。

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce.

Furthermore, a good reply often needs to account for the user's surroundings, facial expressions, and nearby objects, and such responses can be expressed in many different ways, making keyword matching unreliable for evaluating reply quality. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories.

We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org