Omni-Streaming Thinking:让流式全模态模型不再过早跨模态下结论

HuggingFace Daily Papers(社区热门论文)·2026-09-14 08:00·1天前
AI 导读

针对流式全模态模型在音频尚未说完时就把视觉解读当作事实、后续即使音频矛盾仍继续沿用的问题(即"过早跨模态承诺"),研究者提出 Omni-Streaming Thinking(OST),将输出拆为已观察证据、未来证据预测与待验证主张,并按模态分别存储证据、在验证区间结束时校验与反驳。

HuggingFace Daily Papers(社区热门论文)
46AI 编辑部评分,满分 100

Omni-Streaming Thinking:让流式全模态模型不再过早跨模态下结论

2026-09-14 08:00· 1天前
AI 导读

针对流式全模态模型在音频尚未说完时就把视觉解读当作事实、后续即使音频矛盾仍继续沿用的问题(即"过早跨模态承诺"),研究者提出 Omni-Streaming Thinking(OST),将输出拆为已观察证据、未来证据预测与待验证主张,并按模态分别存储证据、在验证区间结束时校验与反驳。

Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence.

Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average.

We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org