一个具备异步委派能力的全双工交互系统
Venus 团队 · 蚂蚁集团 · 清华大学
概览
实时看、听、回应——同时让后台任务与对话并行运行。Realtime-Venus 将全双工对话模型与异步执行框架相结合。你可以让助手处理某项任务，并在其运行期间继续对话。结果会返回到同一段对话中，以语音形式播报。
该项目汇集了三个组件：
| 组件 | 作用 |
|---|---|
| Realtime-Venus-Omni · 9B | 一个面向流式音频与视频、主动交互以及原生语音生成的对话模型。 |
| Realtime-Venus-Audio · 9B | 一个单独训练的模型，用于语音交互、音频理解和原生语音生成。 |
| Realtime-Venus-Harness | 一个共享运行时，负责捕获委派的请求、执行后台工作，并将结果返回给发起该请求的对话。 |
Realtime-Venus 能实现什么
- 持续的视听交互。 Omni 观察流式摄像头和麦克风输入，并能在不断变化的场景需要时主动发起响应。
- 全双工对话。 模型在说话的同时处理传入的语音，学习区分确认应答和背景语音与需要修改回复的打断。
- 异步委派。 模型可以将自然语言任务交给 Harness 进行推理或工具执行，同时实时交互继续进行。
- 结果回到同一对话中。 Harness 准备回复；对话模型负责其时机和原生语音输出。工作追踪将任务完成与语音交付分离。
该论文介绍了该模型系列、双循环运行时、训练数据和评估。本次源码发布包含 Omni 模型集成、可复用的 Harness 包，以及一个使用 Codex 任务后端的浏览器演示。该演示的纯麦克风模式同样使用 Omni。
快速开始
1. 安装依赖
在仓库根目录下运行这些命令。如需独立推理，请使用带 CUDA 和 FFmpeg 的 Python 3.10，并安装你所使用模型的依赖：
# Realtime-Venus-Omni
python -m pip install -r /path/to/Realtime-Venus-Omni/requirements.txt
# Realtime-Venus-Audio
python -m pip install -r requirements.txt
如需运行浏览器演示，请在 Linux 上运行安装脚本。它会创建一个隔离的 Python 3.11/3.12 环境，并安装模型、Web 服务和 Harness 依赖：
bash install.sh
然后选择独立推理或下方的在线体验。
2. 离线推理 · 前端使用
从上方链接下载模型 checkpoint，并将示例路径替换为你本地的目录。
Realtime-Venus-Omni
export REALTIME_VENUS_MODEL_PATH=/path/to/Realtime-Venus-Omni
python frontend/Realtime-Venus-Omni/offline_chat.py
# With long-video Memory
python frontend/Realtime-Venus-Omni/offline_memory_chat.py
这些示例从 checkpoint 的 assets/ 目录读取视频，并输出文本和语音。如需改为对录制输入进行全双工推理：
python frontend/Realtime-Venus-Omni/duplex_chat.py
python frontend/Realtime-Venus-Omni/duplex_speech_in_chat.py
python frontend/Realtime-Venus-Omni/duplex_memory_chat.py
双工示例会保存带字幕的视频。输入与输出路径参见Omni 指南。
Realtime-Venus-Audio
python frontend/Realtime-Venus-Audio/audio_offline_chat.py \
--model-path /path/to/Realtime-Venus-Audio \
--audio frontend/Realtime-Venus-Audio/case/case_offline.wav
离线推理返回文本。要对录制的音频运行全双工推理并保存为 24 kHz WAV：
python frontend/Realtime-Venus-Audio/audio_duplex_chat.py \
--model-path /path/to/Realtime-Venus-Audio \
--audio frontend/Realtime-Venus-Audio/case/case_duplex.wav \
--output output/duplex_response.wav
两个脚本都接受--system-prompt和--prompt；解码选项参见音频指南。
3. 在线双工体验
在线双工演示需要至少一块 NVIDIA A100 GPU。将完整的 Omni checkpoint 放入model_weight/，然后启动演示：
bash start.sh
对于位于其他位置的现有 checkpoint：
bash start.sh --model-path /path/to/Realtime-Venus-Omni
如有提示，请完成 Codex 登录。启动器会在8031上启动模型 API，并在8032上启动浏览器服务。
如果在远程服务器上运行，请在本地计算机上保持此隧道开启，并将user@server替换为你的 SSH 登录信息：
ssh -N -L 8032:127.0.0.1:8032 user@server
打开 http://localhost:8032，选择 语音、摄像头 + 麦克风 或 上传视频，然后开始对话。Checkpoint 布局、配置和服务管理详见 演示指南。
结果
图 1. 来自 论文 的视频与音频理解结果。
图 2. 来自 论文 的全双工交互结果。
代码结构图
Realtime-Venus/
├── harness/ # Context, routing, agents, work state, and delivery
│ ├── README.md
│ └── requirements.txt # Harness media dependencies
├── frontend/
│ ├── Realtime-Venus-Omni/ # Audiovisual inference examples
│ └── Realtime-Venus-Audio/ # Audio inference examples and input samples
├── demos/
│ ├── model/ # Checkpoint adapter and model HTTP API
│ ├── server/ # Web sessions, media, settings, and artifacts
│ ├── static/ # Browser UI, capture, playback, and logos
│ ├── launcher/ # Configuration and process supervision
│ ├── settings.py # Saved task and model preferences
│ ├── install.py # Isolated environment installation
│ └── requirements.txt # Model and application dependencies
├── assets/ # Paper figures, report, and Demo screenshot
├── pyproject.toml # Standalone Harness package
├── requirements.txt # Shared dependency entry for inference examples
├── config.example.json # Server configuration template
└── install.sh / start.sh # Install, start, inspect, and stop the Demo
引用
@article{zhao2026realtime,
title={{Realtime-Venus}: A full-duplex interaction system with asynchronous delegation},
author={{Venus Team,Ant Group;Tsinghua University}},
journal={arXiv preprint arXiv:2609.13814},
year={2026}
}
许可证
本仓库中的源代码依据 Apache License 2.0 授权，带有单独许可声明的组件除外。第三方字体和论文插图保留各自的许可证；请参阅 demos/static/fonts/ 和 assets/README.md 中的许可证文件。
A full-duplex interaction system with asynchronous delegation
Venus Team · Ant Group · Tsinghua University
Overview · Quick start · Results · Code map · Citation
Overview
See, listen, and respond in real time—with background tasks running alongside the conversation. Realtime-Venus combines full-duplex conversational models with an asynchronous execution framework. You can ask the assistant to work on a task and continue talking while it runs. The result returns to the same conversation for spoken delivery.
The project brings together three components:
| Component | Role |
|---|---|
| Realtime-Venus-Omni · 9B | A conversational model for streaming audio and video, proactive interaction, and native speech generation. |
| Realtime-Venus-Audio · 9B | A separately trained model for spoken interaction, audio understanding, and native speech generation. |
| Realtime-Venus-Harness | A shared runtime that captures delegated requests, executes background work, and returns results to the originating conversation. |
What Realtime-Venus enables
- Continuous audiovisual interaction. Omni observes streaming camera and microphone input and can initiate a response when the evolving scene calls for it.
- Full-duplex conversation. The models process incoming speech while speaking, learning to distinguish acknowledgments and background speech from interruptions that call for a revised response.
- Asynchronous delegation. A model can hand a natural-language task to Harness for reasoning or tool execution while live interaction continues.
- Results in the same conversation. Harness prepares a reply; the conversational model handles its timing and native speech output. Work tracking separates task completion from spoken delivery.
The paper describes the model family, dual-loop runtime, training data, and evaluation. This source release includes the Omni model integration, the reusable Harness package, and a browser demo using a Codex task backend. The demo's microphone-only mode also uses Omni.
Quick start
1. Install dependencies
Run the commands from the repository root. For standalone inference, use Python 3.10 with CUDA and FFmpeg, and install the dependencies for your model:
# Realtime-Venus-Omni
python -m pip install -r /path/to/Realtime-Venus-Omni/requirements.txt
# Realtime-Venus-Audio
python -m pip install -r requirements.txt
For the browser demo, run the installer on Linux. It creates an isolated Python 3.11/3.12 environment and installs the model, web service, and Harness dependencies:
bash install.sh
Then choose standalone inference or the online experience below.
2. Offline inference · Frontend Usage
Download the model checkpoint from the links above and replace the example paths with your local directories.
Realtime-Venus-Omni
export REALTIME_VENUS_MODEL_PATH=/path/to/Realtime-Venus-Omni
python frontend/Realtime-Venus-Omni/offline_chat.py
# With long-video Memory
python frontend/Realtime-Venus-Omni/offline_memory_chat.py
These examples read videos from the checkpoint's assets/ directory and output text and speech. To run full-duplex inference on recorded inputs instead:
python frontend/Realtime-Venus-Omni/duplex_chat.py
python frontend/Realtime-Venus-Omni/duplex_speech_in_chat.py
python frontend/Realtime-Venus-Omni/duplex_memory_chat.py
Duplex examples save subtitled videos. See the Omni guide for inputs and output paths.
Realtime-Venus-Audio
python frontend/Realtime-Venus-Audio/audio_offline_chat.py \
--model-path /path/to/Realtime-Venus-Audio \
--audio frontend/Realtime-Venus-Audio/case/case_offline.wav
Offline inference returns text. To run full-duplex inference on recorded audio and save a 24 kHz WAV:
python frontend/Realtime-Venus-Audio/audio_duplex_chat.py \
--model-path /path/to/Realtime-Venus-Audio \
--audio frontend/Realtime-Venus-Audio/case/case_duplex.wav \
--output output/duplex_response.wav
Both scripts accept --system-prompt and --prompt; see the Audio guide for decoding options.
3. Online Duplex experience
The online Duplex demo requires at least one NVIDIA A100 GPU. Place the complete Omni checkpoint in model_weight/, then start the demo:
bash start.sh
For an existing checkpoint elsewhere:
bash start.sh --model-path /path/to/Realtime-Venus-Omni
Complete Codex login if prompted. The launcher starts the model API on 8031 and the browser service on 8032.
If running on a remote server, keep this tunnel open on your local computer, replacing user@server with your SSH login:
ssh -N -L 8032:127.0.0.1:8032 user@server
Open http://localhost:8032, select voice, camera + microphone, or video upload, and start a conversation. Checkpoint layout, configuration, and service management are covered in the Demo guide.
Results
Figure 1. Video and audio understanding results from the paper.
Figure 2. Full-duplex interaction results from the paper.
Code map
Realtime-Venus/
├── harness/ # Context, routing, agents, work state, and delivery
│ ├── README.md
│ └── requirements.txt # Harness media dependencies
├── frontend/
│ ├── Realtime-Venus-Omni/ # Audiovisual inference examples
│ └── Realtime-Venus-Audio/ # Audio inference examples and input samples
├── demos/
│ ├── model/ # Checkpoint adapter and model HTTP API
│ ├── server/ # Web sessions, media, settings, and artifacts
│ ├── static/ # Browser UI, capture, playback, and logos
│ ├── launcher/ # Configuration and process supervision
│ ├── settings.py # Saved task and model preferences
│ ├── install.py # Isolated environment installation
│ └── requirements.txt # Model and application dependencies
├── assets/ # Paper figures, report, and Demo screenshot
├── pyproject.toml # Standalone Harness package
├── requirements.txt # Shared dependency entry for inference examples
├── config.example.json # Server configuration template
└── install.sh / start.sh # Install, start, inspect, and stop the Demo
Citation
@article{zhao2026realtime,
title={{Realtime-Venus}: A full-duplex interaction system with asynchronous delegation},
author={{Venus Team,Ant Group;Tsinghua University}},
journal={arXiv preprint arXiv:2609.13814},
year={2026}
}
License
The source code in this repository is licensed under the Apache License 2.0, except for components with separate license notices. Third-party fonts and paper figures retain their respective licenses; see the license files in demos/static/fonts/ and assets/README.md.