将 VLM 智能迁移至机器人控制:RoboDawn 框架

HuggingFace Daily Papers(社区热门论文)·2026-09-19 08:00·3天前
AI 导读

RoboDawn 通过离散平移、旋转和夹爪命令,让智能体式 VLM 以闭环方式控制机器人,并借助上下文学习用少量演示完成界面使用与任务策略落地。在 RoboTwin 2.0 C2R 上,成功率从零样本 53.2% 提升到单样本 73.6%,超过基线 π0.5 的 46.0%;在 RoboDojo 上从 35.67% 提升到 47.17%,且无需针对任务做机器人训练。

HuggingFace Daily Papers(社区热门论文)
47AI 编辑部评分,满分 100

将 VLM 智能迁移至机器人控制:RoboDawn 框架

2026-09-19 08:00· 3天前
AI 导读

RoboDawn 通过离散平移、旋转和夹爪命令,让智能体式 VLM 以闭环方式控制机器人,并借助上下文学习用少量演示完成界面使用与任务策略落地。在 RoboTwin 2.0 C2R 上,成功率从零样本 53.2% 提升到单样本 73.6%,超过基线 π0.5 的 46.0%;在 RoboDojo 上从 35.67% 提升到 47.17%,且无需针对任务做机器人训练。

Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands.

Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results.

On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline π0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org