EvoSkill-GUI:面向 GUI 智能体的免训练技能进化框架

HuggingFace Daily Papers(社区热门论文)·2026-09-15 08:00·3天前
AI 导读

EvoSkill-GUI 是一个免训练框架,通过“反思-修订-复用”循环让 GUI 智能体在部署时根据执行反馈修订技能,而非依赖部署前生成的静态技能。技能以结构化多文件包形式组织,包含检索元数据、可执行计划、备份定位、失败恢复规则、无障碍工具和失败案例。

HuggingFace Daily Papers(社区热门论文)
45AI 编辑部评分,满分 100

EvoSkill-GUI:面向 GUI 智能体的免训练技能进化框架

2026-09-15 08:00· 3天前
AI 导读

EvoSkill-GUI 是一个免训练框架,通过“反思-修订-复用”循环让 GUI 智能体在部署时根据执行反馈修订技能,而非依赖部署前生成的静态技能。技能以结构化多文件包形式组织,包含检索元数据、可执行计划、备份定位、失败恢复规则、无障碍工具和失败案例。

GUI agents execute long-horizon tasks on dynamic graphical user interfaces, where pop-ups, delayed loads, and relocated widgets routinely invalidate plans fixed before execution. Recent agent-skill frameworks encapsulate reusable procedural knowledge to mitigate this, yet existing skill designs are largely developed without targeting GUI execution dynamics and treat skills as static artifacts produced before deployment rather than living procedural knowledge that improves through it. We argue that what GUI agents need is not better static skills, but skills that can be revised from execution feedback at deployment time, without additional training. We propose EvoSkill-GUI, a training-free framework in which each skill is a structured multi-file package containing retrieval metadata, executable plans, backup localization, failure-recovery rules, accessibility utilities, and failure cases. EvoSkill-GUI operates through a \emph{reflect-revise-reuse} loop: the executor performs instant in-rollout revisions, an isolated critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files through a restricted tool interface. Across MobileWorld, AndroidWorld, and OSWorld, three mainstream GUI benchmarks spanning mobile and desktop platforms, EvoSkill-GUI consistently improves multiple base models without any training, with maximum gains of +16.2%, +6.0%, and +10.5% respectively, and evolved skill libraries continue to benefit related tasks rather than being rebuilt from scratch. Our code is available at https://github.com/ZJU-REAL/EvoSkill-GUI.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org