Arena 研究了 Interaction-as-a-Judge 评估范式，让 computer-use 智能体直接与游戏和应用交互来改进评估。该范式提升了与用户偏好的一致性、模型检测游戏错误的能力以及下游生成效果，并在推理时扩展和纯交互强化学习上均有改善，且效果随 computer-use 能力增强而提升。
Agents are demonstrating increasingly strong capabilities in recreating games and applications. One likely reason is their growing ability to directly interact with these applications.
We studied Interaction-as-a-Judge, a growing evaluation paradigm where computer-use agents directly interact with games and applications to improve evaluation.
Interaction-as-a-Judge improves:
- alignment with user preferences.
- a model’s ability to detect errors in games.
- downstream generation. We see improvements in inference-time scaling and the ability for interaction-only reinforcement learning.
It also scales with computer-use capabilities; the better the computer-use model and method, the better the judgments.
We’re also open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to help support future research. Dive into the thread to learn more.
在 X 查看被引用的帖子Arena research intern and PhD candidate at Carnegie Mellon University, Wayne Chi, explains Interaction-as-a-Judge: letting computer-use agents play games and use apps to better improve evaluation and downstream generation.
A computer-use agent navigates an application, producing interaction traces for evaluation by an LLM judge or a trained reward model. The talk covers preference evaluation using real Code Arena battles and correctness evaluation using requirements mined from real user complaints in Agent Arena.
Watch the full talk:
youtube.com/live/lu--lrbznOk
来源：Arena.ai · x.com