Arena.ai 提出 Interaction-as-a-Judge：让计算机操作智能体上手应用来改进评估
Arena.ai 让 computer-use 智能体真的去玩应用再打分，比传统静态评测更贴近用户感受，还开源了两个轨迹数据集。做 agent 评测或代码生成研究的人可以看看这套思路。
Arena.ai 与卡内基梅隆大学博士生 Wayne Chi 合作研究 Interaction-as-a-Judge 评估范式：由 computer-use 智能体直接操作游戏和应用，产出交互轨迹，再交给 LLM judge 或训练好的 reward model 进行评分。研究发现这一方法在三个方向带来提升：与用户偏好的对齐度、模型检测游戏中错误的能力、以及下游代码生成质量。
机制上，偏好评估使用 Code Arena 的真实对战数据，正确性评估则从 Agent Arena 中真实用户投诉挖掘出的需求出发。团队还观察到 inference-time scaling 的改进，以及仅靠交互（interaction-only）进行强化学习的可行性；评估质量会随 computer-use 模型能力同步提升。
同时开源两个智能体轨迹数据集 ArenaWebdevPreferences 和 ArenaWebdevCorrectness，供后续研究使用。
Agents are demonstrating increasingly strong capabilities in recreating games and applications. One likely reason is their growing ability to directly interact with these applications. We studied Interaction-as-a-Judge, a growing evaluation paradigm where computer-use agents directly interact with games and applications to improve evaluation. Interaction-as-a-Judge improves: - alignment with user preferences. - a model’s ability to detect errors in games. - downstream generation. We see improvements in inference-time scaling and the ability for interaction-only reinforcement learning. It also scales with computer-use capabilities; the better the computer-use model and method, the better the judgments. We’re also open-sourcing two agent trace datasets, ArenaWebdevPreferences and ArenaWebdevCorrectness, to help support future research. Dive into the thread to learn more. Arena.ai @arena Arena research intern and PhD candidate at Carnegie Mellon University, Wayne Chi, explains Interaction-as-a-Judge: letting computer-use agents play games and use apps to better improve evaluation and downstream generation. A computer-use agent navigates an application, producing interaction traces for evaluation by an LLM judge or a trained reward model. The talk covers preference evaluation using real Code Arena battles and correctness evaluation using requirements mined from real user complaints in Agent Arena. Watch the full talk: youtube.com/live/lu--lrbzn… 🔗 View Quoted Tweet📊 4 ⚡
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：computer-use、Arena.ai、LLM judge、智能体，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「lmarena.ai」（@lmarena_ai），经 AITOP 多维评分管线结构化提炼。
