模型精选73°

CUA-Universe:混合GUI+CLI智能体环境

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

精选理由

CUA-Universe让AI智能体能同时使用图形界面和命令行,比纯GUI或纯CLI更高效实用。

CUA-Universe是一个可扩展的环境到数据管道,将真实桌面软件转化为混合GUI+CLI环境。该系统包含App-Forge、Task-Weave和Path-Steer三个组件,支持16个应用程序。训练后的9B模型在CUA-Verse上得分提升39.3点,步骤减少37%;在OSWorld成功率提升16.8点,步骤减少57%;在OSWorld-MCP得分提升7.84点,步骤减少27%。

原文 · arXiv cs.AI

CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents

Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld, but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. We introduce CUA-Universe, a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers rollouts along efficient hybrid paths and harvests verified trajectories for post-training. Training on this data shifts behavior from inefficient GUI interaction and brittle CLI scripting toward effective GUI+CLI orchestration. Our 9B model improves both success and efficiency on CUA-Verse (Score +39.3 pts; -37% steps, -60% tokens), OSWorld (SR +16.8 pts; -57% steps, -44% tokens), and OSWorld-MCP (Score +7.84 pts; -27% steps, -30% tokens). CUA-Universe provides a scalable path toward more capable and efficient computer-use agents.