PrecogUI:预认知模拟与经验检索实现主动GUI智能体
PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval
MIT团队推出PrecogUI,能预测UI布局避免错误,在长时程任务中表现优于现有方法
PrecogUI通过预认知架构解决传统GUI智能体在长时程动态场景中的注意力分散和级联失败问题。该系统包含主动经验池(PEP)存储异常和成功模式,主动模拟执行器(PSE)预测下一个符号UI布局,以及预认知执行控制器(PEC)融合预测确保执行鲁棒性。研究团队开发了AutoTraj数据生成引擎构建InterfereBench基准测试,实验显示PrecogUI在该基准上超越现有方法,同时在公共基准上保持竞争力。
PrecogUI: Proactive GUI Agents via Pre-cognitive Simulation and Experience Retrieval
Existing reactive Graphical User Interface (GUI) agents often fail in long-horizon, dynamic scenarios, where unexpected disturbances trigger attention-diverting and cascading failures. To address this, we propose PrecogUI, a pre-cognitive architecture that shifts the paradigm from reactive execution to proactive decision-making. Specifically, we design a Proactive Experience Pool (PEP), which caches recurring anomaly and success patterns as "state-action-result" tuples in a dual-memory repository. Furthermore, we introduce a Proactive Simulation Executor (PSE) that learns to forecast the next symbolic UI layout given a candidate action, enabling early anomaly avoidance and ranking candidate actions by predicted reliability. Finally, a Pre-cognitive Execution Controller (PEC) fuses these priors and predictions, prioritizes handling of foreseen anomalies, and ensures execution robustness through a closed-loop error correction mechanism. For robust evaluation, we develop AutoTraj, an automatic data-generation engine, to construct InterfereBench, a benchmark for long-horizon tasks with strong disturbances. Experiments demonstrate that PrecogUI surpasses state-of-the-art methods on InterfereBench while maintaining competitive performance on public benchmarks. The code will be publicly available.