自改进智能体很脆弱：任务顺序与规格模糊如何影响记忆效果
SF Research 把记忆型智能体的老方法重新跑了一遍，发现换个任务顺序成绩就崩，还给出能挽回 31% 损失的改进办法，做 Agent 的人都该看看。
SF Research 在 WebArena、VisualWebArena 和 SCUBA 三个基准上重新评估了两种已知的记忆型智能体方法，每种实验多次运行并打乱任务顺序。结果显示不同运行之间的成绩波动很大，单一运行无法反映真实水平。默认基准顺序实际是从易到难的课程式排列，而真实请求是随机到达的。在记忆构建中加入评分标准、环境反馈和更清晰的提示词，可以挽回打乱顺序下 31% 的性能损失。代码和数据已开源。
On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification: https://t.co/uSh4hMOp6G
Memory-based agents learn from a stream of tasks and save lessons for future work. We re-evaluated two established methods across WebArena, VisualWebArena, and SCUBA, running each experiment multiple times and shuffling task order to reflect how requests arrive in practice.
What we learned:
📊 Results can vary widely across runs, so reporting multiple runs gives a clearer picture of real progress.
🔀 Default benchmark orderings follow an easy-to-hard curriculum, while real-world requests arrive in any order.
🧭 Clearer specification leads to more useful memories. Adding rubrics, environment feedback, and clearer prompts to memory construction closed 31% of the performance drop under shuffled task order.
The findings point toward stronger evaluation practices, memory validation, and interfaces that support human oversight. Code and data are available as open science.
#FutureOfAI #EnterpriseAI
