Amazon 论文 FEB-30：Agent 记忆评测只看 QA 分会漏掉管线故障
Amazon 出了篇很实用的论文，做了个 FEB-30 基准，发现只看 QA 分数根本发现不了记忆存错了，建议每个 update 后直接检查存储的事实。
Amazon 发表论文，指出现有 Agent 记忆基准用问答分数评估，会把记忆、检索和答案生成交织在一起，无法定位故障环节。团队构建了 FEB-30 基准，包含 30 条脚本化事实历史、共 147 篇文档，每篇文档处理后的正确记忆状态都已知。在仿照 Mem0 和 Hindsight 搭建的记忆系统上，QA 分数与真实记忆状态的评估平均相差 16.8 分，21% 的历史在通过/失败判定上结论相反。系统选对记忆操作（如 update 或 reinforce）的比例只有 20.1%，这一问题 QA 分数完全无法暴露。
New Amazon paper shows that teams shipping agent memory should check what actually got stored, since QA scores alone did not reveal which pipeline stage was failing.
Agent memory benchmarks grade the agent's answers, which mixes memory with retrieval and answer writing. A strong LLM can hide a broken memory store, and a retrieval miss can make a correct one look bad.
They built FEB-30, 30 scripted fact histories turned into 147 documents, so the correct memory state is known after every document. On a memory system built like Mem0 and Hindsight, they compared the stored facts with that truth and with QA scores.
The 2 scores differed by 16.8 points on average, and pass/fail verdicts disagreed on 21% of fact histories. The system picked the right memory action, like update or reinforce, only 20.1% of the time, which QA never showed.
If you build agent memory, test the stored facts directly after each update, alongside end-to-end QA.
