HEAMS 框架发布:从安全、隐私、用户主权等维度全面评估智能体记忆系统
HEAMS: Holistic Evaluation of Agent Memory Systems
做智能体记忆的团队该看看这个。MilaMem 这套开源基准把隐私、投毒、用户删除权都纳入评测,光跑性能分数会漏掉大问题,选型前建议先跑一遍。
一篇新论文提出 HEAMS(Holistic Evaluation of Agent Memory Systems),一个开源框架,主张把智能体记忆系统视为用户持久信息的保管者,而非只看任务表现。该框架将记忆系统的职责拆成五大类行为准则:性能、效率、安全(security)、隐私和用户主权(如可更正、可删除存储信息)。
作者团队配套构建了覆盖这些维度的基准套件,开发了原型记忆系统 MilaMem,以及一个可包裹任意记忆系统的治理层(governance layer),用于控制变量地研究设计选择对行为的影响。实验对比多个记忆系统后发现:没有任何系统在所有性能基准上领先,性能得分高的系统可能在隐私、用户主权和安全维度表现很差。
治理策略的影响显著:召回策略能降低不当披露,但会牺牲帮助性;召回和保留策略都能提升抗记忆投毒能力。论文结论是仅靠性能与效率基准无法判断记忆系统是否尽到职责,代码已在 GitHub 开源。
HEAMS: Holistic Evaluation of Agent Memory Systems
Memory systems allow AI agents to persist information across interactions, making them more personalized and more capable on long-horizon tasks. This persistence introduces new risks: stored information can be disclosed in inappropriate contexts, resist correction or deletion, be retained beyond what users desire, or be poisoned to influence later agent behaviour. Yet memory systems are typically evaluated on task performance and efficiency alone. We argue that a memory system should instead be understood as the custodian of its user's persistent information, with responsibilities that extend beyond helping an agent complete its tasks, and that it should be evaluated holistically against all such responsibilities. We present Holistic Evaluation of Agent Memory Systems (HEAMS), an open-source framework for evaluating agent memory systems in this way. First, we state the responsibilities of agent memory systems as explicit behavioural desiderata spanning performance, efficiency, security, privacy, and user sovereignty. Second, we assemble a benchmark suite that covers these desiderata and acknowledge the gaps that remain. Third, we develop MilaMem, a prototype memory system, and a governance layer that can wrap any memory system, enabling controlled study of memory system design choices. Fourth, we compare memory systems across the full HEAMS benchmark suite. We find that no memory system leads every performance benchmark, and strong performance scores may coexist with poor privacy, user-sovereignty, and security scores. Governance strongly influences results: recall policies can reduce inappropriate disclosure at a cost in helpfulness, while both recall and retention policies improve poisoning resistance. Performance and efficiency benchmarks alone therefore cannot reveal how well a memory system meets its responsibilities. Code: https://github.com/milatechtransfer/heams
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:智能体、记忆系统、AI安全、MilaMem,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Chirag Goel, Jacob Lavoie, Amirhossein Naghshzan, Jordan Bannister, Marzia Nouri, Mehar Bhatia, Matt Harris, Siva Reddy, Arsène Fansi Tchango, Gaétan Marceau Caron),经 AITOP 多维评分管线结构化提炼。