论文精选

Penn State 论文揭示 BadEvolve 攻击可让自进化智能体自种后门

精选理由

做智能体的一定要看这篇:只要往学习集塞 20 个投毒任务,你的 agent 就会把后门写进自己的技能库,Kimi-K2.5 上成功率 79.6%,性能掉不到 1 个点,基本察觉不到。比以前的直接改技能文件的攻击假设现实得多,用外部任务训练 agent 的团队都得防着点。

Penn State 发布论文《When Experience Becomes a Backdoor》,提出 BadEvolve 攻击:只需向自进化智能体的学习集注入 20 个投毒任务,就能让智能体在自己的技能库里写入后门规则。区别于以往假设攻击者直接改写技能文件的方式,BadEvolve 靠 LLM 优化器迭代调整任务措辞,并逐项检查规则是否被写进技能、被加载、被执行并达成攻击者目标。

正文

在 Kimi-K2.5 上、跨 4 个框架和 3 个基准的测试中,BadEvolve 注入攻击者指定载荷的成功率达 79.6%,最优基线为 37.6%;同时干净任务准确率仅下降 1.1 个百分点,后门很难从性能指标上察觉。由于自进化智能体会把历史运行沉淀为可复用技能,一条被植入的恶意规则会在未来任务中持续触发。

论文给出的防御建议是:凡是智能体从用户提交或外部任务中学习技能的场景,都应把这些任务当作不可信输入,并在技能被复用前做人工审查。

原文 · rohanpaul_ai

New Penn State paper shows that attackers can make self-evolving AI agents plant backdoors in their own skills just by slipping in poisoned training tasks.

Self-evolving agents turn past runs into reusable skills, so a bad rule in a skill keeps firing on future tasks. Earlier attacks assumed direct edits to skills, which rarely holds when the agent writes them itself.

BadEvolve only injects 20 tasks into the agent's learning set, then tunes their wording with an LLM optimizer. It checks where each attempt breaks: whether the rule got written into a skill, loaded, followed, and produced the attacker's goal.

It injected an attacker-chosen payload 79.6% of the time on Kimi-K2.5 across 4 frameworks and 3 benchmarks, versus 37.6% for the best baseline. Clean-task accuracy fell only 1.1 points, so the backdoor is hard to notice.

If your agent learns skills from user-submitted or external tasks, treat those tasks as untrusted and review learned skills before reuse.

– arxiv. org/abs/2611.10542

Title: "When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization"

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:BadEvolve、Kimi-K2.5、智能体、AI安全,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「rohanpaul_ai」(@rohanpaul_ai),经 AITOP 多维评分管线结构化提炼。