论文

ROGUE-AI 复现 108 起 AI 智能体安全事件,验证事故根因

Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)

精选理由

这篇论文把新闻里报过的智能体翻车事故一个个离线复现出来了,108 次复现里 89% 能重现危害。有意思的是 62 起事故根本没有攻击者,模型踩的都是普通坑。做智能体部署的团队可以拿它当发布前的审计清单。

arXiv 论文提出 ROGUE-AI,一个多智能体审计框架,可离线重建媒体报道过的智能体事故(如 Replit 编程智能体删除生产数据库),并构造一个只缺少被归因原因的对照组来验证根因。在 108 次复现尝试中,框架成功重现了 89% 的报告危害。基于 GPT、Claude、Gemini 和 DeepSeek 五款当前模型的智能体在含被归因原因的场景中有 18% 到 33% 的运行犯下报告中的错误,对照组中最多仅 1%。

正文

技术细节上,72 起造成真实危害的事故中有 62 起不存在攻击者:智能体只是执行了任务指令,却错过了普通环境隐患,例如测试数据库与线上站点共用。这类无攻击者的复现造成的危害高于提示词注入复现,而现有防御主要针对注入指令,对这类事故不起作用。实验还发现,在每个工具输出后附加一条点名隐患及其后果的警告能降低全部五款模型的危害,同等长度但不点名隐患的提示则基本无效。

作者将 ROGUE-AI 定位为可复用的审计框架,厂商可以在每次新版本发布前重跑这些基于真实事故的复现场景。

原文 · arXiv cs.AI

Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:ROGUE-AI、GPT、Claude、Gemini,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Mengxiao Wang, Mahbub Alam, Kalyan Nakka, Nitesh Saxena),经 AITOP 多维评分管线结构化提炼。