VAST:把漏洞场景编译成可验证多步环境,评测 LLM 红队智能体
VAST: Compiling Grounded Vulnerability Scenarios into Verifiable, Multi-Step Environments for Evaluating LLM Red-Team Agents
做安全智能体评测的应该看看这篇:VAST 把 CVE 场景编译成可调难度的多步靶场,glm-5.3、grok-4.7、deepseek-v4-pro 在没拿到工作流协议时 90 道题只解出 12-14 道,给全协议就到 76。它还用 Qwen2.5-Coder-7B 微调对比证明头部信息恢复训练有效。
论文提出 VAST(Verifiable Adversarial Scenario Training & Evaluation),一个声明式靶场编译器,把漏洞场景与编译器引入的工作流加固层分离,用于可复现地评测 LLM 安全智能体。主基准经代码与 AST 审计后保留 109 条种子边(其中 45 条带严格上游 CVE 标注)、72 个 family 标识符、81 个 topic 配置和 1387 个实例,按 42:39 划分。
实验层面有三组结果。TopicServer 层上,对 glm-5.3、grok-4.7、deepseek-v4-pro 做 270 集、10 个机制 family 聚类、temperature 0.0 的预算匹配对比:只给合同(contract_only)与泛化提示均只解决 12/90 与 14/90,而完整提供工作流协议(full_guide)提升到 76/90(聚类符号检验 p<=0.00391)。独立 ranges_v2 账本上,6 个非 TopicServer 工作流、多容器 Docker/Linux 栈的 126 集评测解决 88/126(69.8%);对异步 Redis worker 链(v2_02)的 18 集干预显示,只要在 GET /sched/directory 中披露已安装的 worker 模块,解决率即从 0/18 升至 18/18。微调方面,在 Qwen2.5-Coder-7B-Instruct 上跨 37 个 lineage 不相交 family(231 集、33 个保留实例)的三种子 LoRA 对比:Base 为 0/33,Golden SFT 为 1.67±2.89,Header-Recovery SFT 为 4.67±2.31,全部解决集中在 v2_03。
这项工作的核心结论是:智能体失败多源于工作流信息缺失而非模型能力不足,full_guide 与目录披露干预的巨大差距说明评测环境需要精细控制难度变量。
VAST: Compiling Grounded Vulnerability Scenarios into Verifiable, Multi-Step Environments for Evaluating LLM Red-Team Agents
Evaluating Large Language Model (LLM) security agents requires reproducible environments whose workflow difficulty can be varied and whose failures can be diagnosed. We present VAST (Verifiable Adversarial Scenario Training & Evaluation), a declarative range compiler separating vulnerability scenarios from a compiler-introduced workflow-hardening layer. Following a post-hoc code and AST audit across all topics, the main benchmark retains 109 seed edges (45 with strict upstream CVE annotations), 72 family identifiers (62 under shared-HMAC relay grouping), 81 topic configurations (76 core plus 5 bounded execution models), and 1,387 instances (359 canonical four-gate-revalidated specs plus 1,028 seed variants) in a 42:39 partition. On the TopicServer compiler stratum, a budget-matched 270-episode prompt contrast across 10 compiled mechanism-family clusters (30 P2/P3/P4 instances, 40 turns, temperature 0.0) shows that glm-5.3, grok-4.7, and deepseek-v4-pro solve 12/90 instances under contract_only and 14/90 under generic_hint (cluster sign p=1.00), whereas supplying the shared workflow protocol in full_guide lifts completion to 76/90 (28/30, 23/30, 25/30; cluster sign p<=0.00391). On the separate ranges_v2 ledger, a 126-episode evaluation across 6 non-TopicServer workflows and multi-container Docker/Linux stacks solves 88/126 (69.8%) (28/42, 31/42, 29/42), while an 18-episode intervention on asynchronous Redis worker chaining (v2_02) lifts solves from 0/18 to 18/18 (100%) in 4-6 turns when installed worker modules are disclosed in GET /sched/directory. Finally, a three-seed (s in {42, 1024, 2025}) LoRA recipe comparison on Qwen2.5-Coder-7B-Instruct across 37 lineage-disjoint families (231 episodes on 33 held-out instances) yields 0/33 for Base, 1.67 +/- 2.89 / 33 for Golden SFT, and 4.67 +/- 2.31 / 33 for Header-Recovery SFT, with all solves on v2_03.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:VAST、LLM红队、安全评测、智能体,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。