PASTABench 基准测试：智能体主动安全干预仍未解决
PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
这个叫 PASTABench 的新基准测了 16 个大模型给智能体踩刹车的时机，最好的也只拿到 40.74%，还拆穿了不少小模型靠关键词得分。
PASTABench 包含 1,139 条多轮智能体轨迹，覆盖 5 类风险和 13 个子类，用于评估解耦式主动安全监控。基准提出最优干预窗口（OIW），通过标注 Earliest-Signal 和 Trigger 回合来量化干预时机。对 16 个 LLM 的测试显示，表现最好的模型最优时机干预率也只有 40.74%。细粒度诊断还发现，一些小模型的安全分数来自对风险关键词的过度敏感，一旦中和危险词汇，主动能力大幅下降。
PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
As Large Language Models (LLMs) evolve into autonomous agents that alter real-world states, ensuring operational safety across multi-step workflows has become a critical challenge. While recent work has moved beyond single-turn evaluation toward multi-turn paradigms, key limitations persist: step-level methods treat actions in isolation, missing how risks accumulate, while trajectory-level evaluations operate post-hoc, offering no opportunity for timely intervention. To address these limitations, we formalize Decoupled Proactive Safety Monitoring along three dimensions: whether to intervene, when to intervene, and what the risk is. We introduce PASTABench, a benchmark of 1,139 multi-turn trajectories spanning 5 risk categories and 13 subcategories. We further propose the Optimal Intervention Window (OIW), anchored by annotated Earliest-Signal and Trigger turns, to quantify intervention timeliness. Evaluation of 16 LLMs reveals that proactive intervention remains largely unsolved, with the best model achieving only 40.74% optimal-timing interventions. Fine-grained diagnosis further uncovers pervasive lexical overfitting: competitive safety scores of smaller models mask keyword hypersensitivity rather than genuine risk comprehension, as their proactive capability largely collapses once hazard vocabulary is neutralized.
