TIPER：用任务完整性约束的人设演化挖掘 LLM 智能体鲁棒性弱点
Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints
UCSB 开源的 TIPER 自动演化刁钻用户测智能体，成功率最多掉 33.7 点，做 agent 的可以拿来跑一遍。
论文提出 TIPER，一种保持任务完整性的人设演化方法：在保证任务仍然有效可解的前提下，通过反思引导的行为变异和任务级 Pareto 选择演化用户人设，自动找出针对特定智能体的刁钻用户策略，弥补标准 LLM 用户模拟器行为过于合作的问题。在 4 个交互智能体基准上，TIPER 使全部 12 个 GPT-5.4 设置的 held-out avg@3 分数下降，降幅为 5.8 到 33.7 个百分点。在覆盖 7 个智能体的 28 个智能体-域组合中同样全部出现下降，中位数降幅 22.9 个百分点。论文还构建了行为到失败的分类法，把发现的用户行为对应到智能体的常见失败模式，代码开源于 GitHub（UCSB-NLP-Chang/TIPER）。
Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints
Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlooking failures that arise under diverse but realistic user interactions. This can lead to overly optimistic estimates of agent robustness and provide limited failure signals for training agents that generalize across users. Existing approaches largely rely on predefined user behaviors, which may miss agent- or domain-specific weaknesses, while failure-oriented search can reduce task success by invalidating the underlying task rather than exposing genuine robustness failures. To fill this gap, we introduce TIPER, a task-integrity-preserving persona evolution method that automatically discovers agent-specific user policies exposing robustness weaknesses while keeping the underlying task valid and solvable. TIPER evolves persistent user personas through reflection-guided behavioral mutations and task-wise Pareto selection under a task-integrity constraint. Across four interactive-agent benchmarks, TIPER yields lower held-out avg@3 scores in all 12 GPT-5.4 settings, with observed reductions of 5.8 to 33.7 percentage points, and in all 28 agent-domain pairs spanning seven agents, with a median observed reduction of 22.9 points. Further analysis yields a behavior-to-failure taxonomy that connects discovered user behaviors to recurring agent failure modes. Together, these results show that TIPER enables more diagnostic robustness evaluation of interactive agents and provides informative failure analysis for future agent robustness improvement. Code repository: https://github.com/UCSB-NLP-Chang/TIPER.
