SIT-Eval：基于情境的逐轮对话智能体评测框架
SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents
德国团队做了个对话评测新框架，8 个模型跑 1292 轮情境对话，总分看不出的问题逐轮都暴露出来了，做 Agent 评测的可以看看。
arXiv 论文提出 SIT-Eval，用"如果情境是 P 就遵守规则 Q"的形式化方法评估对话智能体。每轮由 LLM-interlocutor 扮演随机采样的情境，再用上下文感知的 LLM-judge 依据适用规则逐轮打分，判官经人工校验的合成黄金对话验证。对 8 个 LLM 在 1292 轮对话上评测发现，总分相近的模型在单条规则遵守度上差异很大。所有模型在歧义澄清和苏格拉底式提问上表现最差。
SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents
The rapid proliferation of conversational AI agents necessitates evaluation methods that assess performance across diverse dialogue scenarios, utilizing multiple situational criteria, as appropriate behavior depends on context and encompasses various aspects. Existing dialogue benchmarks typically use static natural conversations, which underrepresent rare situations---a critical safety gap, as low-probability events become frequent failures at production scale. However, limited sample sizes force a choice: either maintain natural dialogues for realism or turn to adaptive situation sampling for comprehensive yet statistically reliable coverage. We introduce SIT-Eval, an adjustable framework for evaluating situation-based conversational agents. We formalize an agent's desired behavior as situational rules: if situation is P, follow rules Q. At each conversation turn, an LLM-interlocutor enacts a uniformly sampled situation. A context-aware LLM-judge leverages all conditionally applicable rules as criteria to evaluate the agent's reply for that situation. The judge is validated using human-verified synthetic golden dialogues. Benchmarking 8 LLMs on 1292 conversation turns reveals failure modes hidden by aggregate metrics: models with similar overall scores vary widely in their adherence to individual rules. All models struggle most with ambiguity clarification and Socratic questioning.
