双智能体 LLM 流水线实现可扩展的对话式考试系统
Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline
口试能逼学生讲清推理过程，但一个老师考不了全班。这篇论文用两个分工的智能体做对话式考试，一个提问一个暗中按 rubric 打分，还专门防提示注入骗答案。做教育产品或评估工具的人可以看看架构设计。
arXiv 论文 2611.17220 提出 scalable conversational assessment，用 LLM 模拟口试，对学生进行基于文本的交互式论证能力评估。系统采用双智能体架构：interviewer 直接与学生对话但只知道下一个问题，hidden evaluator 掌握整体评估目标、按 rubric 追踪进度并指示何时追问或进入下一题，以此降低答案泄露风险。
评估先用不同能力等级和对抗策略的 LLM 人格学生模拟测试，包括答案提取者、提示注入者、自信编造者，并用虚构的考核主题防止预训练知识泄漏。系统随后在一门小型 AI literacy 选修课中用于形成性和终结性评估。
通过人工与事后 AI 审查转录文本，团队发现追问生成和“只给学生本人贡献的观点计分”仍需改进。论文解决的是口试难以规模化的问题：既保留追问和论证检验，又无需人工考官一对一参与。
Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline
Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral exams scale poorly. In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：LLM、双智能体、教育评估、AI literacy，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Geoffrey Challen, Craig Zilles），经 AITOP 多维评分管线结构化提炼。