arXiv 论文:对话历史会让 LLM 推理准确率大幅下降
The Silent Cost of Conversation History in LLM Reasoning
做多轮 Agent 或对话系统的人该看看这篇。它用 48 个数学任务和 CDR-Stream 基准证明对话历史会把准确率拖垮,摘要替代能高出 0.5 以上,而且常用监控指标根本测不出问题。工程上直接可用的结论:状态放 transcript 外面。
arXiv 论文 2611.20071 研究了多轮对话中累积的历史记录对 LLM 推理准确率的影响。Study 1 在 48 个分片数学任务上发现,保留模型自己的历史回复使 GPT-4o-mini 最终成功率下降 0.21,Mistral-7B 下降 0.29(Holm 校正后 p 值分别为 0.026 和 0.004)。
Study 2 提出 CDR-Stream 基准:100 轮状态追踪对话,每轮有精确答案并做按模型难度校准。在完整历史下,同等难度时准确率从约 0.6 在二十轮内跌到 0.1;而用求解器验证过的状态摘要替代历史,可把准确率保持在高出 0.52-0.59 的水平(p<0.001)。一致性、逻辑连贯性等过程指标的下降远小于准确率,且稳定性反而上升,因此这些指标无法提前预警。
作者测试的一致性指标、配对就绪度测量和校准后的单流漂移检测器都没有给出可靠的跨模型早期预警;单条 100 轮流式数据只能检测到极大的降幅。论文建议系统设计者把任务状态存放在对话记录之外,并跨会话进行监控。
The Silent Cost of Conversation History in LLM Reasoning
Large language models (LLMs) in multi-turn information systems accumulate their own earlier replies and a growing task state. We asked whether this conversation history lowers accuracy, and whether process-level quality metrics warn before it does. Each analysis was fixed before its outcomes were computed. In Study 1, on 48 sharded math tasks, retaining its replies lowered final success by 0.21 for GPT-4o-mini and 0.29 for Mistral-7B (Holm-adjusted p = 0.026 and 0.004); Llama-4 Scout showed the same direction, not significant. Study 2 introduces CDR-Stream, 100-turn state-tracking dialogues with an exact answer per turn and per-model difficulty calibration. With the full history, accuracy fell from about 0.6 to 0.1 within twenty turns at matched difficulty, whereas a solver-verified state summary kept it 0.52-0.59 higher (Holm-adjusted p less than 0.001). Consistency and logical coherence fell far less than accuracy, and stability rose. None of these metrics, a paired readiness measure or calibrated single-stream drift detectors gave robust, model-general early warning, although in Study 2 the repeat-based metrics were complete only for GPT-4o-mini. In the monitors we tested, a single 100-turn stream had power only for very large drops. Systems built on LLMs should keep task state outside the transcript and monitor across sessions.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:GPT-4o-mini、Mistral-7B、CDR-Stream、智能体,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Ali Şenol),经 AITOP 多维评分管线结构化提炼。