研究表明工具调用会放大模型基准对比对措辞的敏感度
Prompt Sensitivity in Tool-Augmented Model Comparisons
用 6 个开源模型实测发现加工具后改个问法就能改变排名，做基准评测的人应该看看这套拆步骤的协议。
arXiv 论文提出一套匹配协议，把问答过程拆分为答案读出、推理、调用决策、工具返回和无调用报告五个步骤，用于检验工具是否让模型对比更稳定。实验覆盖 6 个开源模型、5 个领域、5 种问题措辞，发现工具在程序可回答的问题上提升准确率，却让许多两两对比的措辞敏感度变宽。敏感度加宽与调用频率相关，部分模型的调用频率随措辞变化。使用原生工具调用接口并追加第二份工具描述时，分子领域未检测到加宽，但工具返回中间结果时敏感度仍然存在。论文建议在准确率之外同时报告调用频率和多种措辞下的对比稳定性。
Prompt Sensitivity in Tool-Augmented Model Comparisons
Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.