研究显示本地部署栈会影响工具调用评测结果
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
这篇论文很实用：同一个模型在 Ollama 和 vLLM 上跑工具调用评测，结果可能差几十个百分点，做本地评测前建议先看它的检查清单。
arXiv 论文 2609.26693 研究本地推理服务栈如何影响代码 Agent 的工具调用评测。测试发现 Ollama 会按模型的静态模板标志拦截请求，Phi-3 和 Gemma-3 在推理前就被直接拒绝。被拒和重试耗尽在评测框架中没有被记录为结构化失败元数据，可能被误判为模型不调用工具，错误报告 0% 保真度。在 Ollama、llama.cpp、vLLM、SGLang 四个栈上，相同请求的处理方式各不相同，Llama-3.2 在统一文本协议下保真度反而下降，按轮次汇总与按实例统计的结果差距可达约 55 个百分点。论文给出把服务栈行为纳入评测协议的检查清单。
Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
A coding agent must emit a valid tool call--a parseable invocation of a tool in the provided schema--before the harness can execute its chosen action. We study how local serving stacks affect this protocol step and show that measured outcomes can depend on the serving layer rather than model behavior alone. In Ollama, the default tools= request is gated per model by a static template flag: some models are accepted and return calls as text, some return native tool_calls, while Phi-3 and Gemma-3 are rejected before inference. In our harness, rejection and retry exhaustion are not preserved as structured failure metadata, so downstream analysis can misclassify them as model non-calls and naively report 0% fidelity. Adding a text tool list while retaining the native channel recovers much of the measured fidelity for accepted models, whereas a uniform text protocol reduces fidelity for Llama-3.2, which has native tool-call support. Cross-stack probes on Ollama, llama.cpp, vLLM, and SGLang show different handling of the same request. Constrained decoding removes parse failures but can induce non-termination, and turn-pooled versus per-instance estimates differ by up to about 55 points. We conclude with a checklist for treating serving behavior as part of the evaluation protocol.
