ServerBench：让编码智能体自己设计 LLM 推理服务系统
ServerBench: How can your coding agents design specialized LLM serving systems?
这篇论文挺有意思：让 Claude Sonnet 5.5 这类编码智能体去自己写推理服务系统，成绩做到 vLLM 的 4.87 倍，还附带基准复现方法。
arXiv 论文提出 ServerBench，用于评测智能体设计完整、专用 LLM 推理服务系统的能力。优化目标为 dollars per dollar，即每美元 GPU 成本产出的合格 token 价值，并通过有状态回放保留会话依赖、独立验证检查提交的系统。在 11 个编码智能体模型和 3 种部署环境的测试中，Claude Sonnet 5.5 在 Gemma E4B 上取得 2.97 dollars per dollar，达到 vLLM 参考基线的 4.87 倍；33 次运行中有 22 次产出通过验证的系统，13 次超过最强基线。分析还给出三点发现：高价值设计需要协调容量提升与负载选择，智能体通过与工作负载相关的实验解读性能反馈，可靠交付需要对优化引入的执行路径做验证。
ServerBench: How can your coding agents design specialized LLM serving systems?
Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and operating decisions under session dependencies, latency constraints, and correctness requirements. We introduce ServerBench, a benchmark for agentic design of complete, specialized LLM serving systems. Agents experiment against a common objective, dollars per dollar: latency-qualified token value per dollar of GPU cost. Stateful replay preserves session dependencies, and independent verification checks the submitted system. Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, $4.87\times$ the vLLM reference; 22 of 33 runs yield verified systems, and 13 exceed the strongest baseline. Our analysis identifies three findings: high-value designs coordinate capacity improvements with load selection; agents interpret performance feedback through workload-relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimization. Overall, ServerBench supports systematic study of how agents translate service objectives and feedback into effective, verified serving systems for given task specifications.