SWE-Serve：评测智能体生产级推理工程能力的新基准
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
做了个基准 SWE-Serve，用 SGLang 的 53 个真实任务测智能体写推理工程代码，发现三分之一的补丁看着过了测试、上线就翻车。
SWE-Serve 是一个针对生产推理工程任务的智能体基准，包含源自 SGLang 生产变更的 53 个仓库级任务，覆盖六个推理工程方向。任务在 CPU 或单张 H100 上执行，用隐藏功能测试、回归测试、端到端服务测试和性能门槛评分。评测覆盖 11 个模型、31 种模型-推理力度配置，最佳配置平均 pass@1 达 75%。在 19 个有端到端覆盖的任务上，约三分之一通过了其他所有测试的补丁被模型服务 E2E 测试拒绝（45.9% 对 69.4%），说明本地完成任务与生产正确性之间存在明显差距。
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs. Existing benchmarks provide limited coverage of production inference engineering: repository-level software engineering benchmarks do not target inference, while general terminal-agent benchmarks include only a few inference tasks. Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation. SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families. Each task executes on either CPU or a single GPU (H100) and is evaluated with hidden functional and regression tests, including, where applicable, end-to-end (E2E) serving tests and calibrated performance gates. Executable no-op and oracle controls, adversarial verifier review, and closed-book execution support task validity and evaluation integrity. Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1. SWE-Serve exposes a substantial gap between completing tasks locally and achieving production correctness. On 19 tasks with end-to-end coverage, model-serving E2E tests reject roughly one-third of patches that pass every other test (45.9% under the verifier versus 69.4% with E2E tests excluded from scoring), with pass rate increasing for each model's best-performing configuration. By making the production correctness gap directly measurable, SWE-Serve enables the field to track whether future agents move beyond completing tasks locally to achieving production correctness.
