SGLang 新增 /v1/score 接口与多候选项评分，优化决策模型推理
SGLang 加了个评分接口，做分类、排序这类决策任务时延迟直接砍到三分之一，用 Qwen3 的可以试试。
SGLang 发布博客，介绍面向决策类模型的推理优化。新增 /v1/score 接口可直接返回 Yes/No 或 A/B/C 等指定标签的分数，避免 Generate + top-k logprobs 丢失标签的问题。多候选项评分（MIS）将共享上下文只计算一次，在 Qwen3-8B 上 16 个候选的 p95 延迟从 Generate 的 54.1 ms 降到 20.6 ms。Qwen3-0.6B 在负载上升时 MIS p95 保持约 100 ms 以内，Generate 和 SIS 则达到秒级。该功能由 LinkedIn 团队贡献。
🚀 New blog: Scaling JEV-like decision models with SGLang
Decision models need a score, not prose. Classification, ranking, and agent action selection all ask the same thing: which option wins?
Open-Jev, for example, scores each candidate separately with a Yes/No prompt. Serving this well raises two issues: Generate + top-k logprobs can drop the label you need, and the shared context can be recomputed for every candidate.
SGLang addresses both: - /v1/score returns scores for the exact labels you request (Yes/No, A/B/C) - Multi-item scoring (MIS) computes the shared context once and keeps each candidate isolated - MIS latency stays nearly flat from 2 to 16 candidates, with 16-candidate p95 on Qwen3-8B dropping from 54.1 ms (Generate) to 20.6 ms (MIS) - MIS p95 stays under ~100 ms as load rises on Qwen3-0.6B, vs. seconds for Generate and SIS
Huge thanks to the @LinkedIn team for contributing! Benchmarks and launch commands in the blog 👇
