VelServe 用尾延迟校准调度解决 LLM 解码抖动问题
VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling
做推理服务部署的工程师应该看看这篇。它不换模型,只改调度:按实测尾延迟给每个解码步留余量,Qwen3-8B 上 SLO goodput 直接翻到基线的 2.4–2.9 倍,8 卡分离部署下达标请求数翻倍。比 vLLM 这类常规 SLO 调度器稳,适合流式和实时场景。
arXiv 论文提出 VelServe,一个针对交互式 LLM 服务 SLO(TTFT 和 TBT)的调度器,解决解码阶段的抖动问题。
核心机制是根据每次迭代延迟的实测尾部分布,校准每个解码步可承载的 prefill 工作量,按 SLO 要求的迟到概率预留余量,防止一步超时拖累整个 batch。由于预留余量会拖慢 prefill,VelServe 优先服务等待时间最短的 prompt;
在 prefill 与解码分离部署下,它通过控制请求准入间隔,保证没有请求等待超过其 P99 目标允许的慢准入步数。实验覆盖三种注意力架构和多种负载:在 Qwen3-8B 和 DeepSeek-V4-Flash 上,VelServe 达到最强基线 2.
4–2.9 倍的 SLO goodput;
在 DeepSeek-V2-Lite 上至少 4.4 倍。
在 8 张 GPU 上做 prefill/decode 分离时,其满足 SLO 的请求数超过现有最先进基线的 2 倍。
VelServe: Taming Decode Jitters in LLM Inference with Tail-Calibrated Scheduling
Interactive LLM services must meet latency service-level objectives (SLOs) for the time to first token (TTFT) and the time between tokens (TBT), as streaming and soft real-time applications require predictable token delivery. We identify decode jitters in both aggregated and disaggregated serving. A jitter delays every request in the batch, and long responses span hundreds of steps, so even rare overruns cause many SLO violations. In aggregated serving, SLO-aware schedulers set each step's prefill budget until its predicted latency just meets the TBT target, leaving no room for mispredictions or host stalls. Disaggregation meets average TBT targets at a cost in capacity, yet the steps that admit new requests still run late. We present \sysname, a scheduler that calibrates how much prefill work each decode step carries using the measured tail of its own iteration latency. The SLO determines how rare a late step must be, and \sysname leaves enough headroom in each iteration to prevent overruns and meet the target decode latency. This headroom slows prefill, so \sysname serves the shortest waiting prompt first. When TTFT deadlines grow with prompt length, the long prompts it delays have the most time to spare. Under disaggregation, it spaces admissions so that no request waits on more slow admission steps than its P99 target allows. On three attention families and varying workloads, \sysname sustains 2.4--2.9$\times$ the SLO goodput of the strongest baseline on Qwen3-8B and DeepSeek-V4-Flash, and at least 4.4$\times$ on DeepSeek-V2-Lite. With prefill and decode disaggregated on eight GPUs, \sysname meets the SLO for more than twice as many requests as state-of-the-art baselines.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:VelServe、Qwen3-8B、DeepSeek、LLM推理,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。截至目前,全网已有 2 家独立信源对该事件进行交叉印证与跟进报道(含 DAIR.AI 等),属于近期高关注度行业信号。
- DAIR.AI11-12 23:50原文