论文

Stint:让长输出 LLM 服务中的 KV 缓存抢占不再浪费算力

Making Preemption Productive in Long-Output LLM Serving

精选理由

跑推理服务的人都知道内存被打满后请求互相挤兑有多难受。Stint 把被抢占请求的 KV 快照挪到 CPU 再搬回来,卡一分钟以上的请求直接消失,p99 延迟砍半。自己搭 vLLM/SGLang 之类服务的人值得看看思路。

Stint 是一个针对推理型 LLM 长输出场景的服务系统,解决 KV 缓存占满 GPU 内存后请求被反复抢占、状态丢失又恢复的 KV thrashing 问题。论文指出,现有系统在某些情况下会让超过五分之一的请求卡住超过一分钟。

正文

技术机制上,Stint 将被抢占请求的 KV 缓存以定长快照形式存入 contiguous-first 的 CPU 内存池,通过 exact-extent 分配器管理,配合融合的 K/V gather/scatter 内核与双缓冲搬运数据;读回时按年龄排序,并给恢复的请求一轮保护性解码,避免立刻再次被抢占,同时用解码隐藏状态估计剩余长度来选择牺牲请求。

实测中,在 H200 GPU 上用 Qwen3-32B 回放奥数数学题场景,Stint 下没有请求停顿超过 50 秒,p99 每输出 token 延迟相对同引擎的重计算方案下降 50.7%,整体回放提前 10.7% 完成,吞吐提升 12.0%。

原文 · arXiv cs.AI

Making Preemption Productive in Long-Output LLM Serving

Reasoning LLMs generate long outputs whose key--value (KV) caches occupy GPU memory for minutes. When memory fills, the server preempts requests, and a preempted request can repeatedly lose and recover its state while producing little new output. We call this behavior KV thrashing. In certain cases, existing systems leave more than a fifth of requests stalled for over a minute. Because preemption cannot be avoided without delaying admission, we ask how to make it productive. We present Stint, a serving system that preserves a preempted request's work, bounds how long it waits, and gives it fresh output after it returns. Stint stores fixed-length KV snapshots in a contiguity-first CPU pool with an exact-extent allocator, and moves them with fused K/V gather/scatter kernels and double buffering. Age-based readmission brings stalled requests back, and protection gives them a turn of fresh decoding before another preemption. Remaining-length estimates from decoding hidden states guide victim selection. On olympiad-math replays with Qwen3-32B on H200 GPUs, no request pauses for more than 50 seconds under Stint and p99 time per output token falls by 50.7\% relative to recomputation in the same engine. The replay as a whole finishes 10.7\% sooner (12.0\% higher throughput), because the requests that baselines starve no longer finish last.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Stint、Qwen3-32B、KV缓存、推理服务,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Ziji Shi, Xinyi Wan, Chaoyi Ruan, Guangxing Huang, Min Lin, Jialin Li),经 AITOP 多维评分管线结构化提炼。截至目前,全网已有 2 家独立信源对该事件进行交叉印证与跟进报道(含 Gorden Sun 等),属于近期高关注度行业信号。