论文揭示前缀缓存的智能体服务存在亚稳态：一次突发可致吞吐腰斩70分钟
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
跑过 vLLM 智能体服务的人应该都见过那种莫名的性能崩塌，这篇论文把它解释清楚了：一次几小时的突发流量就能让命中率掉到 10%，吞吐腰斩还自己好不了。论文给了能用的预算器方案，不用调参，直接把 KV 池撑爆的可能性掐掉。
arXiv 论文研究了前缀缓存（prefix caching）在智能体推理服务中的失效模式：一次命中只需增量预填充约一千 token，一次未命中则要重算 10–40K token 的上下文。实验显示，在相同基础负载下，几分钟的额外到达就会让 vLLM 服务器进入退化状态——缓存命中率降至约 10%，吞吐量减半，积压在剩余 70 分钟内持续增长，而无突发的对照组保持健康（12 对中 9 对如此，反向从未出现）。
机制在于：服务变慢使更多会话滞留在系统中，工作集超出 KV 缓存池，之后每一轮都变成全量重算。该退化在缓存预热服务器本可承受的负载下出现，无前缀缓存时不存在，用 Qwen3-32B 跑 SWE-bench 智能体轨迹、换用 SGLang 引擎同样能复现，SGLang 需要更大的触发条件。
应对方面，限制新会话进入的限流策略只救回 7 个崩溃运行中的 1 个；按容量上限挂起已接入会话则全部在几分钟内恢复。CPU offload 层只能延迟而非阻止崩溃。作者提出的预算器将活跃会话的驻留 KV 需求限定在缓存池内，阈值无需按负载调参，在所有测试中都防止了故障。
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
LLM agents drive inference engines with closed-loop sessions whose context grows every turn, and serving them economically depends on prefix caching: a hit costs an incremental prefill of about a thousand tokens, a miss recomputes a 10--40K-token context. We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime--a ~10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes--while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction). The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers. Slower service keeps more sessions in the system, their working set outgrows the KV pool, and every turn becomes a recomputation; a first-order workload-balance condition, calibrated independently, brackets when the regime can drain. Getting out is the harder problem: throttling new admissions recovered 1 of 7 matched collapsed runs, while suspending admitted sessions against a capacity-based limit recovered all of them within minutes. A CPU offload tier delays the collapse but does not prevent it. A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：vLLM、SGLang、prefix caching、KV缓存，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Wen Huang, Jack Liu），经 AITOP 多维评分管线结构化提炼。
