RoCoKV:面向推理模型的 KV Cache 压缩方法,vLLM 中实现 1.27X 加速
RoCoKV: Accelerating LLM Inference Serving through Uncertainty and Redundancy Aware KV Cache Compression
推理模型的账单大头其实是长链条推理吃掉的显存和算力。这篇把压缩算法和 vLLM 内核一起改了,DeepSeek-R1-Distill-Llama-8B 上吞吐提升 38%,不牺牲准确率。自建推理服务、调过 KV Cache 的人应该看看。
arXiv 论文提出 RoCoKV,一个专为大型推理模型设计的 KV Cache 压缩方法,针对推理过程中循环、自我验证、纠正产生的冗余 token 难以安全移除的问题。方法包含两个组件:不确定性引导的自适应剪枝,根据模型置信度动态调整压缩强度,置信时加大剪枝力度;冗余感知的 token 选择,降低与近期推理内容高度相似 token 的权重,保留信息更丰富的 token。
工程上,RoCoKV 与推理服务栈协同设计,开发了融合的三阶段 CUDA decode kernel,并以即插即用插件形式集成进 vLLM,保留其 paged KV layout 和 CUDA graph capture。在 AIME25 等三个推理基准、两个推理模型上测试,对 DeepSeek-R1-Distill-Llama-8B,相比 Full-KV 实现 1.27X 加速、38% 吞吐提升,长合成基准上达 1.43X 加速、45% 吞吐提升,且在受限 KV 预算下准确率优于现有 SOTA 压缩方法。
推理模型生成链条长、KV Cache 成本高是推理服务的主要开销来源,RoCoKV 将算法节省转化为真实端到端加速,对部署 DeepSeek-R1 类推理模型的服务团队有直接参考价值。
RoCoKV: Accelerating LLM Inference Serving through Uncertainty and Redundancy Aware KV Cache Compression
KV cache compression is critical for large reasoning models, where most of the cost comes from long generated reasoning traces rather than the input prompt. These traces often contain redundant and low-utility tokens due to looping, self-verification, and corrections. However, such tokens may still receive high attention scores, making it difficult to remove them without harming the reasoning process. We propose RoCoKV, a KV cache compression method designed for large reasoning models. RoCoKV introduces two key components. First, uncertainty-guided adaptive pruning adjusts the compression strength based on the model's confidence, allocating more memory during uncertain phases and applying stronger pruning when the model is confident. Second, redundancy-aware token selection reduces the impact of tokens that are highly similar to recent reasoning content, encouraging the model to retain more informative and diverse tokens. Beyond the algorithm, we co-design RoCoKV with the serving stack: we develop fused three-phase CUDA decode kernels and integrate them into vLLM as a drop-in plugin that preserves its paged KV layout and CUDA graph capture, turning algorithmic savings into real end-to-end speedups. Extensive experiments across three reasoning benchmarks and two reasoning models demonstrate that RoCoKV consistently improves reasoning accuracy under constrained KV cache budgets, achieving higher accuracy compared to state-of-the-art KV cache compression methods. For DeepSeek-R1-Distill-Llama-8B in vLLM, RoCoKV achieves a 1.27X speedup and 38% higher throughput on AIME25, the most challenging evaluated reasoning benchmark, and a 1.43X speedup and 45% higher throughput on the long synthetic benchmark, compared with Full-KV.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:RoCoKV、DeepSeek-R1、vLLM、KV Cache,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。