论文精选

Twindex:粗到细选择的索引缓存卸载方案,提升长上下文稀疏注意力吞吐

Twindex: Index Cache Offloading with Coarse-to-Fine Selection for Long-Context Sparse-Attention Serving

精选理由

搞长上下文推理部署的可以看看这篇。KV 卸载后 indexer 缓存反而吃掉七成 HBM,这个瓶颈很多人没意识到。Twindex 不改权重就能把单卡并发请求数提最多 3 倍,代价只是分数降 3% 以内,对做推理服务的团队很实用。

长上下文解码中,HBM 容量限制单卡可并发的请求数。研究发现,在 Qwen3.8-Flash-Next、DeepSeek V4.1-Flash 等稀疏注意力模型把主 KV 卸载到主机内存后,indexer 缓存反而成为 HBM 瓶颈:在 1M token 下,它占 Qwen 单请求 HBM 的 63%、V4.1 的 75%。

正文

Twindex 的做法是:HBM 里只保留一份紧凑的 indexer 缓存副本,完整缓存放主机内存。每步解码时,副本先粗选出更大的候选集,再由模型自带的 indexer 从主机内存中的原始行里挑出 top-k。副本由共享组件加压缩残差组成,均离线学习一次;整个过程不改模型权重,且能在 CUDA graphs 内运行。

在 1M token 上下文下,相比仅做 KV 卸载,Twindex 在 Qwen、V4.1、GLM-5.2 上分别可容纳 1.78×、1.62×、3.03× 的请求数;主机内存不限制批大小时,解码吞吐分别达 1.44×、1.25×、2.54×。基准测试的平均分数最多比基线低 3.1%。

原文 · arXiv: DeepSeek

Twindex: Index Cache Offloading with Coarse-to-Fine Selection for Long-Context Sparse-Attention Serving

In long-context decoding, GPU memory (HBM) caps how many requests a GPU decodes at once. Sparse-attention models such as Qwen3.8-Flash-Next, DeepSeek V4.1-Flash and GLM-5.2 add an indexer: at every step it scores all past index entries with its own indexer cache, and attention reads the main KV of only the top-k entries. We find that once serving systems move the main KV to host memory, the indexer cache becomes the HBM bottleneck, at 63% of per-request HBM on Qwen and 75% on V4.1 at 1M tokens. Therefore, we introduce Twindex, which keeps only a compact copy of the indexer cache in HBM and the full indexer cache in host memory. At each step, the copy selects a larger candidate set, and the model's own indexer picks the top-k from the original rows of the candidates in host memory. The copy combines the shared component with the compressed residuals, both learned once offline. Twindex changes no model weights and runs inside CUDA graphs. At 1M tokens on modern GPUs, Twindex fits $1.78\times$, $1.62\times$ and $3.03\times$ as many requests as KV offloading alone on Qwen, V4.1 and GLM-5.2, and reaches $1.44\times$, $1.25\times$ and $2.54\times$ decode throughput when host memory does not limit the batch. Every request keeps its own index, and Qwen and GLM requests share prefix main KV in host memory. In our evaluations, mean benchmark scores are at most 3.1% below the baseline's.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Twindex、DeepSeek、Qwen、GLM-5.2,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。