XM-Sparse:基于扩展内存池的稀疏注意力 LLM 推理系统
XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools
长上下文推理卡在 KV 缓存放不进 HBM 这件事上,这篇给了个新解法:把缓存搬到 X-Mem 这类扩展内存池,还针对 KV 和索引键两套访问模式设计了不同的 I/O 路径。做 LLM 推理优化的工程师可以看看,比 PCIe 方案吞吐高近 5 倍。
论文提出 XM-Sparse,首个面向扩展内存池(scale-up memory pool,如 X-Mem)的稀疏注意力 KV 卸载系统,已在 SGLang 上实现。系统针对 DeepSeek-V3.2 与 GLM-5.3 进行测试,覆盖 16K 至 1M 上下文长度。
其核心机制是双缓存分离处理:热 KV 缓冲与常驻的索引键(K-index)缓存占用大量 HBM,且访问模式不同。对动态选中的 KV 条目,采用 TMA 检索与注意力计算的 tile 级流水线融合,隐藏按需取回延迟;对可预测的 K-index 访问,则用 GPU 发起的 DMA 预取页面,并与延迟敏感操作协调以避免干扰解码。
实验结果显示,XM-Sparse 解码吞吐相对基于 PCIe 的 ECHO 与 HiSparse 分别提升最高 4.98 倍和 4.52 倍,相比同平台直接集成方案提升最高 1.34 倍。该工作表明 GPU HBM 容量仍是长上下文并发的主要瓶颈,即使引入扩展内存池也需要为不同缓存定制 I/O 路径。
XM-Sparse: High-Throughput Sparse-Attention LLM Serving over Scale-up Memory Pools
Sparse attention has been increasingly adopted by production LLMs for long-context inference, yet serving systems must still retain the full KV history, making GPU HBM capacity a key limit on serving concurrency. Existing systems offload the KV cache to host memory, but suffer from the limited bandwidth of PCIe and the limited capacity of host memory. Emerging scale-up memory pools (e.g., X-Mem) relieve both limits and open new opportunities for sparse-attention offloading. In this paper, we first conduct an in-depth analysis and reveal that, even with scale-up memory pools, decode throughput remains bounded by HBM capacity, as hot KV buffers and the fully resident indexer key (K-index) cache occupy substantial HBM. Moreover, the two caches exhibit distinct access patterns. Based on these insights, we present XM-Sparse, the first sparse-attention offloading system over scale-up memory expansion. The core idea of XM-Sparse is to offload both caches aggressively while matching each with an I/O mechanism tailored to its access pattern. Specifically, for dynamically selected KV entries, XM-Sparse fuses TMA-based retrieval with attention in a tile-level pipeline to hide on-demand retrieval latency. For predictable K-index accesses, it prefetches pages via SM-efficient GPU-initiated DMA and coordinates prefetching with latency-critical operations to avoid interfering with decoding. We have implemented XM-Sparse atop SGLang, and our experiments on DeepSeek-V3.2 and GLM-5.3 with an emulated scale-up memory pool show that XM-Sparse delivers up to 4.98$\times$ and 4.52$\times$ higher decode throughput than PCIe-based ECHO and HiSparse, respectively, and up to 1.34$\times$ over a naive integration on the same platform, across 16K-1M contexts.
本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:DeepSeek-V3.2、GLM-5.3、SGLang、稀疏注意力,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。