LeanSparse：面向原生稀疏注意力 LLM 的省显存推理系统
LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs
长上下文推理太吃显存？这个 LeanSparse 系统跑 DeepSeek-V4-Flash，950K 上下文并发翻近 3 倍，首 token 延迟从 21.6 秒干到 0.6 秒。
LeanSparse 是一个推理服务系统，在 SGLang 中实现并在 DeepSeek-V4-Flash 上评估。针对稀疏注意力长上下文场景，它在 prefill 阶段用两个可复用 GPU 缓冲区逐层流式加载历史数据，在 decode 阶段用粗到细索引减少原始索引键的显存占用。在 950K 上下文长度下，其并发请求数达到 HiSparse 的 2.95 倍，decode 吞吐量达到 2.14 倍。在 Open-SWE 多轮对话轨迹上，相比 ECHO，中位首 token 延迟从 21.6 秒降到 0.6 秒，会话完成时间中位数降低 17.7%。
LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs
Sparse attention limits the history read by each query, but serving long contexts still requires substantial GPU memory. Existing KV offloading systems retain the full history of indexer keys on GPU, limiting decode concurrency. Prefill's temporary memory demand can further delay admission even when enough memory remains for the same request's subsequent decode. We present LeanSparse, a serving system that reduces both decode's persistent memory use and prefill's temporary memory demand. Our design exploits two observations: prefill consumes historical data one layer at a time, and decode can identify promising candidates before comparing their original indexer scores. During prefill, LeanSparse streams layer histories through two reusable GPU buffers, overlapping next-layer loading with ongoing model computation. During decode, coarse-to-fine indexing searches compact representations and retrieves original indexer keys only for candidate refinement. Separate indexer key and KV caches reuse data across generation steps. We implement LeanSparse in SGLang and evaluate it on DeepSeek-V4-Flash. At 950K context length, LeanSparse supports 2.95$\times$ as many concurrent requests and achieves 2.14$\times$ decode throughput over HiSparse. On multi-turn Open-SWE traces, it reduces median time to first token from 21.6 to 0.6s and median session completion time by 17.7% relative to ECHO.
