SGLang HiCache 支持跨并行策略与分布式部署的 KV 缓存复用
SGLang 讲了 HiCache 怎么在并行和分布式部署下复用长前缀，讯飞实测命中率 30% 到 95%，跑智能体长上下文的值得看看。
SGLang 发布博客介绍 HiCache 如何在上下文并行、流水线并行、投机解码和 PD 分离部署下实现长前缀缓存复用。博客还提出统一 KV 布局，让兼容的并行配置之间可以复用缓存。iFLYTEK Astron MaaS 团队用 GLM-5.1/5.2 服务 agentic CodingPlan 负载，加入 HiCache 加外置存储后，多轮智能体请求（输入 30K 至 200K token，最高 1M）的缓存命中率从 30% 升到 95%，中位 TTFT 降到 3 秒以内。
HiCache Across Parallelism & Distributed Serving
Agentic workloads reuse long prefixes across many model calls. Our new blog explains how SGLang HiCache makes that reuse work with context parallelism, pipeline parallelism, speculative decoding, and PD Disaggregation serving. It also covers a unified KV layout for reuse across compatible parallel configurations.
For the iFLYTEK Astron MaaS team’s agentic CodingPlan workloads, served by GLM-5.1/5.2 with PP/PD Disaggregation/MTP, the team reports these results after adding HiCache with external storage: · Cache hit rate: 30% → 95% on multi-turn agentic workloads with inputs typically ranging from 30K to 200K tokens, and up to 1M. · Median TTFT: below 3 seconds, down from delays that had made the service largely unusable.
Thanks to @alibaba_cloud Tair KVCache, Alibaba Bailian, @radixark, @NVIDIAAI, @AntGroup and @iflytek Astron MaaS.
Read the full blog: https://t.co/kwIpXluFHc
