vLLM 双向 KV 缓存传输降低多轮智能体首 token 延迟
Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
如果你在跑多轮 Agent 服务，这篇讲 vLLM 的 KV 缓存跨节点复用，Qwen3-32B 和 GLM-5-FP8 实测 TTFT 快了 1.6 到 3.3 倍，附部署细节。
多轮智能体负载每轮都要重复处理不断增长的对话上下文，导致延迟上升。AWS 博客介绍了 vLLM 的双向 KV 缓存传输方案，让 prefill 节点和 decode 节点跨轮次复用已计算的上下文。在 Amazon EC2 P5en 实例（配合 EFA）上，Qwen3-32B 的首 token 时间最多降低 1.6 倍，GLM-5-FP8 最多降低 3.3 倍。文章给出了具体的部署配置和实现细节。
Improve TTFT for multi-turn agentic workloads with bi-directional KV transfer in vLLM
Multi-turn agentic workloads reprocess growing conversation context each turn, driving up latency. See how bi-directional KV cache transfer lets prefill and decode nodes reuse previously computed context across turns, cutting time to first token by up to 1.6x for Qwen3-32B and 3.3x for GLM-5-FP8 on Amazon EC2 P5en instances with EFA.
