vLLM 引入 push-based NIXL 连接器，分离式推理 TTFT 提升最高 3 倍
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
AWS 给 vLLM 的分离式推理换了种 KV cache 传输方式，首 token 延迟最高快 3 倍，做推理部署的可以看看怎么配。
vLLM 的 prefill-decode 分离式推理需要在不同引擎实例间传输 KV cache，传输协调过程会拖慢响应速度。AWS 在 Amazon EC2 P5en 实例上用 EFA 网络测试了新的 push-based NIXL 连接器，把传输协调从关键路径上移除。基准结果显示 TTFT（首 token 延迟）最高提升 3 倍，TPOT（每输出 token 延迟）提升 30%。文章介绍了具体的部署与配置方法。
Push-based KV transfer to improve TTFT and TPOT for disaggregated inference in vLLM
vLLM's prefill-decode disaggregation splits LLM inference across engine instances but must transfer the KV cache between them. This post introduces a push-based NIXL connector that moves transfer coordination off the critical path, delivering up to 3x better TTFT and 30% better TPOT in benchmarks on Amazon EC2 P5en instances with AWS EFA.
