Callo:在 prefill-decode 分离架构下利用失效 KV 加速缓存未命中请求
Callo: Serving Cache-Miss Requests with Speculative Decoding on Invalidated KV under Prefill-Decode Disaggregation
跑智能体服务的朋友应该看看这篇。上下文一改缓存就作废,vLLM 只能干等重算,Callo 拿废弃 KV 先投机生成再验证,实测对 vLLM 提速 1.05-1.19 倍。做推理服务优化的工程师和智能体平台团队都能直接参考。
学术论文 Callo 针对智能体会话中频繁的上下文编辑问题提出改进方案。上下文编辑会使已缓存的 KV 状态失效,请求必须等待重新 prefill 才能继续解码,带来额外延迟。
Callo 的核心思路是把失效的 KV 直接用于投机解码的 draft 阶段,等重新 prefill 完成后再用精确 KV 对草稿 token 逐一验证,通过才提交。系统还会在解码资源空闲时段调度投机计算,并在预判草稿接受率低时主动抢占,避免浪费算力。
评估在双 GPU 测试床上进行,使用 SWE-agent 会话和三个模型家族,对比 vLLM 在中等负载下取得 1.05-1.19x 的全请求加速;基于真实 trace 的集群级仿真显示该收益可随规模扩大保持。
Callo: Serving Cache-Miss Requests with Speculative Decoding on Invalidated KV under Prefill-Decode Disaggregation
Agent sessions may involve frequent context edits, which effectively invalidate cached KV state and force requests to wait for re-prefill before decoding. We present Callo, a disaggregated serving system that leverages invalidated KV for speculative drafts to reduce this latency, while verifying drafts against the exact KV after prefill before committing tokens. Callo makes effective use of decode resources by scheduling speculations during periods of underutilization and preempting speculations when drafts are likely to not be accepted. We evaluated Callo on a two-GPU testbed using SWE-agent sessions across three model families, observing all-request speedups of 1.05-1.19x over vLLM at moderate load. We demonstrate that these gains are also realized at scale through cluster-level simulations driven by real-world traces.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:投机解码、vLLM、KV Cache、prefill-decode分离,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Jianjun Ou, Matthew Lentz),经 AITOP 多维评分管线结构化提炼。