论文

KV-streams 提出流式 KV 缓存方案,训练提速 2.6-5 倍

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

精选理由

做智能体 RL 训练的朋友可以看看,KV 缓存不清空改成流式传递,训练直接快 2.6 到 5 倍,还能当循环记忆用,即插即用。

arXiv 论文提出 KV-streams,一种可即插即用的上下文压缩加速策略。它不再在每次压缩后清空 KV 缓存,而是将其向前流式传递,兼容任意压缩策略。

正文

在三种不同的压缩策略上,KV-streams 将智能体强化学习训练的墙钟时间提速 2.6 到 5 倍。

实验还发现流式传输的 KV 缓存可作为循环状态,保留早已离开上下文窗口的信息,且仅靠 RL 训练即可涌现该行为。

原文 · arXiv cs.LG

KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strategies, achieving a 2.6 to 5x wall-clock speedup in training. Beyond efficiency, we find that the streamed KV cache can act as a recurrent state, carrying forward information that has long since disappeared from the context. Specifically, in a controlled setting we show that, contrary to prior work, RL alone is all that is needed for this behavior to emerge. Overall, we show KV-streams to be an efficient and lightweight plug-and-play addition to any post-training pipeline.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:KV缓存、强化学习、上下文压缩、智能体训练,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda, Roger Creus Castanyer, Siddarth Venkatraman, Abhay Puri, Jonathan Light, Matthew James Sargent, Augustine N. Mavor-Parker, Massimo Caccia, Lucas Caccia, Glen Berseth, Esmeralda S. Whitammer, Alessandro Sordoni, Minseon Kim, Marc-Alexandre Côté, Laurent Charlin, Guillaume Lajoie),经 AITOP 多维评分管线结构化提炼。