BellowsKV：跨层共享 KV 缓存，压缩 13 倍仍保留 97% 精度
BellowsKV: One Latent for Many Layers, Expanded on Demand
免训练的 KV 缓存压缩方案，压 13 倍几乎不掉点，比现有跨层方法强不少，自己部署 LLM 的可以看看代码。
KV 缓存是长对话和智能体场景下 GPU 推理的主要内存瓶颈。研究团队发现相邻层输入隐藏状态高度相似，据此提出免训练方法 BellowsKV：把 n 个相邻层的隐藏状态映射为每 token 一个共享潜变量，各层在推理时通过专属投影读回自己的键值。在 Llama-3.1-8B、Mistral-7B 和 Qwen2.5-7B 上，BellowsKV-Quant 在 3.6 倍压缩下保留约 99% 的全缓存精度，13-14 倍压缩下保留 97-98%，而最强跨层方法在 16 倍压缩下最多只剩 84%。
BellowsKV: One Latent for Many Layers, Expanded on Demand
Large language models (LLMs) now serve as the backbone of agents that reason over many turns, call tools, and pursue long-horizon tasks. To avoid recomputation, a model stores the attention keys and values of all past tokens in a key-value (KV) cache. Each layer maintains its own cache, so KV-cache memory can become the main bottleneck for GPU serving. Most compression methods operate within a single layer, by quantizing the cache entries, reducing the channel dimension, or evicting less important tokens. All three lose substantial accuracy at high compression ratios. Cross-layer methods share one compressed cache across adjacent layers to go beyond this limit. However, these methods also degrade sharply at high compression ratios. We find that the KV-cache representations of adjacent layers are quite different, whereas the input hidden states of adjacent layers are highly similar. We also find that, under the same memory budget, reducing the dimension of this shared latent loses far more accuracy than quantizing it. Based on these findings, we propose BellowsKV, a training-free method that maps the hidden states of n adjacent layers to one shared latent per token. At inference time, each layer reads its own keys and values from this latent through a layer-specific projection. We keep the latent at full dimension and compress it in two other ways: BellowsKV-Quant stores the latent at low precision, which is nearly lossless because the latent has no dominant outlier channels; BellowsKV-Evict makes one shared eviction decision per block, because the layers rank token importance very similarly. Across Llama-3.1-8B, Mistral-7B and Qwen2.5-7B, BellowsKV-Quant retains about 99% of full-cache accuracy at 3.6x compression and 97-98% at 13-14x, where the strongest cross-layer method retains at most 84% at 16x. Code: https://github.com/MorrisZJ/BellowsKV
