MultiAlloc 提出预算感知的 KV Cache 量化精度分配框架
MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization
一篇解决 LLM 推理内存痛点的论文：MultiAlloc 在固定显存预算下自动决定 KV Cache 量化精度怎么分，比现有逐层和逐 token 方法都省心。
MultiAlloc 是一个面向 KV Cache 量化的 profiling 引导精度分配框架，核心问题是在固定内存预算下如何把量化资源分配给逐层精度增强与逐 token 信息保留两种策略。它通过轻量离线 profiling 刻画层敏感度和 token 重要性，自动推导精度分配策略，无需在线搜索。在推理和长上下文基准上，该方法在相同内存预算下超过了逐层、逐 token 以及近期多粒度 KV Cache 量化方法。实验覆盖 Mistral、Llama、DeepSeek-Qwen、QwQ 等多种模型。
MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization
The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation. In practical deployment, however, the key challenge is not merely compression, but how to allocate limited quantization resources among heterogeneous precision preservation strategies under a fixed KV cache memory budget. We propose MultiAlloc, a profiling-guided precision allocation framework for KV cache quantization that adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation. MultiAlloc employs lightweight offline profiling to characterize layer sensitivity and token importance, enabling adaptive precision allocation under a fixed memory constraint. Guided by the profiled memory--accuracy trade-off, it automatically derives an effective precision allocation strategy without online allocation search, and applies the resulting policy during inference. Extensive experiments on reasoning and long-context benchmarks demonstrate that MultiAlloc consistently improves over layer-wise, token-wise, and recent multi-granularity KV cache quantization methods under matched memory budgets across diverse models, including Mistral, Llama, DeepSeek-Qwen, and QwQ.
