BitDecoding-FP4：FP4 KV 缓存解码的算法-系统协同设计
BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
KV 缓存量化到 FP4 还不掉精度，B300 上吞吐直接翻倍多，长上下文部署的可以看看这份开源实现。
BitDecoding-FP4 针对长上下文 LLM 推理中 KV 缓存占用内存和带宽的问题，提出原生 FP4 KV 解码方案。算法层面用 Dual-Residual Query Quantization 将 query logit 误差从 60% 降到 1%，并用 Underflow-Aware Probability Scaling 和 Outlier-Aware Key Quantization 抑制另外两类误差。系统层面通过转置 softmax 融合概率量化消除 softmax 瓶颈。在 NVIDIA B300 上对 Qwen3-4B 和 Qwen3-30B-A3B 的测试中，精度保持在 BF16 的 0.5 点（NVFP4）或 1 点（MXFP4）以内，峰值服务吞吐提升 2.4-3.0 倍。代码已在 GitHub 开源。
BitDecoding-FP4: Native FP4 KV Cache Decoding via Algorithm-System Co-Design
The key-value (KV) cache dominates memory capacity and bandwidth in long-context LLM serving. While 4-bit microscaling formats (NVFP4, MXFP4) shrink the cache and run attention on FP4 Tensor Cores, a fundamental gap remains between algorithmic quantization and system-aware acceleration: existing FP4 KV systems use vanilla quantization that causes severe accuracy drops, whereas accuracy-oriented 4-bit algorithms rely on vector codebooks whose lookup overhead degrades decoding throughput. We present BitDecoding-FP4, an algorithm--system co-design for accurate and efficient native FP4 KV decoding. We mitigate three error sources: Dual-Residual Query Quantization splits queries into two FP4 tensors, cutting query logit error from 60% to 1%; Underflow-Aware Probability Scaling prevents E4M3 scales from rounding long-context probabilities to zero; and Outlier-Aware Key Quantization combines static smoothing, Hadamard rotation, and scale search to suppress key outliers under token-wise scaling. At the system level, we remove the softmax bottleneck via transposed softmax with fused probability quantization and stacked query MMA on paged KV caches. Across dense and MoE models with GQA and MLA, BitDecoding-FP4 stays within 0.5 points of BF16 KV accuracy with NVFP4 (1 point with MXFP4), and tracks BF16 KV in RL training rollouts. In end-to-end serving on NVIDIA B300 with Qwen3-4B and Qwen3-30B-A3B, it delivers 2.4-3.0$\times$ higher peak serving throughput than BF16 KV. The implementation is available at https://github.com/OpenBitSys/BitDecoding-FP4.