FUTRANSFORMER：Hopper GPU 上的 FP8 prefill 与 INT4 decode 推理实现
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
一篇把 Llama 推理在 H100 上抠到极致的 CUDA 论文，decode 比 PyTorch 快最多 2.4 倍，做推理优化的值得看代码思路。
FUTRANSFORMER 是一个面向 NVIDIA Hopper GPU 的 CUDA 实现，针对 Llama 风格 Transformer 块做推理优化。prefill 阶段采用 FP8 输出与延迟缩放，在单张 H100 上最高比编译后的 PyTorch FP8 快 1.48 倍，达到 1.29 PFLOP/s。decode 阶段用 INT4 投影权重和 INT4 KV-cache，在 8B 与 70B 形状、batch 1-32 范围内比 PyTorch BF16/INT4 块快 1.3-2.4 倍。Llama-3.1-8B 单步 decode 耗时 2.8 ms，而编译版 transformers 实现需要 8.1 ms。量化质量方面，WikiText-2 上 INT4 权重路径将困惑度从 6.24 升至 6.99（GPTQ），旋转 INT4 cache 额外增加 0.20。
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache placement, folding the fixed RMSNorm gain into weights, writing FP8 output directly with delayed scaling, and supporting prefill without padding. Decode uses INT4 projection weights and an INT4 KV-cache copy. Weights and keys follow Tensor Core fragment layouts for direct dequantization into MMA operands without full intermediate tiles. A block-diagonal Hadamard transform reduces cache quantization error from outlier channels. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s. For gate/up projection with SwiGLU at the 8B shape, our BF16 path is 1.13x faster than CODA. Our FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path. Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point: 2.1-3.3x faster than compiled, CUDA-graphed BF16 and 1.3-6.3x faster than the same block using PyTorch's INT4 projection kernel, which targets a single query row and loses to BF16 at batch 32. It is also 31-51% faster than our FP8 path. Complete Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation. Decoding every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round-to-nearest weights or 6.99 with GPTQ in the same format; the rotated INT4 cache adds 0.20. The kernels match a quantized reference computation within 0.001 perplexity.