CertiSum:让 MXFP4 训练在不同平台与并行布局下结果完全一致
CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts
跨厂商 GPU 和并行方式训练结果居然会不一样,这坑很多人踩过。这篇提出 CertiSum,让 4-bit MXFP4 训练逐位可复现,实测在 AMD 和 NVIDIA 卡上都一致,速度还比 BF16 快。做分布式训练和审计的朋友值得花时间看看。
CertiSum 是一种让 MXFP4(4-bit 微缩放格式)训练实现逐位一致(bitwise-identical)的方法,解决同一输入在不同并行布局和 GPU 厂商之间产生不同结果的问题。
其核心观察是:MXFP4 乘积的 FP32 原生累加在多数求和顺序下本身就是精确的。CertiSum 在量化阶段收集统计信息来认证矩阵乘法结果,对未通过认证的输出进行精确重算,在认证几乎不成立的场景改用精确 kernel。它保持量化块在不同布局下不变,并在注意力与权重梯度计算中合并不带舍入的局部和。
在 200 步 Qwen3-8B-Base 实验中,CertiSum 在 AMD MI350X 与 NVIDIA GB200 之间以及 MI350X 上数据、张量、上下文并行布局切换时逐位保留训练状态。在 MI350X 上对 FineWeb-Edu 的三个 8B–14B 模型测试,它比非精确 MXFP4 对应实现增加 11.8–60.5% 的单步延迟,同时相对 BF16 最高加速 1.24×,相对确定性 BF16 最高加速 1.70×。
CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts
Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors. Its key insight is that native FP32 accumulation of MXFP4 products is often already exact for every summation order. CertiSum collects statistics during quantization to certify native matrix multiplication results, recomputes uncertified outputs exactly, and uses exact kernels directly where certification rarely succeeds. CertiSum keeps quantization blocks unchanged across layouts and merges partial sums without rounding, including in attention and weight-gradient computation. In 200-step Qwen3-8B-Base experiments, CertiSum preserves training states bit for bit across AMD MI350X and NVIDIA GB200 and across changes in data, tensor, and context parallelism on MI350X. Across three 8B--14B models on FineWeb-Edu using MI350X, it adds 11.8--60.5% to training-step latency relative to its non-exact MXFP4 counterpart at the larger evaluated input shapes, with speedups of up to $1.24\times$ over BF16 and $1.70\times$ over deterministic BF16.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:MXFP4、低精度训练、可复现性、Qwen3,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Steve Rhyner, Yubo Gao, Stefan de Lasa, Sankeerth Durvasula, Nandita Vijaykumar),经 AITOP 多维评分管线结构化提炼。截至目前,全网已有 4 家独立信源对该事件进行交叉印证与跟进报道(含 TestingCatalog、CoreWeave、The Information 等),属于近期高关注度行业信号。