FP4 预训练让模型学罕见事实多花约 1.2 倍曝光，验证损失却看不出
What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
这篇实验做得很细：FP4 训练损失几乎一样，但罕见事实要多喂 1.2 到 1.5 倍数据才学会。做量化训练或挑低精度方案的人该看看，尤其里面 Adam 噪声那组对照实验能帮你避坑。
arXiv 论文（编号 2611.12832）用模拟 4-bit 矩阵乘法从零训练 transformer，在合成传记数据上控制每条事实的曝光次数，检验低精度预训练是否真的只影响损失。结果显示：NVFP4 需要 BF16 的 1.32 倍到 1.52 倍曝光才能匹配事实召回率，但两者验证损失仅相差 0.5% 到 2%；换用 BF16 矩阵乘法评估时差距降到 1.26 倍至 1.30 倍，主读数下匹配损失时的曝光成本约 1.2 倍。
机制上，零均值梯度噪声可以复现只量化反向传播时的成本，而这种量化几乎不改变损失。受控实验发现，进入 Adam 二阶矩估计的噪声相当于调低学习率、成本会消退；进入更新本身的噪声会降低信噪比，约三分之二的成本一直延续到训练结束。单纯调高学习率无法弥补变小的平均步长，对噪声和 NVFP4 都不奏效。
作者还证明用 BF16 检查点的梯度统计即可对各种量化格式、部分量化和噪声控制方案的成本排序，Spearman 相关系数达 0.96，无需实际拟合。结论是评估低精度训练方案时应把罕见事实习得和损失一起测。
What the Loss Curve Misses: FP4 Pretraining Slows the Acquisition of Rare Facts
Low-precision pretraining is usually judged by validation loss. We ask whether matching loss also means learning rare facts as efficiently. We train transformers from scratch with emulated 4-bit matrix multiplications on synthetic biographies with controlled exposure counts, on their own and mixed into web text. NVFP4 needs 1.32$\times$ and 1.52$\times$ as many exposures as BF16 to match its recall while validation loss differs by only 0.5% and 2%, and 1.26$\times$ and 1.30$\times$ when both models are evaluated with BF16 matrix multiplications. Under our primary readout, the exposure cost is about 1.2$\times$ at matched loss. Zero-mean gradient noise reproduces the cost of quantizing only the backward pass, which barely moves the loss. In a controlled experiment at twice the NVFP4 error, noise fed only to Adam's second-moment estimate acts like a lower learning rate and its cost fades, whereas noise in the update lowers its signal-to-noise ratio and carries about two thirds of the cost through the end of training. Raising the learning rate to offset the smaller average step does not recover the cost, either for this noise or for NVFP4. Gradient statistics of BF16 checkpoints rank the costs of formats, partial quantizations and noise controls without fitting (Spearman 0.96). We suggest measuring rare-fact acquisition alongside loss when evaluating low-precision recipes.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：FP4、NVFP4、BF16、低精度训练，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Chang Lu, Mengjiao Zhang），经 AITOP 多维评分管线结构化提炼。