论文

微调能换来多少解码确定性?边际正则化抑制批处理导致的 token 翻转

How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips

精选理由

一篇挺有意思的研究:贪心解码其实不是确定性的,同一 prompt 换个 batch 就可能输出不同 token。作者用 logit 边际正则化在 GSM8K 上把发散率从 53.8% 压到 38.1%,代价 1-3 个准确率点。做需要可复现输出的服务或评测的同学可以看看。

arXiv 论文(编号 2611.19135)研究了零温度贪心解码在批处理下并非确定性的问题:同一 prompt 因与其它请求共享 batch 而产生不同 token。作者在六个模型家族、三种数值精度上测量这种扰动,发现扰动幅度与可表示精度严格对应(bfloat16 对 float16 为 8.00 倍,float16 对 float32 为 8192.0 倍,与尾数位数预测一致),且某些张量形状会意外实现批不变性,仅靠设定序列长度即可获得零不确定性。

正文

在此基础上,论文提出对 top-1 / top-2 logit 差距施加 hinge 惩罚,缩放到实测噪声分位数的两倍。在 GSM8K 上,token 翻转减少 35%,发散率从 53.8% 降至 38.1%,代价是 1.3–2.9 个准确率点,且随边际大小单调增加。由于 hinge 是软惩罚,幸存翻转的 logit 差距中位数(0.125)在各实验组中保持不变。

这一方案把确定性保障从推理阶段(如批不变 kernel,以吞吐量为代价)部分转移到训练阶段,为需要可复现输出的场景提供了训练时选项。论文还检验并否定了七个假设,其中一个关于单模型可行性的诊断在六个模型上成立、在第七个上被证伪。

原文 · arXiv cs.AI

How Much Determinism Can Fine-Tuning Buy? Margin Regularisation Against Batch-Induced Token Flips

Greedy decoding at zero temperature is widely assumed to be deterministic, although the exact same prompt may yield different tokens depending on the number of other prompts sharing the batch for inference. The currently available solutions operate during inference, trading determinism for throughput. Can we train a model in such a way that fewer decoding steps would be susceptible? We characterize the perturbation empirically across six model families and three numerical precisions and observe two empirical facts about the measurement: the perturbation depends exactly on the representable precision (8.00x for bfloat16 vs float16, 8192.0x for float16 vs float32, in line with predictions based on mantissa bits) and the batch invariance happens accidentally at certain tensor shapes, allowing a practitioner to achieve zero nondeterminism merely by setting the sequence length. Then we apply hinge penalization of the model's top-1 / top-2 logit gap, scaled to twice the measured noise quantile. On GSM8K dataset it leads to a 35% reduction in token flips and divergence from 53.8% to 38.1%, at the cost of 1.3 - 2.9 accuracy points, increasing monotonically with the margin size. It does not affect the gap threshold sufficient for determinism: the logit gaps of surviving flips are unchanged (median 0.125 for all arms), as the hinge is a soft penalty and the largest hardware perturbation remains the same. We reject seven hypotheses, including a per-model viability diagnosis, holding on six models and falsified on the seventh.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:GSM8K、微调、解码确定性、数值精度,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Charan Teja Nagisettygari, Siva Raja Sindiramutty),经 AITOP 多维评分管线结构化提炼。