用信息预算量化 MLP 对标签噪声的记忆：89 组 MNIST 实验研究
An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons
一篇挺扎实的理论+实验论文，用 89 组 MNIST 跑出十个假设来验证神经网络怎么记住标签噪声，结论边界也写得清楚。
研究用统计复杂度来衡量多层感知机（MLP）记住了多少标签噪声。89 组 MNIST 实验预设了十个假设，其中五个得到支持，记忆信息量 M 上升但 Fano 型界显示其不提供上界。当所有标签被替换成错误类别时，测试准确率跌至 0.5%，预测仍保留真实类别的信息。在最高 40% 的噪声比例下，统计复杂度的峰值出现在记忆阶段之前，而 280 组扩展实验（Fashion-MNIST、pair-flip 噪声）在均匀噪声下结论类似，在 pair-flip 噪声下复杂度在两个 epoch 内就达到峰值。
An Information Budget for Label-Noise Memorization, and What Statistical Complexity Tracks: A Study with Multilayer Perceptrons
How much label noise has a network memorized, and when? We study a multilayer perceptron and what statistical complexity adds to a description of its predictions. Standard identities decompose predictive information into signal, memorization, and heterogeneity. The memorization term vanishes in the population under the stated noise-independence condition. A Fano-type bound relates it to training accuracies, but the memorized fraction M provides no upper bound. Across 89 MNIST runs with ten pre-specified hypotheses (five supported, four unsupported, one not testable), normalized memorization information rose with M on training samples, remained near zero on held-out samples with fresh noisy labels, and was close to its Fano bound. Unlike raw M, it has a zero population null under the independence condition and chance correction largely removed this difference. When every label was replaced by an incorrect class, test accuracy fell to 0.5%, yet predictions retained information about the true class. Mean statistical complexity peaked before memorization at corruption rates up to 40%. At higher rates, its peak fell within the memorization phase, near the epoch when mean training confidence reached 0.6 and the two could not be separated. As a stopping rule, it matched established rules only at low noise and detected corrupted samples less well than simple label-free scores. An exploratory extension of 280 runs covering Fashion-MNIST, pair-flip noise, and four training variants yielded similar results under uniform noise, but not pair-flip noise, where complexity peaked within two epochs. Conclusions are limited to this architecture, two small datasets, and synthetic noise.
