多轮预训练的缩放理论：集成缓解后期过拟合
A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
数据不够用之后预训练怎么办？这篇论文给了理论答案：与其训一个大模型，不如训练多个小模型做集成，还能推迟早停时机。
一篇 arXiv 论文研究数据总量受限时语言模型预训练的缩放规律。在 FineWeb 数据集上的实验显示，logit 平均的集成方法能减缓训练后期的性能恶化，在相同参数-token 预算下可超越更宽或更深的单模型。作者推导出多轮随机梯度下降的动力学平均场理论（DMFT），追踪跨 epoch 和跨模型的误差相关性。基于集成降低预测方差的机制，论文给出固定数据量下的算力分配法则：集成规模随算力增长的速度至少与单模型训练时长相当。
A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling
In the near future language model pretraining will become bound by total data rather than total compute. Such a regime requires revisiting scaling laws and scaling law theory. Motivated by prior works that have found utility in ensembling, we study how the number of training epochs and ensemble size jointly shape generalization when training data are fixed. In language-model experiments on FineWeb, logit averaging reduces late-training deterioration, delays optimal stopping, and can outperform wider or deeper single models at matched parameter-token budgets over a common training horizon. To analyze this interaction between data reuse and ensembling, we derive a dynamical mean-field theory (DMFT) for multi-epoch stochastic gradient descent in power-law random-feature ensembles, tracking error correlations across epochs and models. Ensembling reduces the prediction variance responsible for late-time growth in expected test error, changing the optimal stopping time. We use these stopping times to derive compute-allocation laws at fixed data and individual model sizes. Under this allocation, ensemble size grows with compute at least as fast as training time per member, while excess test loss above the limiting loss decreases as a power of compute.
