时序快照模型研究预训练中知识的习得差异
What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
一篇挺有意思的预训练分析论文，用不同年份网页快照训练的模型看知识是怎么学进去的，TVS 这个指标可以直接借鉴。
论文提出时序变异性评分（TVS），用多个基于不同年份网页快照训练的 point-in-time 模型，计算条目负对数似然的变异系数来衡量知识在时间上的稳定性。研究在 6 个基准上评估来自多个开源模型家族的 6 个模型，发现高 TVS 条目在训练后期提升更快，预训练结束时在 36 个模型-基准组合中的 34 个里优于低 TVS 条目，平均优势为 7.3 个点的机会归一化准确率。该规律在知识密集型任务上最明显，即使对比难度相近的条目也成立。
What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
Language models pretrained on web corpora collected over multiple years learn both general and domain-specific knowledge. How these different kinds of knowledge are learned as pretraining progresses remains unclear. Recent point-in-time models trained on web snapshots from different years provide a way to probe how knowledge prevalence varies over time. Using these models, we propose a temporal variability score (TVS), defined as the coefficient of variation of an item's negative log-likelihood across models. A higher score suggests that the knowledge may be less consistently represented across years of web text. We evaluate six benchmarks covering diverse topics using six models from multiple open-source model families. We find that low- and high-TVS items are learned similarly early in training, but high-TVS items improve faster later on. At the end of the pretraining run, high-TVS items outperform low-TVS items in 34 of 36 model-benchmark settings, with an average advantage of 7.3 points of chance-normalized accuracy. This pattern is most robust on knowledge-intensive tasks, even when comparing items of similar difficulty. Our findings reveal differences in how knowledge is learned over the course of pretraining, with temporal variability providing a diagnostic of these learning dynamics beyond aggregate benchmark scores.
