预训练与SFT检查点质量研究
Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
精选理由
论文揭示了训练流程中检查点选择的关键问题，对大模型训练实践有重要指导意义。
研究显示，在30B混合专家训练流程中，仅凭预训练损失或基准分数选择检查点可能失效。表现更好的检查点具有更高的解决方案密度，即在局部权重扰动下仍能保持下游性能。这一发现挑战了传统检查点选择方法。
原文 · arXiv cs.AI
Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack
Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the best starting point for subsequent training. We show that this assumption can fail in a full 30B mixture-of-experts training pipeline. The checkpoints that perform better after the full downstream training stack also have higher solution density, i.e., retain downstream performance under local weight perturbations.
