Apple 研究：多语言数据混合优于超参数搜索
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
Apple 这篇论文挺有意思：低资源语言训练别死磕权重衰减这类超参数，混点高资源语言数据进来效果更好，做小语种模型的话值得看看方法细节。
Apple 发表论文《Mix, Don't Tune》，针对低资源语言预训练中数据需重复多轮导致泛化下降的问题，比较两种补救方案。研究显示，混入高资源辅助语言数据（通过可调混合比例扩充训练分布）比高权重衰减等超参数调整更有效。结论指出在数据受限场景下，做双语数据混合比反复调超参数收益更大。
Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings
For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to directly aid the low-resource target. While hyperparameter tuning regularizes the model by shrinking weights to restrict network capacity, auxiliary data mixing uses a tunable mixing ratio to expand the training distribution and diversify the training signal with new…
