研究提出用数据归因指导预训练数据筛选
Data Attribution Should Steer Pretraining
这篇论文把数据归因从"事后分析"变成"事前指导",思路很实在:归因发现的规律能跨语料复用,比一次性排序省钱。做预训练数据筛选的工程师可以直接拿五类证据框架当 checklist 用。
arXiv 论文《Data Attribution Should Steer Pretraining》主张把训练数据归因(training-data attribution)从一次性工具升级为预训练数据筛选的核心机制。论文指出,文档的价值取决于模型当前已掌握的知识,因此静态的质量评分难以反映真实贡献。
作者提出的方案是一个反馈回路:先用归因方法解释哪些数据源真正起作用,再把可解释的数据属性转化为可复用的筛选标准,在新语料和不同 checkpoint 上反复验证。论文还明确了五类主张各自需要的证据类型,指出历史贡献、可解释性和未来数据价值需要不同的证据支撑。
论文最后提出一个可证伪的对比实验设计:在总计算成本对齐的前提下,与 perplexity、分类器和混合比例三类基线方法比较筛选效果。这套框架对做预训练数据工程和低成本数据策展的团队有直接参考价值。
Data Attribution Should Steer Pretraining
Pretraining data curation decides what a model learns from, yet what makes a document worth learning from depends on what the model already knows. Influence-driven selectors already rank documents to improve one run; we argue that training-data attribution should steer pretraining by identifying testable data properties that become reusable curation criteria, evaluated on new data against cheaper selectors at matched total cost, discovery included. A document ranking is consumed by the run it was computed for, whereas an explained data property can be tested across corpora and checkpoints, especially where familiar quality signals diverge from measured usefulness. Because historical contribution, an understandable explanation, and future data value require distinct evidence, we specify a feedback loop that interprets attributed sources, tests candidate data changes, and states for five claim types what evidence each requires and what none establishes. We close with a falsifiable comparison against perplexity, classifier, and mixing baselines at equal total compute.
本条情报归属 AITOP「论文」核心追踪栏目。涉及核心实体与技术议题包括:training-data attribution、数据归因、预训练、数据策展,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Glenn Matlin, Chandreyi Chakraborty, Mohammed Rehan Parwani, Saehee Eom, Anthony Wen-Ming Zang, Akhil Theerthala, Arjun Chatterjee, Tejas Khandwekar, Brandon Michaels, Alvin Deng, Mark Riedl),经 AITOP 多维评分管线结构化提炼。