研究验证 MAGIC 轨迹数据归因在模型能力习得阶段的有效边界
MathaMAGICal: Validating Trajectory Data Attribution During Capability Acquisition
做数据筛选的人应该看看这篇:它实测了 MAGIC 这类影响函数在 86M 和 152M 模型上的表现,发现短 checkpoint 窗口的 LDS 能到 0.86 以上,但全程归因基本失效,数据块排序也不能跨 run 迁移。想用影响函数挑训练数据的团队,这篇能帮你避开明显的坑。
arXiv 论文《MathaMAGICal》测试了影响函数方法 MAGIC 在 86M 与 152M 参数 GPT-2 风格模型从零学习多步算术时的轨迹数据归因效果。在 86M 规模上做全程归因时出现数值溢出,即便使用 metagradient 裁剪,分数与重训练结果的相关性接近随机,线性 datamodeling 分数(LDS)仅为 0.17–0.22。
改为使用短的 checkpoint 窗口后,152M 模型在四个 100-step 窗口上的 LDS 达到 0.86–0.99,而单 checkpoint 的梯度点积基线相关性接近零。单个训练数据块的排序只在同一次训练内有效,跨种子或超参数的迁移性极差(ρ≤0.09),但格式层面的模式可以跨 run 复现。worked examples 和 primitive facts 两类数据在最有益与最有害的 5% 分数尾部中占据语料份额的 2.1–2.6 倍。
结论是轨迹数据归因在经过验证的短窗口内可靠,且能产出跨 run 泛化的 family 级洞察;但多轮次家族删除时的净收益与局部敏感性方向并不一致,开发者用影响函数挑数据时需要重新审视做法。
MathaMAGICal: Validating Trajectory Data Attribution During Capability Acquisition
Influence-function-based methods like MAGIC provide exact first-order scores for trajectory data attribution, yet only retraining indicates whether those scores accurately predict data removal. We evaluate MAGIC on $86$M- and $152$M-parameter GPT-2-style models learning multistep arithmetic from scratch on filtered web text and four synthetic math families. Full-run attribution at $86$M suffers from numerical overflow, and even with metagradient clipping, scores correlate near chance with retraining (linear datamodeling score, LDS, $0.17$--$0.22$). In contrast, short checkpoint windows yield stable, accurate predictions at both scales (LDS $0.86$--$0.99$ across four $100$-step windows at $152$M), whereas a single-checkpoint gradient-dot baseline shows near-zero correlation. Rankings of individual training blocks predict retraining only within that single run and do not transfer across seeds or hyperparameters ($ρ\le 0.09$), whereas format-level patterns replicate across runs. Two data families, worked examples and primitive facts, occupy $2.1$--$2.6\times$ their corpus share in both the helpful and harmful $5\%$ score tails, reflecting strong bidirectional local sensitivity distinct from net benefit under multi-epoch family deletion. These results show that trajectory attribution is reliable within short, validated windows and yields family-level insights that generalize across runs.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:MAGIC、GPT-2、数据归因、影响函数,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Jason Lee, Dikshanta Pokhrel, Phonn Nay Khant, Victor Lee, Moritz Weckbecker, Sebastian Lapuschkin, Wojciech Samek, Mark Riedl, Glenn Matlin),经 AITOP 多维评分管线结构化提炼。