Lie bracket 或成灾难性遗忘早期预警信号:无需旧数据即可预测
Does Learning Order Explain Forgetting? The Lie Bracket as an Early-Warning Signal for Catastrophic Forgetting Without Old Data
用一阶 Lie bracket 预测遗忘,这个想法很新颖,而且实测只需要存梯度和 20 个 Hessian 特征对,不用保留旧数据,比 EWC 的 rho 高出不少。做持续学习的人可以看看它的适用边界,快速遗忘场景它还是打不过 EWC。
论文实证检验了 Dherin (2023) 的猜想:两个任务梯度场的 Lie bracket 可能与灾难性遗忘相关。在 Split MNIST、Split Fashion-MNIST 和 Split CIFAR-10 三个基准上,任务 A 的遗忘量几乎精确等于两种训练顺序下 A 损失之差,Spearman rho 达 0.96-1.00。
关键实验只需在任务 A 结束时存储其梯度和 top 20 Hessian 特征对,即可在不接触 A 数据的情况下、在任务 B 训练早期预测遗忘。当任务 A 学得充分且遗忘缓慢(task-incremental 场景)时,bracket 排序遗忘的相关性达 rho = 0.70-0.81,高于 Elastic Weight Consolidation 惩罚项的 0.33-0.58,并达到或超过在 A 原始数据上重新评估的效果。
该信号有明显边界条件:domain-incremental 等快速遗忘场景下存储摘要会失效,EWC 惩罚反而更好;旧任务只学到一半时,A 上残留梯度比正常情况大一个数量级,一阶损伤占主导,bracket 方法完全失效。作者结论是它只适用于缓慢遗忘场景的早期预警。
Does Learning Order Explain Forgetting? The Lie Bracket as an Early-Warning Signal for Catastrophic Forgetting Without Old Data
A network trained on task A and then task B ends at different weights than one trained on B and then A. To leading order this gap is set by the Lie bracket of the two tasks' gradient fields, and Dherin (2023) conjectured that a non-vanishing bracket may be linked to catastrophic forgetting. I test this conjecture empirically. Across Split MNIST, Split Fashion-MNIST and Split CIFAR-10, the forgetting of A is almost exactly the difference in A's loss between the two training orders (Spearman rho = 0.96-1.00). I then ask whether the bracket can predict that forgetting early in training on B and without any of A's data, using only a small summary stored when A finishes: its gradient and its top 20 Hessian eigenpairs. When the old task is well learned and forgetting is slow (task-incremental), this stored bracket ranks forgetting with rho = 0.70-0.81, against 0.33-0.58 for the penalty of Elastic Weight Consolidation, and matches or exceeds re-evaluating A on its own data. When forgetting is fast (domain-incremental), the stored summary goes stale and the EWC penalty is as good or better. The bracket fails outright when the old task is only half learned; I trace this to a residual gradient on A an order of magnitude larger, which makes first-order (slope) damage matter. The bracket is therefore a useful early-warning signal for the quiet kind of forgetting, under a clear condition.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:持续学习、灾难性遗忘、Elastic Weight Consolidation、Split CIFAR-10,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Ephraim Kofi Ayi Kwapong),经 AITOP 多维评分管线结构化提炼。