SPIN：面向在线持续学习的新型优化器
From Generic to Dedicated: A Novel Optimizer for Online Continual Learning
有人给在线持续学习专门造了个优化器，直接替换 SGD 或 AdamW 就能涨点，连无回放场景都有效，做持续学习的可以看看。
一篇 arXiv 论文提出 SPIN（Stability-Plasticity INterpolation）优化器，针对在线持续学习（OCL）中单遍数据流导致的灾难性遗忘问题。该方法将 Polyak-Ruppert 平均用作"可塑性"引擎、Primal averaging 用作"稳定性"锚点，在两条序列之间插值以解耦二者。理论分析表明 SPIN 隐式采用逆 Hessian 近似，通过 Tikhonov 正则化抑制单样本 Hessian 估计的噪声。实验显示，用 SPIN 替换 SGD、AdamW 等标准优化器后，多种 OCL 方法均获得一致提升，包括无回放（rehearsal-free）和严格内存受限的设置。
From Generic to Dedicated: A Novel Optimizer for Online Continual Learning
Online Continual Learning (OCL) requires models to learn from non-stationary data streams under a strict single-pass constraint, making them highly susceptible to catastrophic forgetting. While existing studies have explored various strategies from data, architecture, and optimization perspectives, the optimizer often directly uses generic approaches such as SGD and AdamW. In this work, we reveal a new link between optimization dynamics and OCL needs by recasting Polyak-Ruppert averaging as the engine of "plasticity" and Primal averaging as the anchor of "stability". This inspires our novel optimizer, SPIN (Stability-Plasticity INterpolation), which explicitly decouples plasticity and stability by interpolating between a fast-moving "plasticity" sequence and an adaptive "stability" sequence. We theoretically demonstrate that SPIN implicitly employs an inverse Hessian approximation, providing crucial Tikhonov regularization to damp noisy, single-sample Hessian estimates and mitigate forgetting. More importantly, SPIN can be seamlessly integrated into existing OCL methods by taking resulted gradients as input and replacing standard optimizers. Extensive experiments show that using SPIN instead of standard optimizers yields consistent and significant performance gains across multiple OCL approaches, including the challenging rehearsal-free setting and under strict memory constraints.