论文

临界阻尼 Polyak 步长:让梯度下降收敛到更平坦极小值的新算法

A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes

精选理由

搞训练的朋友可以看看这篇:不用 SAM 的两次前向,一个改过的 Polyak 步长就能压低 sharpness,连 LLM 微调都验证过,还带一致时间收敛证明。

arXiv 论文提出一种新的自适应步长规则——临界阻尼 Polyak(CDP)步长,在经典 Polyak 规则基础上乘以系数 4 并引入阻尼因子,使梯度下降偏向更平坦的极小值。作者首次对一大类局部二次目标给出了 sharpness 与损失下降的一致时间保证。

正文

机制上,系数 4 会在极小值流形 M 附近产生等振震荡,阻尼项则驱动迭代收敛到 M。CDP 通过适应局部曲率运行在稳定边缘:损失较小时,其投影迭代近似沿 M 上的 log λ₁(∇²L) 做梯度下降,与 Arora 等人 2022 年分析的归一化梯度下降的平坦性偏好类似。区别在于 CDP 的步长随到 M 的距离缩短,能在有限学习率下收敛到局部最小化 sharpness 的插值解。

实证方面,CDP 产生了与 Sharpness-Aware Minimization(SAM,Foret 等,2021)类似的平坦化效果,且在理论范围之外也有效,包括非线性 teacher-student 网络和预训练 LLM 的微调。对需要无需额外前向开销即可降低 sharpness 的训练方来说,这是 SAM 之外的另一个选项。

原文 · arXiv cs.LG

A new algorithmic approach for finding flat minima: critically damped Polyak stepsizes

We present a new adaptive stepsize rule that biases gradient descent toward flatter minima in both theory and practice, and establish the first uniform-in-time guarantees for both sharpness and loss reduction for a broad class of locally quadratic objectives. Our critically damped Polyak (CDP) stepsizes multiply the classical Polyak rule by $4$ and a mild damping factor. The factor $4$ is critical in that it produces equioscillations around the manifold of minimizers $\mathcal{M}$ to leading order. Damping provably drives convergence to $\mathcal{M}$. CDP thus operates at the edge of stability by adapting its stepsizes to local curvature. Once the loss is small, its projected iterates follow gradient descent on $\log{λ_1(\nabla^2 \mathcal{L})}$ along $\mathcal{M}$ to leading order, paralleling the flatness bias of normalized gradient descent (Arora et al., 2022). Unlike normalized gradient descent, CDP step lengths shrink with the distance to $\mathcal{M}$, enabling convergence to an interpolating solution that locally minimizes sharpness at finite learning rate. We establish our theory for a broad class of non-negative losses with locally quadratic Morse-Bott structure near $\mathcal{M}$, capturing regular interpolation regimes in smooth overparameterized neural networks. Empirically, CDP stepsizes produce a flattening effect similar to that of Sharpness-Aware Minimization (SAM, Foret et al., 2021), reducing sharpness even in settings beyond our theory, including nonlinear teacher-student networks and fine-tuning of pre-trained LLMs.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Polyak步长、SAM、梯度下降、优化算法,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Francesco Caporali, Jason M. Altschuler, Boris Hanin),经 AITOP 多维评分管线结构化提炼。