新研究首次证明 SGD 在深度线性网络中几乎必然收敛
Convergence of stochastic gradient descent for learning linear neural networks
搞优化理论的朋友可以看看，这篇把 SGD 在深度线性网络里的收敛性证明补上了，条件还挺宽松，不要求 Lipschitz。
arXiv 论文 2610.28561 研究随机梯度下降（SGD）在深度线性神经网络中的收敛性质，这类网络保留了非凸特性同时便于数学分析。作者在正则化平方损失下将此前梯度下降的收敛保证扩展到了 SGD，并证明了 SGD 迭代序列几乎必然有界。借助随机逼近理论中的渐近伪轨迹框架，论文在温和条件下（不依赖 Lipschitz 性质或白化数据）证明了确定性与自适应步长下 SGD 收敛到临界点集合。作者称这是首个为深度线性网络中的 SGD 提供几乎必然收敛保证的工作。
Convergence of stochastic gradient descent for learning linear neural networks
Stochastic gradient descent (SGD) is the cornerstone of modern deep learning, yet its theoretical convergence properties remain poorly understood in non-convex settings. To address this gap, we study SGD dynamics in deep linear neural networks - simplified models that retain key non-convex characteristics while remaining analytically tractable. Building on recent results for gradient descent, we extend convergence guarantees to SGD under regularized squared loss. We establish almost sure boundedness of SGD iterates. Using the framework of asymptotic pseudotrajectories from stochastic approximation theory, our analysis establishes then, for both deterministic and adaptive step-size choices, convergence of SGD to the set of critical points under mild conditions (without relying on Lipschitz properties or whitened data). To our knowledge, this is the first work providing almost sure convergence guarantees for SGD in deep linear networks, bridging the gap between theoretical optimization and practical stochastic training.
