arXiv 论文:分解 SGD 动力学,揭示正交方向噪声如何损害泛化
Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation
这篇理论论文把两层网络的 SGD 权重拆成信号和正交补两块来分析,发现补空间会被训练诱导出谱尖峰、放大方差拖累泛化。做特征学习或训练动力学研究的读者能从中拿到可复用的分析工具 Decomposed Dynamics,模拟还显示冻结正交方向反而效果更好。
一篇 arXiv 论文研究了在 multi-index teacher 模型上用 SGD 训练两层神经网络,重点分析第一层中与 teacher 子空间正交的参数对泛化的影响。作者将第一层权重分解为对齐 teacher 的信号分量和正交补空间,并在比例极限下与第二层一起分析其联合 SGD 动力学。
论文证明,尽管 SGD 对正交补空间的更新最初是各向同性的,但训练动力学会在补空间 Gram 矩阵中诱发依赖 teacher 的谱尖峰(spectral spikes)。这种涌现的各向异性提供了一个具体机制,解释表示方差如何被放大、泛化如何退化。为隔离该效应,作者提出 Decomposed Dynamics 这一解析可处理的替代模型,其渐近测试误差动力学与 SGD 一致。在 ReLU 学生的模拟中,冻结正交补空间方向比继续训练它们泛化更好;论文还发现信号分量会快速与第二层权重对齐,显示即使在浅层网络中也存在强跨层耦合。
Decomposing SGD Dynamics in Neural Networks: Teacher-Induced Spikes and Variance Inflation
We study two-layer neural networks trained by stochastic gradient descent (SGD) on a multi-index teacher model, focusing on how first-layer parameters orthogonal to the teacher subspace affect generalization. We decompose the first-layer weights into a signal component aligned with the teacher and an orthogonal complement, and analyze their joint SGD dynamics together with the second layer in the proportional limit. We show that although SGD updates to the complement are initially isotropic, the training dynamics induce teacher-dependent spectral spikes in the complement Gram matrix. This emergent anisotropy provides a concrete mechanism by which representation variance is amplified, and generalization degrades. To isolate this effect, we introduce Decomposed Dynamics, an analytically tractable surrogate whose asymptotic test-error dynamics coincide with those of SGD. In simulations with a ReLU student, freezing the directions of the orthogonal complement improves generalization relative to training them. Finally, we show that the signal component rapidly aligns with the second-layer weights, revealing strong cross-layer coupling even in this shallow setting. Together, our results provide a dynamical explanation for how feature learning outside the teacher subspace harms generalization.
本条情报归属 AITOP「论文」核心追踪栏目。涉及核心实体与技术议题包括:SGD、neural networks、特征学习、泛化,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Samet Demir, Zafer Dogan),经 AITOP 多维评分管线结构化提炼。