单变量两层 ReLU 分类网络的最小范数解与全局最优性研究
Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections
一篇理论论文，搞清楚了两层 ReLU 分类器什么时候解唯一、什么时候全局最优，还证明加个 skip connection 就能消除次优局部解，做理论的朋友可以看看。
arXiv 论文研究单变量两层 ReLU 网络在二分类任务中的最小范数插值与 ℓ2 正则化 logistic 损失最小化。论文给出了最优分类器在函数空间中的完整几何刻画：偏置不加惩罚时，最小范数插值器是紧贴每个标签切换点的连续分段仿射函数；偏置受惩罚时，解在函数空间中唯一，且每个同标签段内恰好有一个弯折点。论文还证明加入自由仿射 skip connection 后，约束问题的每个 KKT 点都成为全局最优解，而没有 skip connection 时可能出现次优 KKT 点，并配套了不同数据复杂度与网络宽度下的数值实验。
Minimal-Norm Univariate Two-Layer ReLU Classification: Exact Solutions and Global Optimality with Skip Connections
We study minimal-norm interpolation and $\ell_2$-regularized logistic-loss minimization for binary classification by univariate two-layer ReLU networks. We give complete geometric characterizations of the optimal classifiers in function space, resolving how the solutions depend on whether hidden-layer biases are included in the parameter norm. When biases are unpenalized, the minimal-norm interpolators are exactly the continuous piecewise-affine functions that hug every label switch and have kinks of the appropriate convexity. When biases are penalized, the minimizer is unique in function space, has exactly one kink in each intermediate same-label segment, and is therefore a sparsest positive-margin classifier. We further show that adding a free affine skip connection leaves these function-space solutions unchanged but fundamentally improves the parameter-space landscape: every KKT point of the constrained problem becomes globally optimal, whereas suboptimal KKT points can occur without the skip connection. We establish analogous global-optimality and geometric results for sufficiently weak $\ell_2$-regularization of the logistic loss. In the unpenalized-bias case, we identify an additional sparsity-like restriction, implying that most minimal-norm interpolators cannot arise as small-regularization limits of margin-normalized logistic-loss minimizers. Numerical experiments across varying dataset complexity and network width support the predicted landscape and sparsity phenomena.
