Newton 双层优化新方法：摆脱下层强凸性限制
Newton-Based Methods for Bilevel Optimization Beyond Lower-Level Strong Convexity
双层优化以前要求下层强凸，这篇给了两个 Newton 方法放宽到 Hessian 奇异也能跑，还拿 GPT-2 数据重加权做了实验，搞优化的可以看看。
arXiv 论文提出两个 Newton 类双层优化方法。Newton-based GALET 在正谱隙条件下处理下层 Hessian 奇异的情形，避免 GALET 中的平方 Hessian 乘子系统。Newton-based BAMM 在下层目标凸、上层目标对下层变量强凸时使用强凸聚合模型。论文还给出统一框架 GNBO，并在合成问题、数据超清洗和 GPT-2 数据重加权实验中相对 GALET 和 sl-BAMM 取得更优复杂度界。
Newton-Based Methods for Bilevel Optimization Beyond Lower-Level Strong Convexity
Bilevel optimization provides a framework for machine learning tasks such as hyperparameter optimization, data reweighting, and meta-learning. Existing Newton-based bilevel methods typically rely on the classical hypergradient formula, which is generally unavailable when lower-level solutions are nonunique and the lower-level Hessian is singular. We develop two Newton-based methods beyond lower-level strong convexity by exploiting curvature shared by the lower-level and multiplier updates. Newton-based GALET handles potentially singular lower-level curvature under a positive spectral-gap condition, exploiting its range structure to avoid the squared-Hessian multiplier system used in GALET. Newton-based BAMM uses a strongly convex aggregation model when the lower-level objective is convex and the upper-level objective is strongly convex in the lower-level variable. We establish convergence guarantees and improved complexity bounds relative to GALET and sl-BAMM under the corresponding assumptions. We further introduce GNBO, a generalized Newton-based framework that captures the shared-curvature structure of the proposed methods and several existing Newton and quasi-Newton bilevel methods. Experiments on synthetic problems, data hyper-cleaning, and GPT-2 data reweighting evaluate the proposed methods across different bilevel learning tasks.