研究揭示 max pooling 学习中的两类相变机制
Phase Transitions in Learning Driven by Extremal Data
理论向读者看这篇：把 MIL 里常用的 softmax 近似 max 讲透了，还用随机能量模型和 Hopfield 网络解释了梯度下降何时失效，推导和模拟都对得上。
一篇 arXiv 论文研究极值回归问题，这是多示例学习（MIL）的一个实例，样本以 bag 形式给出并经 max pooling 汇总标签。由于 max pooling 会导致梯度消失，现代 MIL 流程用带可学习锐度参数 λ 的 softmax 近似。作者在大 bag 数 n、高维 d、α=ln n/d 固定的极限下推导出梯度流的低维投影，发现两个相变：一个关联无序系统随机能量模型的内在冻结相变，另一个是 Modern Hopfield Networks 中的检索相变。数值模拟验证了相图和训练梯度收敛到预测的渐近形式。
Phase Transitions in Learning Driven by Extremal Data
We study an extremal regression problem, an example of multiple instance learning (MIL), where the predictors are given as multisets, called bags. The bag elements (called instances) possess hidden labels which are pooled to produce an observed bag-level label. Max pooling is common in MIL applications, but gives rise to vanishing gradients. To handle this, modern MIL pipelines approximate the max with a softmax, treating the sharpness as a learnable parameter, denoted $λ$. When the max pooling model is the true data generating process, learning will be successful when $λ$ is increased during gradient descent, thereby driving the dynamics into regions of parameter space with vanishing gradients. To understand the conditions under which gradient descent breaks down, we study a minimal model and derive a low dimensional projection of the gradient flow in the joint limit of large bags $n$ and high dimension $d$ with $α=\ln n/d$ fixed. The flow has two distinct phase transitions: an intrinsic freezing transition related to the random energy model of disordered systems, and a retrieval transition occurring in modern Hopfield networks. Numerical simulation confirms both the phase diagram and the convergence of the finite-size training gradient to its predicted asymptotic form.