研究探讨归一化层对网络逼近能力的影响与宽度代价
Activation-Adjacent Layer Normalization: Effects on Approximation Power
一篇分析 LayerNorm 和 RMSNorm 放进网络里要多花多少宽度才能保住表达力的论文，结论给得很具体，做架构设计的可以看看。
arXiv 论文研究了在激活函数旁插入 LayerNorm 或 RMSNorm 后，浅层网络能否逼近宽度为 N 的单隐层 MLP。论文将 RMSNorm 视为输入相关缩放，通过宽度增广在紧致特征集上局部线性化归一化操作。结果显示 AN 网络在激活无界时需宽度 N+1，连续非常数激活需宽度 N+r 且误差为 O_f(r^{-1})；PN 网络在仿射参数或一次正齐次性下达到宽度 N+1。论文证明 N+1（RMSNorm）和 N+2（LN）同时是最坏情况界，数值实验在全部五种设置中验证了预测的逼近速率。
Activation-Adjacent Layer Normalization: Effects on Approximation Power
Layer normalization (LN) and root mean square normalization (RMSNorm) depend on their inputs and remain nonlinear at inference. Inserting either next to an activation may therefore change the function a network represents. We study when shallow activation-normalized (AN) and preactivation-normalized (PN) networks can approximate a prescribed width-$N$ one-hidden-layer MLP. Viewing RMSNorm as input-dependent scaling, we use width augmentation to locally linearize normalization on compact feature sets. For AN networks, an unbounded activation range allows width $N+1$, while every continuous nonconstant activation admits width $N+r$ with error $O_f(r^{-1})$. For PN networks, affine parameters or, without them, degree-one positive homogeneity recover width $N+1$; general continuous activations admit width $N+r$ with error tending to zero. The corresponding LN constructions use centered coordinate pairs, yielding widths $N+2$ and $N+2r$, respectively. We prove that $N+1$ for RMSNorm and $N+2$ for LN are also the worst-case bound for approximations, and show that applicable universal-approximation theorems for ordinary MLPs transfer to these normalized architectures. Numerical experiments recover the predicted approximation rates for both normalizations across all five settings and illustrate the shifted-ReLU finite-width obstruction.
