研究解析 MoE 模型中 Softmax 路由坍缩与恢复机制
Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery
训练 MoE 模型卡在路由坍缩上的可以看看，这篇把概率恢复要多久推成了对数或幂律，还有对照实验验证。
一篇 arXiv 论文研究混合专家（MoE）模型中 Softmax 路由将概率集中到部分专家、抑制其余专家的现象。论文用平稳损失障碍刻画固定专家下最优路由输出所在的面，并通过法向速度分解区分互补专家协方差与相对梯度优势。在恢复路径的正双侧幂界条件下，系数的消失阶给出对数或幂律恢复时间。配对 Transformer-MoE 训练实验显示，移除类条件路由器梯度对比会延迟确认恢复并降低固定 token 上的累积对数几率。
Softmax Routing Dynamics in Mixture-of-Experts Models: Collapse and Recovery
Softmax routing in mixture-of-experts models can concentrate probability on a subset of experts, suppressing access to the others. We study how this concentration forms, why it persists, and how suppressed probabilities recover. For fixed experts, a stationary-loss barrier identifies the face supporting the optimal routed output. In a common-target square-loss model with freely trainable expert outputs, slow adaptation tracks fixed-expert dynamics on finite horizons, while every fixed positive adaptation rate yields an interior routing limit. An exact normal-velocity decomposition separates complementary-expert covariance from relative gradient advantage. Under positive two-sided power bounds along recovery paths, uniform over initial masses, the resulting coefficient's order of vanishing gives logarithmic or power-law recovery times. Integrations of constructed fields reproduce these orders. An exact discrete log-odds identity relates neural updates to threshold crossing. In paired Transformer-MoE training, removing a class-conditional router-gradient contrast delays confirmed recovery and reduces accumulated log-odds on fixed tokens, both with and without capacity constraints.
