无训练 MoE 专家剪枝研究：替换算子与门控缩放对结果影响更大
What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning
做 MoE 剪枝的值得看：光是把门控缩放选对就能差 8 个点，还给了个便宜好用的低秩替换算子。
一项针对 training-free MoE 专家剪枝的研究在 OLMoE、DeepSeek-V2-Lite、Moonlight 和 Granite 四个路由器上比较了三类替换算子。研究发现通常被隐式处理的幸存专家门控缩放（gate scale）是影响最大的因素：对未经 top-k 重归一化训练的路由器，采用重归一化会损失 0.8-8.5 个点，并改变重路由与 MoNE 均值输出替换之间的优劣。论文提出 tilted novice，一种剪枝专家输入的闭式低秩映射，在 50% 剪枝下仅增加 1.2-1.8% 参数，就在 OLMoE 和 Moonlight 上比常数替换分别高 3.45 和 1.37 个点。评估方式也会改变结论：25% 剪枝时 MoNE 常数在多选题基准保留 92-97% 准确率，但在 GSM8K 上只剩 5-65%，代码任务只剩 0-17%。作者还提出五项报告清单，并指出校准域对掩码的影响是重采样的 4-9 倍。
What We Keep of What We Forget: Replacement Operators and Routing Semantics in Training-Free MoE Expert Pruning
Training-free expert pruning makes two decisions: which experts to remove, and what the layer computes when its router still selects a removed expert. Published methods change both at once. Holding the mask fixed, we compare replacement operators from three families on OLMoE, DeepSeek-V2-Lite and Moonlight, with Granite as a fourth router, using paired, multiplicity-corrected tests. The largest effect is a choice usually left implicit: the survivors' gate scale. On routers trained without top-k renormalization, renormalizing the survivors' gates, a common implementation choice, costs 0.8-8.5 points and reverses whether rerouting beats MoNE's mean-output replacement; on two routers trained with renormalization, the trained scale wins at 50% pruning. We introduce the tilted novice, a closed-form low-rank map of the pruned expert's input whose rank-0 case is MoNE's constant. At 50% pruning it adds 1.2-1.8% parameters and gains 3.45 and 1.37 points over the constant on OLMoE and Moonlight, stable across calibration draws and largely unchanged by 4- and 3-bit quantization. No operator dominates: trained-scale rerouting is best on DeepSeek-V2-Lite, and a partial copy of each pruned expert is statistically indistinguishable from the tilted novice on two models. Evaluation can change conclusions: at 25% pruning MoNE's constant keeps 92-97% of dense multiple-choice accuracy but only 5-65% of GSM8K and 0-17% of code scores, and the calibration domain moves the mask 4-9 times more than resampling. We propose a five-item reporting checklist.
