MirrorMoE 提出缓解 MoE 路由探索困境的新方法
Rethinking Routing Exploration in Modern Mixture-of-Experts
研究 MoE 的人可以看看这篇，它指出共享专家会抑制路由探索，并给出 MirrorMoE 这个修正方案，多规模实验都有效。
arXiv 论文分析了现代 MoE 架构中细粒度专家分割与共享专家两种设计的冲突：共享专家会降低 token 表示的多样性，削弱路由探索性，而细粒度分割组合扩大的路由空间反而更需要探索。论文提出 MirrorMoE，通过按 token 自适应抑制不必要的专家来缩小探索空间。该方法在多个模型规模上均带来一致的性能提升。
Rethinking Routing Exploration in Modern Mixture-of-Experts
Mixture-of-experts (MoE) models have become an important approach for scaling model parameters. Modern MoE architectures increasingly combine two design choices: fine-grained expert segmentation and shared experts. The former increases expressiveness by enabling more expert combinations at a fixed activation ratio, and the latter introduces experts that are active for every token. We identify an exploration dilemma between these designs: shared experts can make routing less exploratory by reducing the variability of token representations. This conflicts with the greater need for exploration induced by fine-grained expert segmentation due to the combinatorially enlarged routing space. Consequently, the additional expressiveness offered by fine-grained segmentation may not be fully realized in practice. MirrorMoE is proposed to reconcile this exploration dilemma by shrinking the exploration space through token-adaptive suppression of unnecessary experts. MirrorMoE consistently improves performance across model scales.
