研究显示 MoE 模型合并后的路由漂移不能直接判定为路由失败
Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
合并 MoE 模型时看到路由乱了先别急着修,这篇用 DeepSeekMoE 等三个模型证明漂移不等于失败,还给了工具包和修复代码。
arXiv 论文针对 DeepSeekMoE、OLMoE、Qwen3-MoE 三种 MoE 模型,提出一套路由分析工具包,通过反事实干预和 token 级分析研究合并后的路由漂移。
结果显示,大部分专家重新分配来自输入变化而非同层参数变化,且源相对路由差异难以预测恢复源路由后 next-token 似然的提升。论文将路由失败操作化定义为:在固定非路由参数下,特定路由干预可挽回的任务损失,并据此提出 Selective Router Repair(SRR)案例。
分析工具包与 SRR 代码已开源。
Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
Model merging efficiently combines specialized large language models (LLMs) without joint retraining, but can substantially alter expert routing in Mixture-of-Experts (MoE) models. Such \emph{routing drift} is often interpreted as routing failure, raising a fundamental question that remains unclear: \emph{does routing drift after MoE merging actually indicate routing failure, and what evidence should justify repair?} We investigate these questions across DeepSeekMoE, OLMoE, and Qwen3-MoE proposing a routing analysis toolkit for controlled counterfactual interventions and token-level analysis. By crossing source and merged router inputs and parameters, we attribute most expert reassignments to input shifts rather than parameter changes at the same layer. However, source-relative routing differences poorly predict next-token likelihood gains from source-route restoration, and different expert selections can produce directionally similar mixture outputs. We therefore operationalize routing failure as \textit{task loss recoverable under a specified routing intervention, with non-routing parameters fixed.} These tests detect recoverable loss under deliberate router corruption, whereas source-route restoration does not establish reliable task benefits in the evaluated merged models. Motivated by these, we propose \emph{Selective Router Repair (SRR)} as a case study, and find that source-specialist token-likelihood advantages do not reliably identify beneficial local corrections. Together, these findings show that \textbf{routing drift alone is insufficient evidence of routing failure}: source-informed corrections must be judged by their task-level intervention effects. The analysis toolkit and SRR code are released.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:MoE、DeepSeekMoE、Qwen3-MoE、OLMoE,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Yuanyi Wang, Yanggan Gu, Su Lu, Guanghao Zhu, Pengkai Wang, Yifan Yang, Congkai Xie, Zhaoyi Yan, Jianmin Wu, Hongxia Yang),经 AITOP 多维评分管线结构化提炼。