MoE 路由干预研究:专家路径如何影响 Qwen3-30B 输出
What MoE Routing Controls: Causal Interventions on Expert Trajectories
这是篇挺有意思的可解释性工作:直接给 Qwen3-30B 的路由加偏置就能控制输出格式。搞 MoE 或可控生成的人可以看看,中间层才是关键这个发现挺有用。
arXiv 论文(编号 2611.22120)研究了混合专家模型中的路由控制问题,实验对象为 Qwen3-30B-A3B。研究方法是在生成时向目标专家的 router logit 加入固定偏置,把 token 强制引导到由某概念或角色常用专家构成的目标路径上。
研究发现两类目标的行为差异明显:格式路径(format path)被引导后能实际改变模型写出的文本,而概念路径(concept path)虽能把 next-token 概率拉向该概念——即使路径是从另一语言构建的——但对最终生成文本影响很小。作者通过分段生成实验找到解释:格式一旦开始就会自我强化。实验还显示 religion 和 money 两个概念引导效果最强,且效应集中在模型中间层,作者进一步用嵌入区分度和词频来解释不同概念间效应强度的差异。
What MoE Routing Controls: Causal Interventions on Expert Trajectories
In mixture-of-experts (MoE) language models, a token is routed to a sequence of experts across layers. We investigate how forcing tokens onto chosen expert paths influences model behavior. We build a target path from the experts that a concept's or role's tokens most often use, and we add a fixed bias to each target expert's router logit during generation to steer the token toward this expert path. We study two kinds of targets. Format targets concern the form of the output, and concept targets concern its topic. In Qwen3-30B-A3B, we show that steering a format path changes the text the model writes. We also show that steering a concept path shifts the model's next-token probabilities toward the concept, even when the path is built in another language, yet this shift has little effect on the generated text. We investigate why formats reach the text but concepts do not by steering only part of the generation, and find evidence that a format, once started, reinforces itself. We show that the concepts religion and money steer most strongly, and that the effect is concentrated in the middle layers. We attempt to explain why the strength of the effect varies by concept through an analysis of embedding distinctiveness and word frequency.
本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:MoE、Qwen3、可解释性、路由控制,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Mukund Athmaram, Manoj Rao, Liang Ma),经 AITOP 多维评分管线结构化提炼。