Task Context 如何改变语言模型中共享注意力头的角色
Task Context Shapes Component Roles in Language Models
研究共享注意力头在不同任务里角色怎么变，Gemma-2-9B 和 Qwen3-14B 都做了激活编辑实验，做可解释性的朋友可以看看。
论文提出用网络视角分析语言模型计算，通过相同的 graph facts 语句比较固定组件在不同任务下的作用。实验覆盖 4 个家族、1.5B-14B 参数的 5 个开源权重模型，发现任务切换会重新分配各注意力头的影响。在 Gemma-2-9B 和 Qwen3-14B 上，激活编辑在保持书面指令不变的情况下切换模型读取的 fact，同一批头随后响应不同的 graph facts。作者为每个 prompt 构建前后两个线性响应映射，对应条件的映射在 Gemma 上将均方预测误差降低 53.6%，在 Qwen 上降低 5.0%。
Task Context Shapes Component Roles in Language Models
Large language models reuse the same attention heads across tasks, but identifying these shared heads does not explain what they do differently in each task. A natural source of inspiration is task-dependent interactions among shared brain regions, which motivate a network view of language-model computation. Here, we compare fixed model components across tasks using identical graph facts, statements specifying which directed connections are present. We measure how interventions affect attention heads and the activation state in later layers (influence), as well as the answer (importance). Task changes redistribute the influence in five open-weight models spanning four families and 1.5-14 billion parameters. In Gemma-2-9B and Qwen3-14B, an activation edit changes an early internal activation vector and switches which fact the model reads while the written instruction stays fixed. The same heads then respond to different graph facts and influence later components differently. For each prompt, we construct two linear response maps, one before the edit and one after it. Each map predicts a later head's response from a measured change in an earlier head. For a separate factual intervention, the map for the corresponding condition reduces mean squared prediction error by 53.6% in Gemma and 5.0% in Qwen compared with the average of the two maps. These findings show that task context changes both which graph facts the shared components respond to and how their outputs affect later computation. A network account may help explain how models use shared parameters to perform different computations. Measurements of task-dependent influence could also guide interventions intended to alter one behavior while preserving others.
