Nvidia 论文发现注意力头是 RAG 噪声干扰的关键源头
做 RAG 或长上下文应用的人值得看：检索回来的噪声文本会偷偷降低准确率，这篇论文找到了具体的元凶注意力头。不用重新训练，屏蔽十来个头就能找回一半以上的损失，开源模型可直接照做。
Nvidia 与多个实验室联合发表论文，指出无关文本对模型的主要伤害方式是削弱模型对问题本身的利用，而不是直接对抗正确答案。即便是一句'Computers are useful'这样的无关内容，也会降低模型对正确答案的支持度。
研究者在 7B 规模的开源模型内部追踪了这一机制，发现一组被称为 intrusion heads（入侵头）的注意力头：它们读取无关文本并削弱答案置信度。关键发现是，只屏蔽这些头对上下文的注意力，不会影响其他正常利用有效证据的注意力头。
实验覆盖 4 项任务，屏蔽约 10 个头平均可恢复 57% 被干扰项损失的准确率；需要额外一次干净运行做参照的 oracle 修复可恢复 97.6%。对部署开源权重模型、处理噪声检索上下文的团队来说，这种免训练的屏蔽方案可以低成本挽回大部分准确率损失。
New Nvidia + other labs paper shows that irrelevant text hurts mainly by weakening the model's use of the question, not by directly pushing against the right answer.
Prompts in RAG, tool use, and long-context apps often carry text that has nothing to do with the question. Even a line like "Computers are useful" lowers a model's support for the right answer.
They traced this inside open models and found intrusion heads, a small set of attention heads that read the unrelated text and weaken the answer. Masking those heads' attention to the context leaves the heads that use helpful evidence untouched.
Across 4 tasks on 7B-scale models, masking about 10 heads recovered 57% of the accuracy lost to distractors on average. An oracle fix that needs a matching clean run recovered 97.6%.
If you serve open-weight models with noisy retrieved context, a training-free mask on a few model-specific heads can win back much of the lost accuracy.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Nvidia、RAG、attention heads、开源模型，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「rohanpaul_ai」（@rohanpaul_ai），经 AITOP 多维评分管线结构化提炼。
