UC San Diego 研究：长文本检索失败主因是干扰内容而非长度
UC San Diego 这篇论文说明长文本丢分不是长度问题，是相似但错误的语句抢走了注意力，RAG 场景该先去重。
UC San Diego 的论文测试了 LLM 长上下文失败的真实原因。在 Qwen2.5-1.5B 上加入 32K 无关文本后，准确率仍保持在 97%-100%。但当上下文中出现与答案相似但数值错误的语句时，准确率跌至 16%-33%。126,473 次试验显示，注意力偏离正确段落的程度可解释 58% 的错误差异，而输入长度最多解释 8%。论文建议在 RAG 和智能体记忆场景中优先去重并消解冲突语句。
Great paper from UC San Diego on why long-context retrieval fails.
(bookmark it)
It also offers an interesting discussion of why context rot happens in LLMs.
When a model fails on a long input, people usually blame the length. This paper finds that length alone does very little. Adding 32K tokens of unrelated text left accuracy at 97%-100% on Qwen2.5-1.5B.
What hurts is text that looks like the answer. When the context also holds similar statements with wrong values, accuracy falls to between 16% and 33%. A single statement that gives a different value for the same question is enough to cause a large drop.
The model fails when those statements pull its attention away from the correct passage. Across 126,473 trials, the amount of attention left on the correct passage explains up to 58% of the variation in errors. Input length explains at most 8%.
For RAG and agent memory, dedupe and resolve conflicting statements before they reach the model. That matters more than trimming tokens.
https://t.co/4tsHECWR9v
Chat with Paper: https://t.co/vDtVPCQKWf