研究发现长文本生成退化的关键在采样器而非模型训练
Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)
写代码智能体或者长文生成经常跑崩的朋友可以看看这篇。作者测了 29 个开源模型,发现崩不是模型问题,是 top-p 在高温下候选 token 池爆炸,换成 top-nsigma 同样的 Qwen3.5 智体能从解 0 题变成解 60 多题,代码全开源直接能用。
一篇 arXiv 论文测试了 29 个 1B 到 122B 参数的开源模型,覆盖 SWE-bench Verified、needle 检索(最长 100 万 token)、长链推理和长文本写作等十项基准,结论是:长生成退化主要由采样器决定。采样器在高温下是否让低概率 token 进入候选池,直接决定长文本和智能体任务能否存活。具体对比:温度 2 时,Qwen3.5 模型上跑 top-p、top-k 或 Top-H 的编程智能体在 100 个 GitHub issue 中解出 0 个,而换成 top-nsigma 能解出 46 到 62 个;全基准上 top-nsigma 保住温度 1 成绩的中位数 100%,top-p 只剩 26%。
机理解释也很直观:以 Llama-3.1-8B 的一个故事生成为例,温度从 1 升到 2,top-p 的候选池从 4 个 token 暴涨到 51,845 个,top-nsigma 则稳定在 3 个。跨 133 组采样器与温度组合,候选池大小与人工评判质量呈 -0.77 的 Spearman 相关。只要在温度 1 时锁定候选池、之后再加热,Top-H、min-p、p-less 都不再崩溃。
对开发者的实际意义是:智能体和长输出场景想跑高温采样,不必重新训练模型,换一个池子稳定的采样器即可。默认温度 1 下各采样器差异不大,问题集中在高温长生成。论文代码和数据全部开源。
Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)
Open-weight language models are widely observed to get worse the longer they generate. Often, attempts to fix this involve more training. At high temperature, The sampler, i.e. the rule that picks the next token out of the model's probabilities, decides whether long and agentic generation survives and/or thrives. We test 29 open-weight models from 1B to 122B parameters on ten established benchmarks plus needle retrieval out to one million tokens, covering SWE-bench Verified coding agents, long chain-of-thought reasoning, long-form writing, long structured output and long-context reading. Samplers that keep their pool of candidate tokens small and steady as temperature rises don't lose accuracy. Samplers that let in low-probability tokens fall apart. At temperature 2, coding agents running top-p, top-k or Top-H resolve 0 of 100 GitHub issues, while the same agents running top-nsigma resolve 46 to 62 and win every issue the two samplers disagree on, on three Qwen3.5 models. Over all benchmarks top-nsigma keeps a median 100% of its temperature-1 score where top-p keeps 26%. Counting how many tokens each sampler lets through explains why. In one Llama-3.1-8B story, going from temperature 1 to 2 grows top-p's pool from 4 tokens to 51,845 while top-nsigma stays at 3. Across 133 combinations of sampler setting and temperature, the typical pool size predicts judged quality (Spearman correlation of -0.77), and when we pick the pool at temperature 1 and only apply temperature afterwards, Top-H, min-p and p-less stop collapsing. At the default temperature of 1 most samplers perform about the same, We release all code and data to the public.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:top-nsigma、Qwen3.5、Llama-3.1、SWE-bench,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Allen G. Roush, Minh Nhat Nguyen, Ravid Shwartz-Ziv, Judah Goldfeder, Sanjay Basu),经 AITOP 多维评分管线结构化提炼。