论文

研究:Claude Opus 与 Gemini 版本迭代导致审稿结论大幅漂移

Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review

精选理由

用 ICLR 2027 在审论文做的预注册审计,发现 Opus 换个版本接收率直接掉 52 个百分点,Gemini 也掉 53 个。正在拿 LLM 当审稿人或评审工具的团队必须看看:每个模型版本都得重新和人类决定校准,旧阈值不能沿用。

一项在 ICLR 2027 审稿期间预注册的审计测试了 OpenAI、Google、Anthropic 的二十个前沿模型。结果显示,Claude Opus 与 Gemini 的前后两个版本之间,论文被判定为接收的比例分别下降 52 和 53 个百分点(183 和 60 篇抽样),另外六组配对比较中降幅为 43 到 79 个百分点,且没有任何论文评分上升。在提示词中明确写出 30% 接收率后,两个 Opus 版本之间的差距仍有 79 个百分点。

正文

机制上,把旧版生成的评分剥离后的评审意见交换给新版模型,仅评审文本本身就让 Opus 的接收比例下降 35 个百分点、Gemini 下降 51 个百分点;即使由第三方模型打分、或重写后保留全部技术观点只去掉评价性措辞,Opus 的降幅仍分别为 35 和 24 个百分点。这说明漂移主要来自模型对评价性表述的倾向变化,而非技术内容判断。此外,二十个模型在 475 份评审中给出 2 或 3 分的占 473 份,而人类评审 72 份中只有 60 份如此集中。

结论层面,模型版本迭代会让基于 LLM 评分的阈值失效:Opus 4.6 与 5.5 在 57 篇论文上的百分位相关性为 0.73,但接收比例相差 44 个百分点。同时,回顾性测试存在记忆污染风险——Claude Opus 5.5 仅凭标题就能正确说出 27 篇已决 ICLR 2026 论文的第一作者且零错误,Opus 4.6 对其自称认识的 11 篇已决论文全部给出接收,远超 5 篇的预期基线。研究者建议每个模型版本都需对照人类决定重新校准阈值,针对 ICLR 2027 决定的验证测试定于 2026 年 12 月 16 日进行。

原文 · arXiv: OpenAI

Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer Review

Venues and benchmarks read LLM review scores as measurements of a paper, while providers replace their models within months. We audit twenty frontier models from OpenAI, Google and Anthropic on ICLR 2027 submissions while they were under review, with tests registered before the decisions. Between an earlier and a later version of Claude Opus and of Gemini, the share of papers placed on the accept side falls by 52 and 53 percentage points on 183 and 60 randomly drawn submissions, and by 43 to 79 points in six further paired comparisons, with no paper moving up in any of them. Stating a 30% acceptance rate in the prompt leaves a gap of 79 points between the two Opus versions. When score-stripped critiques are swapped between versions, the later critiques alone lower the accept share by 35 points for Opus and 51 for Gemini; for Opus the drop is 35 points under a third-party scorer and 24 after a rewrite that keeps each technical point and removes evaluative wording. On 57 papers, the percentiles of Opus 4.6 and 5.5 correlate at 0.73 while their accept shares differ by 44 points. On 99 to 119 submissions, the self-estimated percentiles of three models correlate with two human reviewers more strongly than a third reviewer does (point estimates 0.17 to 0.21, 95% intervals between -0.04 and +0.42). All twenty models give a 2 or a 3 in 473 of their 475 reviews, against 60 of 72 human reviews. Retrospective tests on decided papers can draw on memory. Claude Opus 5.5 names the true first author of 27 decided ICLR 2026 papers from their titles alone, with no wrong name, and Claude Opus 4.6 accepted all 11 decided papers it said it recognized, against 5 expected. Thresholds on LLM-reviewer scores therefore have to be validated against human decisions for each model version; our tests against the ICLR 2027 decisions follow on 16 December 2026.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Claude Opus、Gemini、ICLR、LLM审稿,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始信息发布自一手权威信源「arXiv: OpenAI」(Tier-1 一手发布),具备高确定性参考价值。