研究：用户评价 LLM 的标准在提高，模型本身没变差
AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models
一篇挺有意思的实证研究：把 Chatbot Arena 上 126 万条投票拆开分析，发现模型没变差，是用户眼光变高了。做产品的人应该有共鸣——你原地不动，用户就觉得你退步了。做 RLHF 数据的人更该看看这篇。
arXiv 论文分析 Chatbot Arena 上 2023 年 5 月至 2024 年 8 月的 1,260,967 条盲投数据，选取版本固定的模型配对（含带日期的厂商快照与固定权重的开源模型），检验用户评价标准是否随时间漂移。
结果显示，对同一对未变动的模型，"两个都不好"的投票比例以每月 1.71 个百分点的速度上升（均值为 17.1%），2024 年达每月 1.91 个百分点；在 2023 与 2024 年都有足够观测的 14 组配对中全部上升，从 15.5% 升至 23.3%。但相对排名没有趋势：2024 年较强模型在决定性投票中的份额保持平稳。
该漂移在同一投票者、同一配对、主要语种、有无提示词控制和答案长度稳定等条件下均成立，且可由其他模型配对估计出的月份效应消除。平台层面，按 2023 年标准平均模型本应得 78%，而"都不好"票占比最终回到起点附近。论文结论是评价标准随品类进步水涨船高，产品停滞会被感知为倒退，这对 LLM 排行榜和用于训练模型的人类反馈数据都有影响。
AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models
Users often complain that AI assistants such as ChatGPT are getting worse. Some versions do change, but human evaluation leaves a second possibility untested: the people judging may change. We separate the two on Chatbot Arena, a public platform where people compare answers from two anonymous large language models (LLMs) and pick the better one or say both are bad. We study 1,260,967 blind votes cast from May 2023 to August 2024 on pairs of models whose versions are fixed: dated vendor snapshots and models with fixed open weights. For the same pair of unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points a month on a mean of 17.1%, and by 1.91 points a month in 2024. In fourteen pairs observed often in both 2023 and 2024, it rose in every one, from 15.5% to 23.3%. The ranking does not trend: in 2024, the stronger model's share of decisive votes is flat. The rise holds within the same voter and on the same pair, and across arrival cohorts from their first recorded vote. It holds with prompt controls, in every major language, for dated vendor snapshots, open-weight models and models with stable answer length, and between interface changes. The drift is shared across models: month effects estimated from other models' pairs remove it from each model's own pairs. Platform-wide, the average model would be expected to score 78% against the 2023 average, yet the share of "both are bad" votes ended near where it began. The pattern fits a standard that follows the category's progress, so a product that stands still is judged as declining. Absolute verdicts on AI drifted while relative comparisons held steady, an evaluation drift that matters for LLM leaderboards and for the human feedback used to train models.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Chatbot Arena、LLM、RLHF、ChatGPT，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Zihan Chen, Lei Nico Zheng, Di Zhu），经 AITOP 多维评分管线结构化提炼。
