模型多源确认88°

TasteVal 基准发布:量化 AI 的实验研究品味,Opus 5.5 超过人类专家

TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

精选理由

这个 TasteVal 把研究品味换算成省了多少算力来打分,Opus 5.5 用 1/30 的成本跑赢人类专家,数字挺有意思的。

新基准 TasteVal 用 8 个开放式任务评测前沿模型的实验研究品味,即给定固定研究问题后迭代设计实验并解读结果的能力。

正文

被测模型扮演 Researcher,由固定的 Coder agent 负责实现实验,预算上限为 40 H100 小时或 120 小时。团队招募 24 名人类专家建立基线,评测了 2023 至 2026 年发布的 20 个模型。

成绩最好的 Opus 5.5 超过专家基线,计算乘数为 2.

3 倍(95% CI 1.15-4.

37),单次运行成本约为人类基线的 1/30。数据还显示自 2025 年 12 月起,前沿模型的计算乘数约每 3.

0 个月翻一倍,明显快于此前的 14 个月周期。

原文 · arXiv cs.AI

TasteVal: Measuring the Experimental Research Taste of AI Systems Against Human Experts

We introduce TasteVal, a benchmark to evaluate the experimental research taste of frontier models. We define research taste as the ability to pick interesting problems to solve, design experiments, and interpret experimental results. TasteVal measures the experimental component of research taste; given a fixed research problem, we measure how well a model iteratively designs experiments and draws conclusions from their outcomes. We operationalize experimental research taste as compute efficiency; a Researcher who reaches the same score as an expert human using half the serial experimental compute has twice the experimental taste. Experimental taste thus acts as a multiplier on experimental compute, making it a key input to forecasts of AI progress. TasteVal consists of 8 novel, challenging, open-ended tasks representative of frontier AI R&D. To isolate taste from coding ability, the model under evaluation acts as a Researcher that iteratively designs experiments while a fixed Coder agent implements them and reports their results. The Researcher executes until either the 40 H100 hour or 120 wall-clock hour budgets are exhausted. We recruit 24 human experts, at least 2 per task, and take the best expert attempt per task as the expert baseline. We evaluate 20 models released between 2023 and 2026. The best-performing model, Opus 5.5, exceeds our expert baseline, with a compute multiplier of 2.3x (95% CI 1.15-4.37), at roughly 1/30 of our baseliners' average per-run cost. On TasteVal, the compute multiplier of frontier models has doubled approximately every 3.0 months since December 2025 (95% CI 1.7-5.0), up from every 14 months between 2023 and December 2025. Measured by final normalized performance, frontier models show no trend break, doubling every 14.6 months. To keep TasteVal uncontaminated, we do not release the tasks.

产业坐标与信源核验

本条情报归属 AITOP「模型」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:TasteVal、Opus 5.5、基准测试、AI R&D,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Oliver Jaffe, Dane Sherburn),经 AITOP 多维评分管线结构化提炼。截至目前,全网已有 3 家独立信源对该事件进行交叉印证与跟进报道(含 Teortaxes、Decoder 等),属于近期高关注度行业信号。