论文多源确认

SciResearch-Eval:用带缺陷模拟器评测科研智能体的实验设计能力

SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

精选理由

这个基准专挑模拟器物理缺陷这类科研智能体最怕的坑,比 MLE-bench 更考察实验设计判断力。GPT-6 Astra 和 Claude Fable 5.2 靠设计聪明实验保持校准,Claude 5.5 则要烧 13-31 倍推理才追上。做 agent 评测或科研自动化的人值得跑一下开源的 harness。

arXiv 发布 SciResearch-Eval,一个交互式基准:智能体自主设计实验,用可能欠分辨或物理错误的模拟器测量材料热学性质,并以 95% 置信区间报告结果。评测在 93 个任务、9 个模型、2,511 个 episode 的 held-out 集上进行。

正文

结果显示模型能处理程序性陷阱,但当模拟器缺失物理机制时,多数模型报告错误数值却声称 95% 置信:区间覆盖率仅 20-62%,而 GPT-6 Astra 达 94%、Claude Fable 5.2 达 99%。默认推理强度下 Claude Opus 仅在 35% 的 episode 中诊断出缺失物理(Astra 为 66%),Astra 和 Fable 则会设计让缺陷模拟器仍然准确的实验(如缓慢加热),从而保持校准。

要求更宽的区间没有用;明确陈述策略可使区间分数提升 1.6-2.5 倍。预注册的消融实验显示 Claude 5.5 的纠错能力是潜在的:最大推理强度下 Opus 提升逾十倍,但需消耗 13-31 倍推理量;老模型和开源模型则无改善。Fable 5.2 默认区间分数比 Astra 好 1.9 倍,但 token 消耗是其 6 倍。环境、评分器和测试框架已开源。

原文 · arXiv cs.LG

SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experimental Agents

We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, against 94% for GPT-6 Astra and 99% for Claude Fable 5.2. At default effort the failing Claude 5.5 models rarely diagnose the missing physics (Opus in 35% of episodes, against 66% for Astra), and even when they do, their ranges often miss; Astra and Fable design experiments in which the flawed simulator is accurate, such as gentle heating, which keeps Astra calibrated even when it does not name the cause. Asking for wider intervals does not help; stating the strategy improves interval scores 1.6-2.5-fold, replicated on a pre-registered fresh split, but coverage barely moves. A pre-registered effort ablation shows that the correction is latent in the Claude 5.5 models: at maximum effort Opus improves more than tenfold and is no longer detectably behind Astra at default, but only with 13-31 times more reasoning; older and open-weight models gain nothing. Fable 5.2 is the most accurate model at default effort (interval score 1.9 times better than Astra's) but writes 6 times as many tokens, and at Astra's token count it falls short. We release the environment, graders and harness.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:SciResearch-Eval、Claude、GPT-6、智能体,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora),经 AITOP 多维评分管线结构化提炼。截至目前,全网已有 6 家独立信源对该事件进行交叉印证与跟进报道(含 IT之家、kimmonismus、Andrew Curran 等),属于近期高关注度行业信号。