论文

Openhagen Worlds:用于衡量与训练科学发现智能体的校准环境套件

Openhagen Worlds: Calibrated Environments for Measuring and Training Scientific Discovery Agents

精选理由

搞 agent 评测的可以看看这套环境,它把"会做实验"和"靠记忆"分开测,每个难度都经过 oracle 和捷径策略双重校准。DeepSeek V4.1 Flash 在最难的八阶段任务上离 oracle 还差 22 个,说明科学发现 agent 还有很大空间,做 RL 训练或评测的人值得拿来跑基线。

Openhagen Worlds 是一套包含 27 个程序化生成环境的科学发现智能体评测与训练套件,覆盖从经典实验推理谜题到基于模拟器的物理与化学场景。

正文

智能体在预算内运行实验并提交答案,由确定性验证器对照隐藏真相打分;每条观测都完整声明评分规则,以避免隐含规则干扰。

全部 178 个难度级别均经过校准:只有当 64 个开发实例上脚本化 oracle 成功率至少 0.75、最强非演绎策略至多 0.

25、各捷径策略至多 0.35 时才准入。

92 个核心级别覆盖全部 27 个环境族,规模适合训练小模型;86 个前沿级别在 15 个发现世界上扩展出三、五、八阶段以上的多轮实验任务。

在 84 个保留前沿实例上,DeepSeek V4.1 Flash 在默认推理力度(每次调用 16,384 tokens)下解决 32 个,最大力度(最高 393,216 tokens)下解决 57 个,oracle 为 79 个,八阶段任务的剩余提升空间最大。

对 9B 模型做 on-policy expert iteration 后,训练环境族的核心层级成功率从 0.090 升至 0.

148,GPQA Diamond 无明显下降。核心层级的保留实例以托管服务形式开放。

原文 · arXiv: DeepSeek

Openhagen Worlds: Calibrated Environments for Measuring and Training Scientific Discovery Agents

Agents that do science must choose informative experiments and reason from their results, but most discovery benchmarks cannot separate that skill from recall, unstated scoring rules or shortcut policies. We introduce Openhagen Worlds, a suite of 27 procedurally generated environments (some with finite truth spaces), from classic puzzles of experimental reasoning to simulator-grounded physics and chemistry, in which an agent runs experiments under a budget and commits an answer that a deterministic verifier checks against a hidden truth. Every observation states the complete scoring rule, and each of the 178 difficulty levels is calibrated: it is admitted only if, on 64 development instances, a scripted oracle succeeds on at least 0.75, the strongest plausible non-deductive policy on at most 0.25 and each gated shortcut on at most 0.35. The 92 core levels span all 27 families and are sized to train small models; the 86 frontier rungs extend the 15 discovery worlds with harder questions and with campaigns of three, five and eight or more committed stages on one hidden system. On the same 84 withheld frontier instances, one per family and rung, DeepSeek V4.1 Flash solves 32 at its default reasoning effort with 16,384 tokens per call and 57 at its maximum effort with up to 393,216 tokens per call (oracle: 79); the headroom that remains is largest in eight-stage campaigns. On-policy expert iteration gives a 9B model modest gains: its core-tier success on trained families rises from 0.090 to 0.148, partly through better action format, with no detectable loss on GPQA Diamond. A hosted service plays the core tier's withheld instances.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Openhagen Worlds、DeepSeek、智能体、基准测试,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始信息发布自一手权威信源「arXiv: DeepSeek」(Tier-1 一手发布),具备高确定性参考价值。