LUCA 发布：一个可评估实验推理过程的合成世界
LUCA: A Synthetic World for Experimental Reasoning
这篇论文做了个叫 LUCA 的沙盒，专门测智能体是靠正确推理还是蒙对答案，7 个前沿模型测下来诊断能力差了 7 倍，挺有意思的。
arXiv 论文 2611.03100 提出 LUCA，一个在完全已知真值下评估智能体实验过程的可配置合成环境。智能体通过预算化实验学习未知调控系统，再完成预测未见干预、诊断隐藏故障和执行抢救三类任务。测试覆盖 7 个前沿 LLM 和 3 个基线，结果显示各能力差异明显：诊断能力跨模型波动达 7 倍，预测仅 1.4 倍。一个典型失败模式是智能体做了能区分竞争原因的实验，却没能利用所得证据。
LUCA: A Synthetic World for Experimental Reasoning
An experimental agent can succeed for the wrong reasons. Evaluations, however, often focus on the outcome, not the experiments, evidence, and inferences that produced it. We introduce LUCA, a configurable synthetic world for evaluating this process under fully known ground truth. Agents learn unknown regulatory systems through budgeted experimentation, then predict unseen interventions, diagnose hidden faults, and attempt rescues. This setup allows us to evaluate experiment selection, prediction, diagnosis, and intervention separately. Across seven frontier LLMs and three baselines, these capabilities dissociate sharply. Agents frequently rescue malfunctioning systems while diagnosing the wrong fault; diagnosis varies 7x across models while prediction varies only 1.4x. Models also adopt surprisingly different experimental strategies, and greater exploration does not reliably yield better performance. In one failure mode, agents often perform the experiment needed to distinguish competing causes, but still fail to use the resulting evidence. LUCA turns these hidden failures into measurable behaviours, providing a controlled environment to study where experimental agents succeed and fail. Code: https://github.com/RelationRx/luca.