CAE-Bench 发布:评估 AI 智能体开发可复用 CAE 求解器的能力
CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development
搞 CAE 仿真的工程团队和做智能体评测的研究者应该看看。它专挑求解器泛化能力下手,80 个任务、12 个模型测下来,连 GPT-6 Astra 在流体任务上都翻车,比通用代码基准更能暴露智能体的真实短板。附带了数据格式和 SDK,可以直接接进自己的测试环境。
研究者发布 CAE-Bench,一个用于评估 AI 智能体开发计算机辅助工程(CAE)求解器能力的基准。CAE 求解器不只是 PDE 求解器,还需要在不同边界条件、几何形状和材料属性等输入上保持泛化。基准的设计方式是:同一任务(如传热问题)提供多个用例,智能体只能使用少量公开用例开发求解器,然后在从未见过的保留用例上测试。
实验覆盖 12 个模型、80 个计算任务和多种推理设置。结果显示,AI 开发的求解器即使公开用例上表现良好,在保留用例上也可能得 0 分;在流体动力学任务上,连 GPT-6 Astra 开发的求解器在保留用例上也表现很差。基准附带数据格式和任务注册 SDK,研究者可以在自己的基础设施上运行。
CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development
We developed CAE-Bench, a benchmark to evaluate AI agents on Computer-Aided Engineering (CAE) solver development. A CAE solver is more than a PDE solver; it must be able to generalize across various inputs, such as different boundary conditions, geometries, and material properties. CAE-Bench evaluates the generalizability of AI-developed CAE solvers by providing the same task, such as a heat transfer problem, multiple cases, with each case specifying an input. The AI agent develops a solver for a fixed task using only a few public cases, and the resulting solver is then tested on held-out cases, i.e., cases it has never seen before. We evaluate 12 models across a range of reasoning settings on 80 computational tasks. Our experiments show that even if a solver developed by an AI agent performs well on the public cases, it may even get 0 score on held-out cases. For tasks such as fluid dynamics problems, even the solver developed by GPT-6 Astra performs poorly on held-out cases. A data format and an SDK for task registration are provided, so the researcher can use the benchmark on their own infrastructure. We hope that CAE-Bench will help the community to develop AI agents that can develop reusable CAE solvers that generalize to various inputs.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:CAE-Bench、GPT-6 Astra、智能体、基准测试,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Zhiyu Zhou, Tianyou Liu, Xinping Yang, Yiyao Liu, Jiayi Dong, Yihan Lin, Haokai Ye, Yifan Lu, Zeyu Dong, Yansong Li, Shengqi Zhang),经 AITOP 多维评分管线结构化提炼。截至目前,全网已有 6 家独立信源对该事件进行交叉印证与跟进报道(含 IT之家、OpenAI、HIT Consultant 等),属于近期高关注度行业信号。