游戏开发自主软件生成基准测试集 GameASG-Bench 发布
GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development
想了解游戏开发中 AI 生成软件的性能？这个新基准测试集 GameASG-Bench 很有用，它有 47 个具体任务和详细的实验数据，能帮你判断不同模型的效果。
我们介绍了 GameASG-Bench 基准测试集，用于评估游戏开发中的自主软件生成（ASG）性能。该基准包含 47 个浏览器原生游戏生成任务，涵盖 12 个主要游戏类型和 2D/3D 交互。实验结果显示，在九个代理堆栈中，最高 L2 检查通过率是 93.2%，但最高严格任务成功率（需要通过所有 L1 和 L2 检查）仅为 55.3%（26/47 个任务）。DeepSeek-V4-Flash 在获得完整工具访问权限和更大的名义回合预算时表现更好。
GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development
Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessarily establish that their interacting components satisfy the specified behavioral requirements. We introduce GameASG-Bench, a benchmark that makes behavioral testability part of the generation task for game development. Our design declares an evaluation interface specification before generation, fixing legal starting scenarios, player-level actions, stable snapshots, rejection behavior, and invariants while leaving private implementations open. Concretely, we include: (i) static L1 checks that assess source-level compliance; and (ii) browser-executed L2 checks that combine semantic observations with real input and runtime evidence. We implement this protocol as 47 browser-native game-generation tasks spanning 12 primary genres and both 2D and 3D interaction, each with executable checks and an independently verified reference implementation. Our experiments answer four key questions about end-to-end agent performance, tool access and nominal turn budget, reasoning effort, and harness choice. Across nine agent stacks, the highest observed mean L2 check pass rate is 93.2%, yet the highest observed strict task success rate, requiring all L1 and applicable L2 prerequisite and core requirement checks, is only 55.3% (26/47 tasks). For DeepSeek-V4-Flash, full tool access and larger nominal turn budgets yield more strict task successes, while the strict task success rate is not monotonic in reasoning effort. Both tested harnesses achieve 18 strict task successes, but only ten tasks succeed under both. These results expose task-level compliance gaps that high average check pass rates actually obscure.
