可验证产物研究:要求 LLM 输出可执行程序使 ARC-AGI-2 得分从 93 降至 68
The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back
做 LLM 推理评测或 agent 验证的人该看看这篇:它实测了强制输出可执行代码在 ARC-AGI-2 上到底损失多少,还给出不花额外推理预算的进化搜索补救法。相比纯 best-of-n 采样,省预算还提分,评测设计者可以直接借鉴。
一项 arXiv 论文在 ARC-AGI-2 的 120 个任务上量化了要求 LLM 产出可验证产物(可执行程序)的代价:得分从 93/120(77%)降到 68/120(57%),损失 25 个任务。
深入分析发现其中 21 个任务在任何预算、采样或搜索方式下都没有程序配置能解出;而 28 个仅答案模式可解的任务里,27 个对应的程序能复现全部训练示例却在测试上失败。
结论是瓶颈在于拿到能泛化的解释,而不是找到候选解释。机制层面,论文提出零额外模型开销的改进:对已采样的程序做短程进化搜索,在 2 个程序上比 best-of-2 采样代码多解 9.
2±1.4 个任务;
同时程序可与训练示例核对,允许提前终止——同一停止规则下 best-of-n 最多节省 38% 预算,进化方案最多 49%,答案-only 基线则无法提前停止。论文还把同一方法论用于量子纠错的译码器合成,用已发表译码器的认证面板在留出考试样本上评估可解释译码器。
The Price of an Explanation: What Demanding Verifiable Artifacts Costs, and What It Buys Back
Generative models produce many candidate solutions but provide no intrinsic signal by which to rank them. One mitigation is to demand a \emph{verifiable artifact}: a program that can be executed against the available evidence. A common concern is that requiring executable verification incurs costs in both solution quality and computational budget. We measure these two costs on the $120$-task ARC-AGI-2 evaluation set. Demanding a verifiable artifact from an LLM reduced the score from $93$ to $68$ of $120$ tasks ($77\%$ to $57\%$). Of those $25$ tasks, $21$ were reached by no program-producing configuration we ran, at any budget, by sampling or by search; yet for $27$ of the $28$ tasks that only the answer-only arm solved, some program arm held a program that reproduced every demonstration and still failed the test. The bottleneck is obtaining an explanation that \emph{generalizes}, not finding one. A short evolutionary search over two already-sampled programs, at no additional model cost, beats best-of-$2$ sampled code by $9.2 \pm 1.4$ tasks, and over a single sampled program beats $1$-shot sampling by $9.9 \pm 1.3$. Moreover, because a program can be checked against the training examples, any program-producing protocol can terminate early once a task is explained: under the same stopping rule, best-of-$n$ sampling declines up to $38\%$ of its budget and the evolved arm up to $49\%$, whereas an answer-only baseline cannot terminate early. Applying the same methodology to decoder synthesis for quantum error correction yields interpretable decoders evaluated against a certified panel of published decoders on held-out exam shots.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:ARC-AGI-2、可验证性、推理模型、LLM评测,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Edgar A. Duéñez-Guzmán, Eduardo Dueñez, Edgar Said Hernandez Sanchez, Suzanne Sadedin),经 AITOP 多维评分管线结构化提炼。