RetroChessBench 基准发布：GPT 6 Astra 在国际象棋回溯推理中大面积失败
RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories
GPT 6 Astra 做数学题很厉害，但给它回溯国际象棋题，24 次只对 2 次，这个新基准专门测模型的纯推理能力，失败原因分析很有意思。
研究者推出 RetroChessBench，一个包含 12 项任务的回溯国际象棋基准，要求模型重建合法棋史并区分必然与可能事件。在每任务两次试验中，GPT 6 Astra High 无工具时 24 次尝试中仅 2 次给出正确最终答案，使用代码执行后也只改善到 6/24。终端 RL 奖励均值分别为 14.75% 和 27.19%。失败案例显示模型会拒绝合法棋史、把临时阻碍当作永久阻碍，研究者推测是网络训练数据缺乏此类详细推导文本所致。
RetroChessBench: A Benchmark for Reconstructing and Proving Chess Histories
Frontier language models such as GPT 6 Astra have contributed to research mathematics, demonstrating substantial reasoning capabilities. Yet serious reasoning failures persist even in tasks governed by explicit rules. We demonstrate this in retrograde chess which is a playground for pure reasoning, where solvers reconstruct legal histories and distinguish possible events from forced ones. We introduce RetroChessBench, a 12-task benchmark covering exact-length constructions, unique endings, and exhaustive answer sets, with references, rubrics, and a terminal-reward environment. In two trials per task and condition, GPT 6 Astra High fails to produce the required final answer in 22/24 attempts without tools and 18/24 with code execution; mean terminal RL rewards are 14.75% and 27.19%, respectively. The examples identify faulty deductions: failed responses reject legal histories and treat temporary obstructions as permanent. Online retrograde archives often provide puzzles and final move sequences without detailed derivations. We hypothesize that limited exposure to these missing explanations in web-derived training data contributes to the observed reasoning failures. This questions how far frontier mathematical reasoning really generalizes beyond familiar solution patterns. Claims of general intelligence require evidence that models can transfer learnt principles to areas with few worked examples.