Stanford 与 Google DeepMind 提出 TURTLE：把太简单的工具调用基准自动改造成难题
模型刷分刷太容易？这篇 Stanford 和 DeepMind 的论文教你用 AlphaEvolve 把老任务自动加难度，735 个任务全落在 40%-60% 通过率，不用重写基准。
Stanford 和 Google DeepMind 的论文提出 TURTLE 方法，用 AlphaEvolve 对 AppWorld 等静态工具调用基准中的现有任务做变异，加入过滤器、额外步骤和隐藏查询。每个进化后的任务用求解模型跑 10 次，成功率约 50% 时难度最优，且必须至少被解出一次。在 Gemma 4 31B-it 上，原 AppWorld 任务有 69% 总能被解出，而 735 个进化后任务几乎全部落在 40% 到 60% 完成率区间；LLM 评审判定 52.5% 的进化任务是难题，原任务这一比例只有 19.8%。
New Stanford and Google DeepMind paper shows that you can turn too-easy tool-use benchmarks into hard, still-solvable tests by evolving existing tasks against the model.
Tool-use benchmarks like AppWorld are static, so models soon solve most tasks, and building new ones takes human work.
TURTLE uses AlphaEvolve to mutate existing tasks, adding filters, extra steps and hidden lookups, then runs the solver model 10 times on each. Tasks score highest when the solver succeeds about 50% of the time, and each must be solved at least once.
69% of original AppWorld tasks were always solved by Gemma 4 31B-it, while nearly all 735 evolved tasks sat at 40% to 60% completion. An LLM judge rated 52.5% of evolved tasks hard, versus 19.8% of the originals.
If your model aces a benchmark, evolve its tasks toward a 50% success rate instead of writing a new benchmark from scratch.
