APB 基准发布：测试 LLM 多轮主动提问式解谜推理能力
Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving
这篇论文造了个新基准，让模型靠多轮提问猜出 960 个专业概念，结果开源推理模型反而输给普通指令模型，挺有意思。
arXiv 论文提出 Active Probing Benchmark（APB），将谜题求解扩展为交互式推理任务。模型需通过多轮主动提问，在 8 个学术领域中定位 960 个目标概念。评测发现一个反直觉现象：多数开源推理模型的成绩明显低于对应的指令模型，说明它们在交互场景中难以用程序化推理调用领域知识。论文同时提供了可扩展的基准生成流水线。
Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving
Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require multi-step decisions to reach a solution. However, existing puzzle benchmarks typically focus on narrow environments with rigid mechanics and tightly constrained solution targets. To bridge this gap, we introduce the Active Probing Benchmark (APB), which generalizes puzzle-solving by casting concept identification as an interactive reasoning task that integrates specialized knowledge assessment into the puzzle setting. Specifically, APB evaluates whether models can acquire information via multi-turn active probing and inference to pinpoint 960 target concepts of specialized difficulty across eight diverse academic domains. Our evaluation reveals a counter-intuitive phenomenon: most open-source reasoning models significantly underperform their standard instruction counterparts. This finding suggests that in interactive settings, these models struggle to leverage procedural reasoning to reliably access and apply specialized domain knowledge. Beyond these empirical insights, APB provides a scalable pipeline for generating benchmarks that probe knowledge reasoning under interaction across diverse settings.
