ARCHER:用动作轨迹强化学习让小型模型学会可执行推理
ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models
做小模型数学推理的值得看这篇:ARCHER 把思维链编译成带计算器调用的动作轨迹,再用分层 RL 打磨策略。在 MATH-500、AIME-2024/2025 上比工具增强基线高出 5-9 个点,7B 推理模型也能再涨,本地部署团队可以直接照着复现。
论文提出 ARCHER 框架,用两阶段方法训练小型语言模型(SLM)生成可执行、可审计的数学推理轨迹。第一阶段把自由格式的思维链编译为类型化动作轨迹,包含推理步骤、计算器调用、运行时观察、验证动作和最终答案,把监督目标从模仿解释性文本转为学习工具接地的推理策略。
第二阶段通过非对称分层强化学习进一步优化该策略,奖励信号覆盖答案正确性、工具接地、过程纪律和轨迹对齐四项条件门控指标。在五个 sub-10B 主干上,ARCHER 相比最强的工具增强基线在基准准确率宏平均上提升 5.51 个百分点,其中 MATH-500 提升 8.95 点,AIME-2024 和 AIME-2025 分别提升 5.33 和 5.34 点。
论文还在两个额外的推理特化 7B 主干上验证:即使经过推理向强化学习或蒸馏,可执行接地仍然带来收益。对需要在本地低成本、隐私敏感场景部署数学推理能力的小模型开发者,这套训练范式提供了可复用的思路。
ARCHER: From Rationale Imitation to Executable Reasoning in Small Language Models
Small language models are attractive for local, low-cost, and privacy-sensitive deployment, but they remain unreliable on mathematical reasoning tasks that require decomposition, exact computation, tool use, and verification. We introduce \textbf{ARCHER}: \textbf{A}ction-trace \textbf{R}einforcement for \textbf{C}alculator-grounded \textbf{H}ierarchical \textbf{E}xecutable \textbf{R}easoning, a two-stage framework for training small language models (SLMs) to produce executable and auditable reasoning trajectories. In the first stage, ARCHER compiles free-form chain-of-thought rationales into typed action traces containing reasoning steps, calculator calls, deterministic runtime observations, verification actions, and final answers. This shifts the supervision target from imitating explanatory text to learning a tool-grounded reasoning policy. In the second stage, ARCHER refines this policy through asymmetric hierarchical reinforcement learning, using grouped and conditionally gated trajectory rewards for answer correctness, tool grounding, process discipline, and trace alignment. Across five standard sub-10B backbones, ARCHER improves over the strongest tool-augmented baseline by 5.51 percentage points on macro-average across benchmark accuracies, including gains of 8.95 points on MATH-500 and 5.33 and 5.34 points on AIME-2024 and AIME-2025. Evaluations on two additional reasoning-specialized 7B backbones further show that executable grounding remains beneficial after reasoning-focused reinforcement learning or distillation. These results demonstrate that executable supervision and reward-guided behavioral refinement provide a general and effective training paradigm for compact mathematical reasoners.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:ARCHER、SLM、强化学习、MATH-500,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Dhruv Dixit, Khushi Patel, Congcong Xu, Min Song),经 AITOP 多维评分管线结构化提炼。