SeTPO 训练方法让小语言模型更擅长调用外部工具
Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints
小模型学用工具的老难题有了新解法：SeTPO 用双向 KL 同时管住探索和模仿，BFCL v4 上跑赢了蒸馏和纯 RL，做端侧模型的可以看看。
arXiv 论文提出 Selective Teacher guided Policy Optimization（SeTPO），用于提升小语言模型（SLM）的工具调用能力。方法上，SeTPO 在低奖励轨迹上引入反向 KL 的 token 级教师反馈，并用前向 KL 持续保留教师示范行为。论文在 BFCL v4 和 API Bank 两个基准上验证，SeTPO 在三种学生模型规模上均超过 SFT、强化学习和蒸馏基线。理论上，作者证明最优策略的期望奖励不低于教师模型，同时保留教师一定比例的概率质量。
Teacher-Guided Tool Learning for Small Language Models via Bidirectional KL Constraints
External tools can extend the capabilities of small language models (SLMs), enabling their deployment in resource constrained environments. However, their limited capabilities make it difficult for SLMs to discover effective tool use behaviors through reinforcement learning alone, while direct teacher supervision suffers from teacher student behavioral distribution mismatch and may propagate erroneous preferences. To address these challenges, we propose Selective Teacher guided Policy Optimization (SeTPO), which coordinates student exploration with selective teacher guidance and persistent demonstration supervision. Specifically, SeTPO incorporates reverse KL derived token level teacher feedback into policy advantages on low reward trajectories, while auxiliary forward KL supervision preserves teacher demonstrated behaviors throughout training. Under an idealized bidirectional KL objective, we theoretically establish that the optimal policy achieves expected reward no lower than the teacher's while retaining a guaranteed fraction of the teacher's probability mass. Extensive experiments on BFCL v4 and API Bank demonstrate that SeTPO achieves superior overall performance over supervised fine tuning, reinforcement learning, and distillation baselines across all three student scales.
