Critical-State RL：定位多轮工具调用中值得训练的状态
Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use
多轮工具调用失败，不知道该练哪一步？这篇论文给出诊断方法，BFCL v4 缺失函数任务提升 14 个百分点。
Critical-State RL 针对多轮工具调用场景，解决单看奖励变化无法判断该训练哪一次模型调用的问题。当奖励依赖后续交互时，波动可能来自下游随机性而非当前动作差异，该方法用嵌套采样把两类变化分开，再用上下文老虎机训练选中状态。在 Berkeley Function Calling Leaderboard（BFCL）v4 上，缺失函数任务选择工具可用后的响应来训练，提升约 14 个百分点，训练替代状态则持平或更差。缺失参数任务则选择缺失参数补上前的那次响应，该配方还扩展到避免重复调用和记忆管理等任务。
Critical-State RL: Diagnosing Trainable States for Multi-Turn Tool Use
Multi-turn tool-use failures can hinge on a single model call, yet reward variation alone does not reveal which call would benefit from training. When rewards depend on later interactions, their variation can reflect downstream randomness rather than differences between the current actions. We introduce Critical-State RL to identify trainable states in multi-turn interactions. Given task-defined candidate calls and local rewards, the method assesses whether each reward captures the action's effect on task success and whether improvement over a reference policy is possible. It then uses nested sampling to separate action-dependent reward variation from continuation noise and optimizes the policy at the selected states using contextual-bandit training. Experiments on the Berkeley Function Calling Leaderboard (BFCL) v4 compare training at diagnostic-selected states with training at alternative states. For missing-function tasks, the diagnostic selects the response after the tool becomes available; for missing-argument tasks, it selects the response before the missing argument is supplied. Training the selected responses improves performance, including about 14 percentage points on the missing-function task, while training the alternatives leaves performance flat or worse. We further apply the recipe across models and tasks, including logged repeat-call avoidance and memory management.