密集奖励优于结果奖励：临床多轮智能体强化学习研究
Safe Reinforcement Learning for Multi-Turn Clinical Agents: When Dense Rewards Beat Outcome Rewards
做医疗或客服这类多轮智能体训练的人必看。论文用三个临床环境实测发现结果奖励训练会让模型走捷径，高风险病例崩掉，而给提问环节加过程奖励能救回来。想避免智能体“偷懒式”收敛的团队可以直接参考它的实验设置。
arXiv 论文（arXiv:2611.12728）针对临床多轮智能体的强化学习训练提出新发现：仅用可验证结果奖励（outcome-only GRPO）训练时，模型在三个受控环境——药物续方分诊、急性诊断检查、剂量滴定——中虽然总奖励提升，但会出现过早决策、只问一个问题、罕见高风险病例失败等问题。论文设计了带隐藏临床状态、查询成本和确定性结果验证器的实验环境，并引入精确自适应 oracle 做基准对比。
在诊断任务中，对降低不确定性的提问赋予密集过程奖励，相比结果奖励训练在三个训练种子上都提升了 held-out 表现和中难度病例覆盖率，且避免了结果奖励训练中出现的高风险病例崩溃。将探索与利用分离的机制进一步提高高风险病例表现，但代价是在低急症病例上过度采集信息。
实验还表明过程奖励的最优强度因任务而异；监督预热已学会谨慎提问后，强化学习靠删减问题提升总奖励却损害高风险表现；强化学习无法可靠修复带偏见的监督数据遗留的罕见行为缺口。结论是可验证的最终结果不足以训练可靠多轮智能体，奖励设计必须考虑达到答案的过程，评估需区分自适应信息采集与钻奖励漏洞的捷径。
Safe Reinforcement Learning for Multi-Turn Clinical Agents: When Dense Rewards Beat Outcome Rewards
Reinforcement learning with verifiable rewards is increasingly used to improve language model reasoning, yet its effects on sequential decision procedures remain poorly understood. We study this problem in clinical multi-turn agents that must decide what information to request, when to stop gathering evidence, and which action to take. We introduce three controlled environments: medication refill triage, acute diagnostic work-up, and dosing titration, each with hidden clinical states, query costs, exact adaptive oracles, and deterministic outcome verifiers. Across these tasks, outcome-only group relative policy optimization often improves aggregate reward while producing undesirable procedures, including premature commitment, one-question collapse, and failures on rare high-risk cases. In diagnostics, assigning dense credit to questions that reduce uncertainty improves held-out value and moderate-case support across three training seeds relative to outcome-only reinforcement learning, without the high-risk seed collapse observed under outcome-only training. Separating exploration from exploitation further increases high-risk support at its best checkpoint, but at the cost of excessive information gathering on low-acuity cases. Additional experiments show that the optimal strength of process credit is task-dependent; that after a supervised warmup that already gathers carefully, reinforcement learning raises aggregate reward by trimming questions while degrading high-risk performance; and that reinforcement learning does not reliably repair rare behavior gaps inherited from biased supervised data. Verifiable final outcomes are therefore insufficient for training reliable multi-turn agents: reward design must account for the procedure used to reach an answer, and evaluation must distinguish adaptive information gathering from reward-exploiting shortcuts.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：强化学习、临床智能体、GRPO、奖励设计，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Brendan P. Murphy, Agustina Saenz），经 AITOP 多维评分管线结构化提炼。