Targeted Active Preference Learning：让机器人主动提问补齐奖励学习的盲区
Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
机器人训练不再让人类重头再教，而是机器自己定向提问并解释该看哪，低查询量下效果提升三成。
这篇 arXiv 论文针对奖励学习中人类演示覆盖不全的问题，把补问负担从人类转移到机器人一侧。方法包含两个部分：一个 informed prior 将偏好查询定向到监督不足的任务特征，并为每个查询附带解释，引导人类注意相关特征。仿真实验中，该方法在低查询量场景下比标准 active preference learning 基线高出最多 30.8%。用户研究显示 informed prior 与解释各自显著提升奖励恢复效果，且二者相互增强。
Asking Smarter Questions: Targeted Active Preference Learning for Underspecified Rewards
Reward learning typically assumes the human's input covers everything that matters for the task. In practice, humans can only attend to so much at once, so their teaching naturally captures some features---or aspects of the task---better than others. Prior work has shown how the robot can identify which features were poorly supervised directly from its teaching data. But the current solution is to ask the human to try again, putting the burden on someone who already struggled to convey those features. We propose shifting this burden to the robot. Our key insight is that the same information that tells the robot what it doesn't know can help it ask better questions. Preference queries offer a low-effort channel for resolving this uncertainty, but only if the robot knows where to direct them and the human knows what to attend to when answering. We act on this insight in two ways: we build an informed prior that targets preference queries at the features that need resolving, and we accompany each query with an explanation that focuses the human's attention on exactly those features. In simulation, the informed prior beats the standard active preference learning baseline by up to $30.8\%$ in the low-query regime, where reducing human effort matters most. In a user study, the informed prior and explanations each significantly improve reward recovery and amplify each other.
