研究区分目标错误泛化的三种失败模式：目标丢失在不同深度
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
智能体跑得挺顺却追错了目标？这篇论文把这种失败拆成三种，还给出哪种干预能救回多少到达率的数据。
一篇 arXiv 论文在 3 个 Procgen 环境和 5 个机器人任务上研究目标错误泛化（goal misgeneralization），即智能体在分布外表现正常却不追求预期目标的现象。论文区分出三种策略-目标错位：功能目标替代、策略捷径、预期目标线索与代理目标线索之间的竞争。通过追踪目标信息在内部保留的位置、对策略的影响强度及在动作选择中的作用，研究发现目标在不同深度丢失。在线索竞争情形下，steering 把预期目标到达率从 16.0% 提升到 95.3%；在动作捷径情形下，只有重接下游线路能把到达率从 31.6% 恢复到 84.3%；目标完全惰性时两种方法都无效。这说明目标错误泛化不是单一的目标表征失败，而是发生在不同深度的策略对齐失败，且决定了哪种恢复手段有效。
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents. Across three Procgen environments and five robot tasks, we find three distinct forms of policy-goal misalignment: (1) functional target substitution, where behavior follows an alternative goal, (2) a policy shortcut, where behavior follows a learned action strategy, and (3) cue competition between intended and proxy goal cues. We distinguish these regimes by tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, and find that the goal is lost at different depths. Where the goal reaches the decision, as in cue competition, steering redirects the policy to the goal's location, raising intended-goal reach from 16.0% to 95.3%. Where it reaches the policy but not the decision, only new downstream wiring recovers goal reach, from 31.6% to 84.3% in the environment with the action shortcut. Where it is inert, neither works. Our results show that goal misgeneralization is not a single failure of goal representation, but distinct failures of policy alignment at different depths, tightly connected to which recovery works.
