Routed Ensembles 与轨迹归因：多目标强化学习新算法
Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution
训练模型同时兼顾多个目标时总顾此失彼？这篇论文给了可量化的冲突度量加两个能落地的算法，Meta-World 上实测涨了 4.3%。
这篇论文研究多目标强化学习中相互冲突目标（如 helpfulness 与 honesty）的权衡问题。作者提出 objective-conflict coefficients 来量化目标冲突程度，并推导出在线设置下接近最优的 regret bounds。算法层面设计了 routed ensemble policy learning，维护少量基础策略并按潜在奖励动态分配轨迹，在 Meta-World 和 MO-Gymnasium 在线环境中平均性能比基线提升 4.3%。另提出 kernel surrogate trajectory attribution 估计轨迹间反馈信号的非线性协同与拮抗作用，与 leave-one-out 分析的相关性比现有基线高 50%，用于轨迹选择时奖励比基线归因方法提升 27%。
Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution
Multi-objective reinforcement learning is a fundamental problem for modern AI, with applications spanning post-training, reasoning models, and robotic control. A central challenge is to learn policies that balance competing objectives, such as helpfulness versus honesty in safety and alignment, or speed versus energy in robotics. The optimal policy can vary sharply with objective-weighting combinations, and the extent of this variation depends crucially on the degree of conflict among objectives, making per-preference policy learning sample-inefficient. In this paper, we introduce objective-conflict coefficients and derive near-optimal regret bounds for online settings using this measure, thereby quantifying the impact of misalignment on learning efficiency. Then, we design routed ensemble policy learning, an algorithm that maintains a small set of basis policies and dynamically routes trajectories to the policy with the highest potential reward. Finally, we consider trajectory selection as another approach to mitigate objective conflict, and design a kernel surrogate trajectory attribution algorithm that estimates how feedback signals from a set of trajectories influence policy learning on other trajectories. The kernel can capture nonlinear interactions such as synergy and antagonism between trajectories. We evaluate our approach across diverse environments and benchmarks: routed ensembles improve average performance by 4.3% over baselines for online environments from Meta-World and MO-Gymnasium, and kernel surrogate attribution improves correlation with leave-one-out analysis over existing baselines by 50%. When applied to trajectory selection, kernel-based attribution yields a 27% improvement in reward over baseline attribution methods.