CAST结合规划器引导行为提升强化学习
CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning
新提出的CAST方法结合规划器与策略，在强化学习中实现更高效的值学习，并在四足机器人上验证效果。
CAST方法使用规划器引导行为改进值学习，同时用当前策略正则化值估计。该方法将动作值批评家替换为状态值批评家，训练目标结合真实规划器引导转移和当前策略下的想象转移。研究团队在DeepMind Control和HumanoidBench套件上评估CAST，并与多种最先进方法对比。CAST成功应用于执行动态手倒立的Unitree Go2四足机器人。
CAST: Alternating State-Value Targets and Expanded Policy Gradients for Model-Based Reinforcement Learning
Model-based reinforcement learning (MBRL) is a family of RL methods that learn a model of the environment and use it for action selection, making it well suited to robotics due to its sample efficiency. Combining learned models with online planning can further improve action selection, as the planner can exploit the model to find better actions than the learned policy alone. Recent methods combining learned policies with online planning typically learn the value of the policy rather than the stronger planner-guided behavior. We present CAST (Critic with Alternating State-value Target), which uses planner-guided behavior to improve value learning while regularizing the value estimate with the current policy. CAST replaces the action-value critic with a state-value critic, trained using a target that combines a real planner-guided transition and an imagined transition under the current policy. The resulting value function corresponds to an alternating process between planner-guided behavior and the current policy, allowing it to benefit from the stronger planner behavior while being regularised by the policy being learned. We evaluate CAST on the DeepMind Control and HumanoidBench Suites against several state-of-the-art methods, and demonstrate successful transfer to a physical Unitree Go2 quadruped performing a dynamic handstand.