论文

Dynamics-Aligned Representation Learning:用随机遮蔽改进基于模型的强化学习

Dynamics-Aligned Representation Learning under Induced Partial Observability

精选理由

一篇 MBRL 论文,做法是在训练时随机遮挡观测,逼潜空间模型学到真正的动态而非依赖画面重建,在 DeepMind Control Suite 上验证有效。

arXiv 论文提出一种随机门控机制,通过从 Beta-Bernoulli 随机场采样空间二值掩码,在训练时对原始观测施加可控的部分可观测性。门控概率在每个时间步重新采样,编码器由此经历从近乎完全遮挡到近乎完全可见的条件。方法以通道拼接方式集成,对现有 dynamics 模型的架构改动极小。在 DeepMind Control Suite 的像素级连续控制任务上,配合仅依赖开环 rollout 的模型预测控制评估,随机门控提升了规划性能和数据效率。分析显示训练时均衡的可观测性配置优于偏稀疏或偏密集的设置。

原文 · arXiv: Google DeepMind

Dynamics-Aligned Representation Learning under Induced Partial Observability

Model-based reinforcement learning (MBRL) has enabled sample-efficient learning of continuous control policies from high-dimensional observations through learned latent dynamics models. These models are typically trained with observation reconstruction objectives, which provide strong supervision but can bias representations toward visual fidelity leading to an undertrained prior and unreliable long-horizon predictions. We propose a stochastic gating mechanism that induces controlled partial observability during training by sampling a spatial binary mask from a Beta--Bernoulli random field and applying it to the raw observation. The gate probability is resampled at each time step, exposing the encoder to conditions ranging from near-total occlusion to near-full visibility. Integration of our method requires minimal architectural changes from existing dynamics models: the gated observation and the binary mask are channel-wise concatenated and passed to the encoder as a single augmented input, with no modifications to the broader training pipeline. We evaluate the proposed approach on a set of pixel-based continuous control tasks selected from the DeepMind Control Suite. Using model predictive control for evaluation, which relies solely on open-loop model rollouts, we show that stochastic gating improves planning performance and data efficiency across tasks. Further analysis demonstrates that performance is robust to the spatial scale of the mask, and a balanced observability during training outperforms configurations biased toward either sparser or denser observations.