HTC² 框架用 Hamiltonian 结构改进物理环境探索
Multi-Horizon Future Discovery for Physics-Informed Exploration
一篇物理控制的探索方法论文，HTC² 用 Hamiltonian 相空间结构加多步时序对比学习算内在奖励，在 DeepMind Control Suite 六个任务上和 ICM、Plan2Explore 这些老方法比了个遍，做 RL 探索的可以看看。
论文提出 Hamiltonian Temporal Contrastive Curiosity（HTC²），把多时间跨度的时序表征学习与 Hamiltonian 相空间结构结合用于强化学习探索。 horizon-conditioned contrastive critic 从同一 episode 内的未来状态对学习时序关系，正对表征间的距离作为内在奖励，并用有界的多步 Hamiltonian 变化调制奖励。HTC² 分别与 SAC 和 PPO 集成，在 DeepMind Control Suite 的 6 个任务上评测，对比 ICM、RIDE、E3B、Plan2Explore 及单步 Hamiltonian curiosity 基线，并附状态覆盖率评估、组件消融和相空间诊断。
Multi-Horizon Future Discovery for Physics-Informed Exploration
Exploration in physical control environments requires agents to identify actions whose consequences reveal previously unexplored aspects of the underlying dynamics. However, many intrinsic motivation methods rely on immediate prediction errors, state novelty, or local physical changes, which may overlook informative outcomes that emerge across extended temporal horizons. We investigate whether learning structured representations of reachable futures can provide a more effective basis for exploration. To this end, we introduce Hamiltonian Temporal Contrastive Curiosity (HTC$^2$), a framework that combines multi-horizon temporal representation learning with Hamiltonian phase-space structure. A horizon-conditioned contrastive critic learns temporal relationships from future-state pairs sampled within the same episode. The distance between positive-pair representations is then used as an intrinsic reward, encouraging the agent to explore futures that remain insufficiently organized in the learned representation. To incorporate information about physical dynamics, we additionally modulate this reward using bounded multi-step Hamiltonian variation, rather than directly rewarding changes in the Hamiltonian itself. We integrate HTC$^2$ with both Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO), and evaluate it across six DeepMind Control Suite tasks. Comparisons with ICM, RIDE, E3B, Plan2Explore, and a single-step Hamiltonian curiosity baseline are complemented by state-coverage evaluations, component ablations, and phase-space diagnostics to characterize the contributions of temporal learning and physical structure to exploratory behavior.