论文精选

MINT:用相遇时间指示器替代截断系数,改进多步 off-policy 强化学习

MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

精选理由

做强化学习的朋友可以看看这篇:MINT 把多步 TD 里最难调的 trace 长度超参数换成了自动的停止指示器,MuJoCo 和 Atari 100k 上跑赢基线,还不用逐环境调参,写 RL 代码或调 R2D2 类算法的人会感兴趣。

arXiv 论文 2611.24413 提出 MINT(Meeting-time INdicators for Truncation),针对多步 off-policy 时序差分方法中逐步 trace 系数乘积导致的方差控制与长时序信号的权衡问题。传统方法里最大 trace 长度是敏感超参数,MINT 用一个二元停止指示器替代:在耦合的 behavior 与 target 轨迹首次相遇前取 1,之后取 0。

正文

在真实价值函数处,该指示器的幂等性给出由 meeting-time survival 决定的方差界。实现上只需一次采样动作即可为停止规则提供 model-free 替代,无需转移模型或 mixing-time 估计,论文还分别推导了状态耦合算子及其替代算子在固定策略评估下的收敛条件。

实验覆盖 MuJoCo、DeepMind Control、Atari 100k、RWRL 和 OGBench,MINT 超过已报告的多步基线,其中 MuJoCo 主对比使用统一 cap,不需要逐任务调 cap 长度;备份时域自适应调整,相比长固定时域备份降低了 target 波动。

原文 · arXiv: Google DeepMind

MINT: Meeting-time Indicators for Truncation in Multi-Step Off-Policy RL

Many multi-step off-policy temporal-difference methods handle policy mismatch through products of per-step trace coefficients, leading to a trade-off between variance control and long-horizon signal. The maximum trace length consequently becomes a sensitive hyperparameter. We propose MINT (Meeting-time INdicators for Truncation), which replaces these products with a binary stopping indicator: one until coupled behavior and target trajectories first meet, zero thereafter. At the true value function, the indicator's idempotence yields a variance bound governed by meeting-time survival. For practical implementation, a single sampled action provides a model-free surrogate for the stopping rule, requiring no transition model or mixing-time estimate. We derive separate sufficient conditions for contraction of the state-coupled operator and its practical surrogate under fixed-policy evaluation. On MuJoCo and DeepMind Control tasks, MINT outperforms the reported multi-step baselines; the main MuJoCo comparison uses a common cap without per-task cap-length tuning. We also demonstrate the effectiveness of MINT on Atari 100k, RWRL, and OGBench experiments; MINT adapts the backup horizon to reduce target variability relative to long fixed-horizon backups across tasks and dynamics.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:off-policy RL、时序差分、MuJoCo、Atari 100k,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始信息发布自一手权威信源「arXiv: Google DeepMind」(Tier-1 一手发布),具备高确定性参考价值。