Permanent与Transient表征分离方法用于持续强化学习
Permanent and Transient Representations for Continual Reinforcement Learning
Mila团队的新论文，把持续强化学习的长期记忆和短期适应拆成两套表征，在Craftax-Classic上跑赢了基线，做CRL的可以看看。
论文针对持续强化学习（CRL）中的稳定性-可塑性权衡，提出将长期学习与短期适应拆分到两套系统。相比Anand & Precup (2023)两套价值函数共用同一特征的方案，该方法为transient价值函数引入非参数近似，让长期系统使用参数化表征、短期系统使用非参数表征。实验在图像任务和Craftax-Classic上在线学习并超过竞争基线。
Permanent and Transient Representations for Continual Reinforcement Learning
Continual Reinforcement Learning (CRL) agents struggle to adapt to new situations while retaining prior knowledge, leading to a stability-plasticity trade-off. One approach to solving this problem is by using complementary learning systems: one for slow, long-term learning and another for quick, transient adaptation. Anand & Precup (2023) instantiated this idea using permanent and transient value functions, but used the same features in both systems. Intuitively, long-term learning should rely more on parametric representations for broad generalization, while transient learning would be best served by non-parametric representations for situation-specific adaptation. In this paper, we explore this idea both theoretically and empirically. By introducing a novel non-parametric approximation for estimating the transient value function in complex tasks, we demonstrate that our method enables online learning and outperforms competitive baselines on image-based tasks and Craftax-Classic, underscoring the effectiveness of our system-level decomposition.
