用信息价值统一强化学习记忆检索与信用分配
Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information
一篇有意思的理论论文：一个信息价值量同时解决记忆检索贵和信用分配难两个老问题，还给了可证下界。
arXiv 论文指出，强化学习中的记忆检索成本与信用分配由同一个量控制：信息价值。作者提出用学习到的世界模型从存档 rollout 中估计该量，并证明在观测处读取时，它给出获取动作最优价值的可证下界，可抬高 TD target 并把功劳归于对应观测。在决策处读取时，它定义最优检索集合，试点检索系统据此用更短的上下文完成记忆任务。
Cheap Retrieval and Observation Credit Assignment from One Quantity: The Value of Information
Memory in reinforcement learning raises two problems. Attention finds the relevant item by scoring the whole store, which is costly for robots under latency and energy limits. When a retrieved memory pays off, credit for acquiring it must cross the whole sequence. We argue that both are governed by one quantity, the value of information, and propose to estimate it from archived rollouts with a learned world model. We obtain that, when read at the observation, it yields a provable lower bound on the optimal value of the acquiring action, which floors its TD target and attributes this floor to the observation that earned it. Read at the decision, it defines the optimal retrieval set. A pilot retrieval system, to be trained and improved with this signal, solves a memory task with reduced context.
