RIME：用强化学习训练模型把经验写成记忆，4B 模型比肩 GPT-5.4

RIME 让同一个模型扮演两个角色：actor 负责用已有记忆执行任务，writer 负责回顾过程并更新供下一任务使用的记忆笔记。两个角色的奖励信号不同——writer 在接下来几个任务的表现在近期任务基础上提升时获得奖励，actor 完成任务和做出可入记忆的发现时获得奖励，并由一个 AI judge 对比新旧笔记，寻找有用补充、修正的信念和具体线索，重复改写则几乎不给奖励。

作者指出这两种能力必须共同演化：actor 的发现喂给 writer，writer 保存的知识又反哺 actor。当前外部记忆系统的短板不在存储本身，而在模型对何时写入、写什么的判断力。研究者认为高效的持续学习最终仍需更新权重，开源权重模型因此会发挥重要作用。

Highly recommended read. My notes: I have recently started independent research on co-evolved models. My focus has been on co-evolving models and harnesses. This work focuses on memory writing specifically for continual learning. Why is this a big deal? As you work with an agent, it should get better at your tasks, but that obviously doesn't happen easily, and current external memory systems often aren't enough. This has less to do with memory systems (there are many great ones out there) and more to do with the model's awareness and judgment about when to act. To illustrate, imagine an AI agent spends a week fixing bugs in the same project. We would expect it to learn how the project works and avoid repeating mistakes. Giving it a place to save notes lets it carry that knowledge into later tasks. But choosing what belongs in those notes requires judgment. This work looks at how to train that judgment with RL. They essentially teach an agent to discover useful information while working and write memories that help it succeed on later tasks. A trained 4B parameter model reached a score similar to GPT-5.4 and Claude Opus 4.7. RIME uses one model in two roles. As the actor, it works on a task using its existing memory. As the writer, it reviews what happened and updates the notes available for the next task. The training rewards each role differently. The writer receives credit when performance on the next few tasks improves relative to performance on the recent tasks. This gives it a reason to preserve information that will help later. The comparison is an approximate training signal, since other actions and memory updates also affect those results. The actor earns a reward for completing its task and an additional reward for discoveries that reach memory. An AI judge compares the old and new notes, looking for useful additions, corrected beliefs, and concrete leads. This encourages the agent to investigate enough to learn something reusable. Rewriting the same notes in different words earns little credit. The researchers found that these abilities need to develop together. A writer depends on the information the actor discovers, and an actor depends on the knowledge the writer preserves. One big question I have here is efficiency and, obviously, how it all integrates into harnesses and workflows. I think approaches like this can save a lot of tokens and turns. While this is on a small scale, it's great to see it working already. It feels like a more feasible approach for continual learning. Frontier models are getting better at retaining knowledge, but I think efficient continual learning will still require touching the weights, which is where open-weight models will play a huge role. Aviral Kumar @aviral_kumar2 How can we train LLMs to write memories from experience collected on the fly to as best as possible solve the next task, i.e., optimize for both capability and "continual learning" ability? In our new post, we study this question in the setting continual learning over a sequence of tasks, and develop a simple approach to train models to write effective memories cmu-aire.github.io/RIME/ *Spoiler*: the picture is more nuanced than you think 🧵 ⬇️ 🔗 View Quoted Tweet📊 12 ⚡

本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：RIME、持续学习、强化学习、智能体，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明：本文原始线索采编自「elvis」（@omarsar0），经 AITOP 多维评分管线结构化提炼。