ReMaRL:在冻结扩散语言模型中学习解掩码与重掩码策略
ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models
掩码扩散模型的解码顺序其实很影响准确率,这篇论文用 0.3M 参数的小策略在冻结的 LLaDA-8B-Instruct 上学怎么解掩码和重掩码,数学和代码都跑赢了启发式方法。做扩散语言模型推理优化的人值得看。
论文提出 ReMaRL(Remasking with Reinforcement Learning),一个约 0.3M 参数的轻量策略,用于决定掩码扩散语言模型中哪些 token 应解掩码、保留或重新掩码。该策略使用 GRPO(Group Relative Policy Optimization)从最终答案奖励直接训练,训练期间扩散模型保持冻结。
现有方法要么用启发式规则控制解掩码和重掩码,要么训练监督式重掩码器检测 token 错误,都无法直接以最终答案正确性为目标优化决策。ReMaRL 的关键点是允许重掩码:模型可以基于新生成的上下文修正早期预测,而现有的任务奖励解掩码策略只能单向解掩码。
在 LLaDA-8B-Instruct 上的实验显示,ReMaRL 在数学任务(4-32 次模型调用)和代码任务(8-16 次调用)的推理预算内取得了对比方法中的最高分数。
ReMaRL: Learning to Unmask and Remask in Frozen Diffusion Language Models
Masked diffusion language models generate tokens in parallel, and the order in which tokens are unmasked affects final-answer accuracy. Several methods learn unmasking policies from task rewards, but these policies cannot remask earlier predictions. Remasking lets the model correct those predictions using newly generated context. Heuristic rules guide both unmasking and remasking, while supervised remaskers learn to detect token errors. Neither approach directly optimizes these decisions for final-answer correctness. To address this gap, we introduce ReMaRL (Remasking with Reinforcement Learning), a lightweight policy that learns which tokens to unmask, retain, or remask. We train this roughly 0.3M-parameter policy with Group Relative Policy Optimization (GRPO) from final-answer rewards while keeping the diffusion model frozen. With LLaDA-8B-Instruct, ReMaRL achieves the highest scores among the compared unmasking and remasking policies within the tested inference budgets of 4-32 model calls on math and 8-16 calls on code.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:ReMaRL、LLaDA、GRPO、扩散语言模型,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Egor Shibaev, Danila Biktimirov, Ivan Klimov, Artem Makoian, Viacheslav Meshchaninov, Dmitry Vetrov),经 AITOP 多维评分管线结构化提炼。