CanvasAnneal框架提升扩散语言模型推理能力
CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
这是篇研究论文，作者提出了一种叫CanvasAnneal的新框架，专门用来提升扩散语言模型做复杂推理的能力，比如数学题和用工具，效果比之前的方法好。
本文提出CanvasAnneal框架，通过将强教师模型的推理先验注入扩散画布来指导强化学习探索，解决了扩散语言模型（DLMs）在复杂推理任务中的探索瓶颈问题。在数学推理和工具使用基准测试中，该框架在MATH500、Countdown和Tau2等任务上优于标准diffu-GRPO方法，并显著加速了奖励提升。
CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models
Diffusion Language Models (DLMs) offer promising parallel generation capabilities but lag behind autoregressive models in complex reasoning and tool-use tasks. While Reinforcement Learning (RL) has recently been applied to enhance DLMs, standard RL approaches suffer from an exploration bottleneck. To address this, we inject reasoning priors from a stronger teacher model to guide RL exploration. In this paper, we introduce CanvasAnneal, a curriculum-guided diffusion RL framework. During the initial RL phase, we warm-start exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas. As training progresses, we gradually remove this guidance and require the model to generate more of the reasoning trajectory independently. Across mathematical reasoning and tool-use benchmarks, CanvasAnneal improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and substantially accelerates reward improvement on several tasks, while gains are task-dependent. Our results suggest that structured training-time guidance can alleviate exploration bottlenecks in diffusion RL and speed up convergence on harder tasks.
