MLR方法：将长链推理拆解为规划与执行两层，提升小模型推理稳定性
Enhancing Language Model Reasoning with Structured Multi-Level Modeling
一篇把长推理拆成规划加执行两层的论文，小模型只用10%数据就跑赢DeepSeek-R1蒸馏基线，做推理优化的可以看看。
arXiv 论文提出 Multi-Level Reasoning（MLR），把长思维链生成分解为高层规划器和低层执行器组成的两级随机过程，规划器输出推理模式与语义子目标，执行器基于描述生成具体推理。为解决 outcome-reward RL 在数千 token 长轨迹上反馈稀疏的问题，作者引入迭代 Step-DPO，借助 Twisted Sequential Monte Carlo（TSMC）提供逐步监督。基座模型充当低层策略，高层策略仅用一个轻量 LoRA 模块实现。在数学、科学与逻辑推理基准上，MLR 仅用 DeepSeek-R1 蒸馏设置 10% 的 SFT 数据和 5% 的偏好数据就超过了 SFT 蒸馏和强 RL 基线。长思维链场景下 MLR 的性能衰减也更慢。
Enhancing Language Model Reasoning with Structured Multi-Level Modeling
Inference-time scaling enhances a model's reasoning by extending its chain-of-thought (CoT). However, existing approaches typically rely on a single policy trained with outcome-reward reinforcement learning (RL), which often suffers from long-horizon plan failures where the implicit plan drifts from valid strategies, especially for small LMs with limited capacity. To address this, we propose Multi-Level Reasoning (MLR), which reformulates long-CoT generation as a two-level stochastic process. A high-level planner generates structured step descriptors specifying both the reasoning mode and the semantic subgoal. The low-level executor then produces detailed reasoning conditioned on these descriptors, forming an alternating plan--execute loop. To maintain scalability, we adopt a minimal design where the base model serves as the low-level policy and a lightweight LoRA module implements the high-level policy. For training, we observe that outcome-reward RL provides sparse and delayed feedback for long trajectories (e.g., several thousand tokens), hindering credit assignment. We therefore introduce iterative Step-DPO, a process-level preference optimization scheme that leverages Twisted Sequential Monte Carlo (TSMC) to provide scalable stepwise supervision. This yields more effective training, improved stability, and higher accuracy. Extensive experiments on challenging math, science, and logical reasoning benchmarks show that, under the same reduced data budget (10% SFT and 5% preference relative to the DeepSeek-R1 distillation setup), MLR outperforms both SFT-based distillation and strong RL/preference-optimization baselines across multiple base models and tasks. Moreover, MLR exhibits slower performance degradation on long-horizon reasoning, demonstrating stronger robustness under extended CoT generation.