HiLo-RD：用跨预算自蒸馏提升掩码扩散语言模型小窗口推理
HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models
arXiv上新出的自蒸馏方法，专门修MDLM小窗口推理时过早下结论的毛病，GSM8K涨了9.4个点还更省token。
HiLo-RD（High-to-Low Budget Representation Distillation）针对掩码扩散语言模型（MDLM）在低生成预算下的问题，提出让大窗口教师状态与小窗口裁剪视图配对，并在已提交位置匹配隐藏表示的自蒸馏方法。论文指出模型存在预算导致的过早提交现象：正确解尚未完成中间推理时，答案片段已在窗口边界附近被固定，进而误导后续预测。在 GSM8K 上，该方法将低预算准确率提升 9.4 个百分点，平均可读长度下降约 25%。改进还迁移到代码生成任务，且无需代码域训练。
HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models
We propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget. Unlike autoregressive models, MDLMs choose both what to generate and where to commit it. This creates two coupled challenges: predicting useful content before reasoning is complete, and deciding when to fix answer tokens that shape later predictions. We identify budget-induced premature commitment: even when a correct solution fits, the model can fix answer fragments near the window boundary before resolving intermediate steps. These fragments become context for later predictions and redirect subsequent reasoning. Existing post-training methods do not directly transfer the stronger trajectories that the same model follows with a larger window. HiLo-RD addresses this gap by (1) pairing large-window teacher states with cropped small-window views and (2) matching their hidden representations at committed positions. This supervision guides both token prediction and commitment order without an external teacher. Across mathematics benchmarks, HiLo-RD improves low-budget accuracy and shortens responses; these gains extend to code generation without code-domain training. On GSM8K, accuracy rises by 9.4 percentage points as mean readable length falls by about 25%. Attention diagnostics and fixed-budget order interventions support the proposed mechanism. Together, these results show that large-window reasoning can improve small-window decoding.