论文

LOOT 架构让掩码扩散语言模型具备自纠错能力

Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers

精选理由

掩码扩散模型一直被人诟病生成后改不了字,这篇论文用 LOOT 架构解决了这个痛点。在 OpenWebText 上推理开销只有之前方法的零头,MAUVE 还更高。做非自回归生成、对推理成本敏感的团队可以认真读一下。

Intel 等机构提出 Leave-One-Out Transformer(LOOT),针对掩码扩散模型(MDM)无法修改已生成 token、训练信号稀疏两大缺陷,在 hollow transformer 基础上重新设计左右上下文融合机制,并推导出更低方差的 MDM NELBO 估计器,使所有 token 都获得监督信号。

正文

推理阶段,LOOT 可以基于上下文独立预测每个位置,从而在生成过程中对 token 进行有效自纠错。基准成绩方面,在 LM1B 数据集上验证困惑度追平最强稠密监督 MDM 基线;在 OpenWebText 上 MAUVE 得分和生成质量-多样性权衡均优于已有 remasking 方法,前向评估次数减少 8 倍,推理 FLOPs 减少 3.5 倍。

对开发者而言,这意味着扩散式并行生成语言模型在效率指标上已具备与自回归方案竞争的资本,self-correction 能力也让非自回归文本生成路线更接近实用。

原文 · arXiv cs.LG

Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers

Masked diffusion models (MDMs) have emerged as a promising paradigm for language modeling, enabling parallel token generation with competitive performance. Yet standard MDMs suffer from two fundamental limitations: (1) inability to revise tokens, and (2) sparse training signal. We revisit hollow transformers, which address both limitations by predicting each token from its surrounding context, independently of the input at that position. We introduce the Leave-One-Out Transformer (LOOT), a more efficient and expressive architecture that advances hollow transformers through two complementary advances. First, we redesign the merging mechanism to enable frequent interaction between left and right contexts, substantially improving predictive performance over prior hollow transformers. Second, we derive a lower-variance estimator of the MDM NELBO that supervises all tokens and improves validation perplexity. Together, these advances yield a competitive standalone diffusion language model capable of effective self-correction during inference. On LM1B, LOOT matches the strongest dense-supervision MDM baseline in validation perplexity. On OpenWebText, LOOT achieves higher MAUVE and better generation quality-diversity tradeoffs than prior remasking methods, using $8\times$ fewer forward evaluations and $3.5\times$ fewer inference FLOPs.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:LOOT、MDM、扩散模型、文本生成,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Sofian Zalouk, Vincent Counathe, Paul Jünger, Daniel Cao, Daniel Zlotnick, Kilian Q. Weinberger, Christopher De Sa),经 AITOP 多维评分管线结构化提炼。