LLaCO：面向大语言模型的连续扩散生成框架
LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling
一篇把连续扩散做文本生成的理论工作，8B 模型追平了同规模离散扩散，搞生成模型理论的可以看看。
LLaCO 是一个基于连续扩散的非自回归文本生成框架，采用 8B 参数规模。论文提出 Competition Threshold Theory（CTT），指出嵌入范数衰减会导致欧氏距离产生虚假超越，token 竞争必须按方向性评估。为解决时间压缩问题，作者提出 Semantic Retention Scheduling（SRS），将去噪时间分配与 token 身份衰减的对数秩衰减对齐。LLaCO 通过嵌入方向适配器和时间步条件调制改造预训练骨干，并使用球面加噪过程消除范数衰减。在 8B 规模上，LLaCO 超越此前连续扩散基线，与同规模离散扩散模型相当。
LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling
Continuous diffusion models have emerged as a promising non-autoregressive paradigm for text generation over continuous embedding spaces. Along the continuous trajectory, the residual token identity directly shapes the denoising objective. However, the geometric trajectory misaligns with temporal progression, triggering severe temporal compression. In this paper, we show that embedding norm decay causes Euclidean distance to yield spurious overtaking, indicating that token competition must be evaluated directionally. In this space, a candidate overtakes the source token only when the effective noise magnitude surpasses their semantic separation. We formalize these pairwise dynamics as the Competition Threshold Theory (CTT). Crucially, high-dimensional overtaking requires extreme noise levels, causing standard trajectories to concentrate identity decay into a narrow window near the noisy endpoint. To resolve this, we propose Semantic Retention Scheduling (SRS) to align time allocation with the progression of identity decay, and demonstrate that existing token identity metrics are fundamentally statistical aggregations of these pairwise crossings. Building on these theoretical insights, we introduce LLaCO, a generative framework powered by competition-aware semantic scheduling. LLaCO adapts a pretrained backbone to continuous representations via an embedding directional adapter and timestep-conditioned modulation. To operate within the directional space, LLaCO employs a spherical noising process formulated as a geodesic interpolation toward a mask-biased endpoint, eliminating norm decay while preserving accurate token competition. Guided by SRS, LLaCO reallocates the denoising trajectory proportionally to the logarithmic rank decay. Instantiated as an 8B model, LLaCO outperforms previous continuous diffusion baselines and rivals discrete diffusion models of comparable scale.