Accordion 发布：单一 MoE 模型支持 5 档推理预算
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
一个 6B 激活 0.5B 的 MoE 模型，跑循环层数就能切 5 档推理深度，小预算也能拿到 89% 的深度增益，挺实用的思路。
学术论文 Accordion 提出基于 fixed-prefix layer looping 的 mixture-of-experts 模型，单一模型可支持 5 档推理预算，每档对应一条预定义执行路径，最低档保留未循环的原模型。预训练消耗 3.6T tokens，每个 micro-batch 只采样一个预算并优化其 next-token prediction loss。Accordion-6B-A0.5B 在 37 个任务上的五类 macro-average 通过增加推理深度提升 3.01 个百分点，其中间档用一半的额外层执行就拿到 89% 的增益。经 SFT 和强化学习后，最终模型通过参数组合并与 budget-matched multi-teacher on-policy distillation，将三个领域专家合并进一个模型，在对应任务与预算的平均成绩上超越各领域专家。
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
Large language models serve applications with diverse inference budgets, but training and maintaining separate models for each budget incurs substantial costs. Recent Looped Transformers enable flexible inference computation through parameter reuse, but maintaining strong performance across different inference budgets remains an open challenge. We introduce Accordion, a mixture-of-experts-based model that supports five inference budgets through fixed-prefix layer looping. Each budget follows a pre-defined execution path, while the minimum budget retains the unlooped model. Joint pretraining samples one budget per micro-batch and optimizes its next-token prediction loss without additional cross-budget consistency objectives. We further preserve these inference budgets through supervised fine-tuning and reinforcement learning, where parameter-group merging and budget-matched multi-teacher on-policy distillation integrate three domain experts into one model. After 3.6T tokens of pretraining, increasing inference depth improves Accordion-6B-A0.5B's five-category macro-average across 37 tasks by 3.01 percentage points, with the middle budget capturing 89% of this gain using half as many additional layer executions. The final model surpasses each domain expert on its corresponding task-and-budget average, while achieving strong instruction-following performance across all budgets and retaining depth-dependent gains in mathematical reasoning and code generation.