10个开源LLM新架构一览：从Kolibri-1到Mistral Large 4
rasbt 整理了10月10个开源 LLM 架构，从 Kimi Delta Attention 到 LatentMoE，想跟进架构趋势的别错过这份清单。
10月出现10个开源LLM架构发布。德国的 Kolibri-1（78B）采用 GQA 加 Sliding Window Attention。Meta 的 Muse Spark Open（816B）在 GQA+SWA 基础上引入 LatentMoE。Nemotron 3.5 Super VL（120B）把 Mamba-2 层与仅8个注意力层结合，MiniMax M3.1-Flash（428B）使用自研 Sparse Attention，Mistral Large 4（1T）则组合 MLA 与稀疏注意力。
Catching up on the 10 (!) open LLM architecture releases from the past month. Quite a lot happened in October. I guess we can call it "Open October"!
1. Kolibri-1 (78B): A notable model from Germany. Features classic Grouped Query Attention (GQA) and Sliding Window Attention (SWA) attention module.
2. Ling 3.1 Flash (560B): Uses Kimi Delta Attention and the eighth uses gated Multi-head Latent Attention (MLA) inspired by DeepSeek.
3. Muse Spark Open (816B): Besides Musse Glimmer, the first major open weight release from Meta (since Llama). Uses classic GQA + SWA but adds a LatentMoE.
4. Nemotron 3.5 Super VL (120B): Combines Mamba-2 layers with just eight attention layers. Also LatentMoE
5. Dory1.0 (4B): A looped transformer design.
6. Reflection Beam (501B): GQA+SWA architecture.
7. Recursive Flint (40B): Note that "Recursive" is the company name; the architecture is a fairly standard MoE with GQA and SWA.
8. MiniMax M3.1-Flash (428B): Uses their own MiniMax Sparse Attention module.
9. LFM3 (1.7B): Hybrid architecture that mixes short convolutions and attention.
10. Mistral Large 4 (1T) combines MLA with sparse attention.
Will probably cover/discuss things in more depth in the coming weeks!
