百度开源 ERNIE-5.5-Flash-Base：280B MoE 基座，支持 1M 上下文
百度开源的 280B 基座模型，激活参数只有 14B，KV cache 压缩方案挺巧，自测分数比 DeepSeek-V4-Flash-Base 高一截，做二次训练的可以看看。
百度发布 ERNIE-5.5-Flash-Base 基座模型权重，280B MoE 架构，每 token 仅激活 14B 参数，基于 33.5T token 从头预训练。模型只有 43 层中的 6 层保留完整上下文访问，其余 37 层将每 128 个 token 压缩为单一缓存条目，以缩小 KV cache，同时支持最高 1M token 上下文。还引入多路由 MoE 与 ECHO 多 token 预测机制，后者复用模型自身的最终层而无需独立模块。在百度对 DeepSeek-V4-Flash-Base 的初步评测中，MMLU-Pro 为 78.4 对 69.2，HumanEval+ 为 82.3 对 69.5。权重已以 FP8 和 BF16 格式上线 Hugging Face，但许可条款将欧盟、英国和韩国排除在允许地区之外。
Baidu just dropped ERNIE-5.5-Flash-Base, and the weights are available.
280B MoE, only 14B active per token, pretrained from scratch on 33.5T tokens. It's a pure base model with no mid- or post-training, so labs can adapt it for domain-specific tasks, reasoning, and agents.
What makes it exciting is how hard it pushes on efficiency:
- Only 6 of 43 layers keep token-level access to the full context. The other 37 squeeze every 128 tokens into a single cached entry, which keeps the KV cache small. Context still goes up to 1M tokens for fine-tuning and inference. - Multi-Router MoE: several routers pick the experts together instead of just one, without increasing activated expert compute or communication costs. - ECHO: multi-token prediction runs through the model's own final layer, no separate MTP module needed.
And the early numbers look strong. In Baidu's preliminary evals against DeepSeek-V4-Flash-Base, a model of almost the same size:
- MMLU-Pro: 78.4 vs. 69.2 - HumanEval+: 82.3 vs. 69.5 - LiveCodeBench: 40.1 vs. 31.9
Weights are live on Hugging Face in FP8 and BF16: https://t.co/QsoSuaTeCt
The model license excludes the EU, UK, and South Korea from its permitted territory.
