Meta 发表论文，字节级模型在算力提升后超越 token 模型

字节模型在达到更高性能上限的同时，还能用更少的数据（六分之一）和更小的词汇表（256 个）进行训练，存储开销仅为 token 模型的五分之一。

Banger paper from Meta. This work shows that byte-level models start out behind token models and then pass them as compute grows. They show this for distilled 1B models trained on up to 1 trillion bytes. To distill a byte student from a token teacher, they convert the teacher's token logits into byte logits, either approximately (Marginalize-It) or exactly (End-Of-Token). Token models lead at low compute but plateau. Byte models reach a higher ceiling, and the fitted scaling laws predict the End-Of-Token model ends up to 4% ahead of the distilled token model. The byte models also match the distilled token model with one-sixth of the training data, and a 256-entry vocabulary cuts teacher-logit storage to about a fifth. Paper: arxiv.org/abs/2609.12303 Chat with Paper: academy.dair.ai/papers/breakin…📊 12 ⚡

本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Meta、字节级模型、token 模型、蒸馏，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明：本文原始线索采编自「elvis」（@omarsar0），经 AITOP 多维评分管线结构化提炼。截至目前，全网已有 2 家独立信源对该事件进行交叉印证与跟进报道（含 量子位 等），属于近期高关注度行业信号。