Meta 发表论文，字节级模型在算力提升后超越 token 模型
Banger paper from Meta. This work shows that byte-level models start out behind token models and th...
Meta 新论文说，字节级模型在算力够大时能超过 token 模型，还提到用 256 个词表能省存储，挺有意思的。
Meta 的研究显示，字节级模型在初始阶段落后于 token 模型，但随着计算资源的增加，它们会逐渐超越。该研究针对 1B 参数的蒸馏模型，在 1 万亿字节数据量上训练。字节模型在达到更高性能上限的同时，还能用更少的数据（六分之一）和更小的词汇表（256 个）进行训练，存储开销仅为 token 模型的五分之一。
Banger paper from Meta. This work shows that byte-level models start out behind token models and th...
Banger paper from Meta. This work shows that byte-level models start out behind token models and then pass them as compute grows. They show this for distilled 1B models trained on up to 1 trillion bytes. To distill a byte student from a token teacher, they convert the teacher's token logits into byte logits, either approximately (Marginalize-It) or exactly (End-Of-Token). Token models lead at low compute but plateau. Byte models reach a higher ceiling, and the fitted scaling laws predict the End-Of-Token model ends up to 4% ahead of the distilled token model. The byte models also match the distilled token model with one-sixth of the training data, and a 256-entry vocabulary cuts teacher-logit storage to about a fifth. Paper: arxiv.org/abs/2609.12303 Chat with Paper: academy.dair.ai/papers/breakin… 💬 6 🔄 4 ❤️ 31 👀 2055 📊 12 ⚡
