Stanford 论文：BPB 并非真正的 tokenizer 无关指标
选 tokenizer 别只看 BPB 分数，Stanford 这篇用未训练模型实测证明它会骗人，词表大小也影响它，记得对同一文本验下准确率。
Stanford 的新论文指出，bits-per-byte（BPB）常被称为 tokenizer 无关的评测指标，但它做不到。作者用完全未训练的随机模型做实验，发现这些模型在不同 tokenizer 下 BPB 依然不同，其中 24K 词汇量在英文文本上得分最好，而准确率仍停留在随机水平。在用不同 tokenizer 训练的 released 模型上，BPB 与任务分数的相关系数仅落在 +0.02 到 −0.30 之间。论文还复核了一项声称另一个 tokenizer 中立指标与 BPB 一致的已发表结论。作者建议比较 tokenizer 时不要把更低的 BPB 当作模型更好的证据，而应在同一文本上核对任务准确率。
New Stanford paper finds that no metric can be tokenizer-agnostic in every sense, so anyone comparing tokenizers should not trust a single score and should also check whether the model answers correctly.
To compare models with different tokenizers, researchers usually report bits-per-byte, or BPB, which divides the loss by bytes instead of tokens. Many papers call it tokenizer-agnostic.
They scored untrained models that guess uniformly, plus released models that share one recipe but use different tokenizers. They also rechecked a published claim that another supposedly tokenizer-neutral metric agrees with BPB.
Models that learned nothing still got different BPB by tokenizer, with English text scoring best near a 24K vocabulary, while accuracy stayed at chance. On models trained with different tokenizers, correlations between BPB and task scores ran only from +0.02 to −0.30.
When comparing tokenizers, don't treat lower BPB as proof of a better model, and check task accuracy on the same text before picking a vocabulary.
