arXiv 论文拆解 tokenization 评测三大误区
Tokenizer Myths and Where To Find Them
做 tokenizer 研究或选词表大小的都该看看：BPB 和那些"tokenizer 无关"指标没你想的那么可靠，文中给出了 0.989 相关性崩到 0.057 的具体分析。
一篇 arXiv 论文指出 tokenization 研究中的三个常见误区。其一是 Bits-Per-Byte（BPB）并非与 tokenizer 和词表大小无关，用未训练模型实验显示 BPB 会偏好特定 tokenizer-model 组合。其二是更低 BPB 不等于更好模型：未训练模型上 BPB 强烈偏好某些词表大小，而准确率仍是随机水平；真实模型上 BPB 与下游性能仅弱相关。其三是所谓 tokenizer 无关指标并不成立，unigram-normalized loss 与 BPB 的 0.989 相关来自混杂因素，修正后降至 0.057，两个指标在相同数据下会选出不同词表。论文提出 tokenizer 三难困境，认为选择评测指标就是在选择研究问题。
Tokenizer Myths and Where To Find Them
Tokenization is a fundamental component of modern language models: before a model sees data, its tokenizer has already determined how the data appears. We identify and challenge three myths in the tokenization research literature. Myth #1: Bits-Per-Byte (BPB) is agnostic to the tokenizer and/or the vocabulary size: BPB is tokens per byte times bits per token, and the two factors do not cancel; using untrained models, we show that BPB prefers some tokenizers over others: the model has learned nothing, but the tokenizer--model pair has. Myth #2: lower BPB means a better model: On untrained models, BPB strongly prefers some vocabulary sizes, even though accuracy remains at chance, and on real models trained on the same data, BPB correlates weakly with downstream performance. Myth #3: different "tokenizer-agnostic" metrics yield consistent results: The "unigram-normalized loss" is not tokenizer-agnostic; its reported correlation of 0.989 with BPB comes from a confounder and falls to 0.057 once fixed; from the same data the two metrics select different vocabularies at each model size. We present a tokenizer trilemma: no metric can score only the true distribution best, give the same predictions the same score, and tie all untrained models, so choosing a metric is choosing a research question. We conclude with a caution for reading the tokenization literature and a conjecture for future work.
