论文分析两种主流分词算法的优劣
Objective vs. Search: Decomposing What Makes a Good Tokeniser
这篇论文很硬核，专门拆解了两种分词算法的优劣，对做大模型的研究者很有参考价值。
本文通过对比两种主流分词算法（BPE和UnigramLM），发现搜索过程（bottom-up vs. top-down）是影响模型性能的关键因素，而非优化目标（压缩vs.对数似然）。研究通过新算法验证了这一结论，并指出在bits-per-byte基准上，bottom-up方法表现更优。
Objective vs. Search: Decomposing What Makes a Good Tokeniser
Two dominant tokenisation algorithms are used by modern language models: byte-pair encoding (BPE) and UnigramLM. These differ along two orthogonal axes: their optimisation objective (compression vs. log-likelihood) and their search procedure (bottom-up merging vs. top-down pruning). Existing comparisons confound these axes, making it unclear whether their observed differences stem from what is being optimised vs. how it is being optimised. We disentangle the two by introducing two new tokenisation algorithms that complete this 2x2 design space: BottomUpLL, a bottom-up likelihood-based tokeniser, and TopDownComp, a top-down compression-based tokeniser. We train language models with tokenisers produced by each algorithm, varying: model size, vocabulary sizes, and domain (English-only vs. multilingual). Evaluating models on bits-per-byte, we find that the search procedure -- not the objective -- is the dominant factor: bottom-up tokenisers consistently achieve lower bits-per-byte in most settings. Evaluating models on the BLiMP task, however, shows no consistent relationship between design choice and performance. Overall, our results disentangle the effect of tokeniser design choices on language modelling performance, offering concrete guidance for their more principled construction.
