研究量化语言模型的分词溢价：阿姆哈拉语成本最高达10倍
Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South
写论文或做多语言产品的朋友可以看这份测量：7 个 tokenizer、31 种语言的真实成本账，阿姆哈拉语最高要多付 10 倍 token 费。它还给了买家 8 项黑盒审计测试，不需要权重就能评估模型的主权风险，这对非英语市场团队很实用。
arXiv 论文（编号 2611.13441）测量了 2023-2024 年间 7 个主流 tokenizer 在 31 种语言上的分词溢价，基准材料取自《世界人权宣言》的 27 篇条文，并修正了未知 token 塌缩带来的偏差。结果显示，阿姆哈拉语在旧 tokenizer 中的溢价达 10 倍，在最新商用 tokenizer 中仍有 6.8 倍；南美原住民语言在全球 tokenizer 下需支付 1.6 至 2.7 倍的成本。
作者的核心论点是：大多数部署在美国和中国之外的模型，本质上是少数预训练家族微调的后代，其词表、知识、文化对齐与许可证等关键属性全部继承自基座模型。实验证明，用葡萄牙语构建的词表可将 13 种拉丁字母全球南方语言中 11 种的溢价降低，其中瓜拉尼语和克丘亚语约降低 24%。
论文还提出一套包含 8 项测试的黑盒主权审计方法，买家无需访问模型权重即可运行；并在能耗与责任约束下，主张以每焦耳准确率、校准度和选择性风险取代裸准确率作为设计目标。数据与测量代码已随论文公开。
Inherited Weights: Measuring the Tokenization Premium and the Sovereignty Cost of Language Models for the Global South
Most language models deployed outside the United States and China are fine-tuned descendants of a small number of pretrained families. We argue that fine-tuning leaves the decisive properties of a model inherited from its base: vocabulary, knowledge, cultural alignment, memorized data and license. We contribute (i) an updated, reproducible measurement of the tokenization premium across 31 languages and 7 tokenizers released in 2023-2024, on 27 articles of the Universal Declaration of Human Rights, with explicit correction for unknown-token collapse. Premiums reach 10x for Amharic in older tokenizers and remain at 6.8x in the newest commercial one; Indigenous South American languages pay between 1.6x and 2.7x under global tokenizers. A vocabulary built from Portuguese lowers the premium for 11 of 13 Latin-script Global South languages relative to every global tokenizer tested, by about 24% for Guarani and Quechua. (ii) An eight-test black-box sovereignty audit that a buyer can run without access to weights. (iii) A regime argument: under energy and liability constraints, accuracy per joule, calibration and selective risk should replace raw accuracy as design objectives, with metrics that abstention cannot inflate. We close with preliminary internal measurements from a proprietary architecture built around these objectives, reported with their limitations. Data and measurement code are included.
本条情报归属 AITOP「论文」核心追踪栏目。涉及核心实体与技术议题包括：tokenizer、分词、多模态、Global South，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Paulo Camara），经 AITOP 多维评分管线结构化提炼。
