家用 GPU 从零训练葡萄牙语小模型：自回归与掩码扩散对比教程
Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese
一块消费级 GPU 就能从零训个葡萄牙语小模型，还附代码包和评测协议，想动手训 SLM 的可以直接照着跑。
这篇教程展示如何在一块消费级 GPU 上从零训练 11-12M 参数的语言模型，包括 tokenizer，并开源了 Python 包和两个巴西葡萄牙语 notebook。模型在同一网络上同时支持自回归和掩码扩散两种生成方式，并用预注册的对比协议在巴西散文和联邦法律文本上评测。结果显示自回归模式在 64 个 epoch 后开始记忆训练文本，掩码扩散可提升到 256 个 epoch，最优时扩散的对数似然比自回归高 0.06 bits/byte。在训练集之外的法律文本上，模型与 6-gram 基线打平，但在检测篡改文本和填写法令交叉引用方面超过基线以及 160M、630M 参数的公开模型。
Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese
Building a language model of one's own no longer requires a data centre. This tutorial trains one from scratch, tokenizer included, on one consumer GPU, and shows how to tell whether it is good for the tasks of a specialist domain. It proposes no new method. It releases a tested Python package and two notebooks in Brazilian Portuguese with autoregressive and masked diffusion generation on one network, a preregistered comparison protocol, legal tasks built from text alone with a Kneser-Ney floor and public yardsticks, reference numbers for models of 11 to 12M parameters, and a catalogue of defects that only measurement revealed. On Brazilian prose and federal laws, at minutes per run, the autoregressive mode peaks at 64 epochs over the same text and then memorises it, while masked diffusion improves up to 256. At their best points on test, the diffusion bound exceeds the autoregressive likelihood by 0.06 bits per byte, within the preregistered minimum effect. On held-out laws and on acts published after training, the model ties a 6-gram floor in likelihood on the same windows, and it loses to the floor at continuing those acts. It beats the floor at detecting tampered text, and fills cross-references more often than the floor and public models of 160M and 630M parameters, only in decrees, a genre absent from training, where those scorers stay near or below chance, and in filling the blank only when each group weighs the same, since per item it ties the 630M model. On laws, the public models, trained with about 5x10^2 to 10^5 times its compute, lead in tampered text and continuation. The blind human reading is preregistered and not part of this version.
