DatologyAI 推出 Zephon，让 TorchTitan 换 GPU 数量恢复训练时全局批次不变
DatologyAI 做了 Zephon，在 TorchTitan 里换 GPU 数量恢复训练，8/32/64 卡的全局批次哈希都能对上 16 卡原始运行，做分布式训练的值得看看。
DatologyAI 发布 Zephon 与 TorchTitan 的集成方案，解决训练恢复时更换 GPU 数量导致数据批次不一致的问题。在一项 1B 参数模型的测试中，训练从 16 GPU 的 checkpoint 恢复到 8、32、64 GPU，每个全局批次的 SHA-256 哈希都与原始运行完全一致。Zephon 在训练作业内部完成数据的分词、打包、打乱和混合，无需为语料库生成预处理副本。恢复训练时需保持数据、分词器、种子、混合比例、上下文长度和全局批次大小不变。由于浮点归约顺序依赖 GPU 数量，训练损失仍可能有差异。
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ Josh Wills, Maximilian Böther, Ties Robroek (DatologyAI)
Read more: https://t.co/JKn45hQarm