Meta 公开 Prometheus 超级集群技术细节：支撑 Muse 模型训练
Meta 拆解了自家吉瓦级超算怎么保证几十万块 GPU 训练不中断，85% 的 goodput 数据对搞分布式训练的人很有参考价值。
Meta 在俄亥俄州 New Albany 的吉瓦级超算集群 Prometheus 上训练了 Muse 基础模型，包括 Muse Wonder。该集群拥有数十万块 GPU，Meta 针对单台主机故障拖慢整体训练的问题做了容错工程。结果显示预训练阶段 goodput 达 85% 以上，强化学习阶段达 90% 以上，Muse Wonder 训练期间平均无中断时长约 4.7 万 GPU 小时。
We trained our Muse foundational models, including Muse Wonder, at Prometheus, our gigawatt-scale supercluster in New Albany, Ohio. Across a supercluster with hundreds of thousands of GPUs, even one underperforming host can slow an entire training run. We engineered Prometheus for resilience at scale, achieving 85%+ goodput on pretraining, 90%+ on reinforcement learning and ~47K GPU hours mean time between interruptions during Muse Wonder training. Take a look at the tech powering Prometheus: go.meta.me/4yXXApZ Your browser does not support the video tag. 🔗 View on Twitter 💬 16 🔄 13 ❤️ 157 👀 6308 📊 34 ⚡
