MCCL：基于测量的集合通信库，应对灰色故障
MCCL: Measurement-based Collective Communication Library
华为系出的一篇系统论文，链路悄悄变慢时能边测边重排通信调度，比静态方案快最多 4.99 倍，做分布式训练的可以看看。
集合通信是大规模分布式训练的性能瓶颈，现有库和调度合成框架都基于静态拓扑和峰值链路带宽一次性算好调度。MCCL 通过被动测量 GPU 间吞吐量，检测持续的链路劣化，用热启动合成器重新生成受影响的通信调度并热替换，整个过程不中断训练。在多节点硬件原型上验证，开销低于 2%，测量误差小于 1.2%，调度热替换时间约 12 ms。模拟实验中，在灰色故障下针对 Llama、DeepSeek 等模型的训练负载，MCCL 比静态调度最多将训练时间缩短 4.99x。
MCCL: Measurement-based Collective Communication Library
Collective communication is a critical performance bottleneck in large-scale distributed training. Existing libraries and synthesis frameworks compute open-loop schedules once, assuming a static topology and peak link capacities. In production, however, persistent gray failures can silently degrade link throughput and latency for hours, causing pre-computed schedules to lose much of their efficiency during execution. In this paper, we present Measurement-based Collective Communication Library (MCCL), a collective communication library that closes the loop between runtime network measurement and dynamic schedule synthesis. MCCL passively measures GPU-to-GPU throughput, detects persistent degradation, re-synthesizes the affected communication-intensive schedules with warm-started synthesizers, and hot-swaps them without interrupting training. We validate the complete loop on a multi-node hardware prototype and use simulations to evaluate cluster-scale scenarios. In our experiments, MCCL incurs negligible overhead (<2%) while maintaining high measurement accuracy (<1.2% error) and low schedule hot-swap time (about 12 ms). Notably, under gray failures, MCCL reduces training time by up to 4.99x over static schedules on simulated training workloads for state-of-the-art LLMs such as Llama and DeepSeek.
