TIDE：零拷贝集合通信优化让 FSDP2 训练步时最多降低 27.1%
The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch
训练大模型时通信开销很头疼，这篇把 FSDP2 的布局转换和 reduce-scatter 都重写了，64 卡上步时省 27%，还给出了具体实现思路。
TIDE 是一套基于 NCCL 2.31+ 公开 API 构建的零拷贝集合通信方案，解决 FSDP2 中布局转换和通信内核的性能损耗。在 Llama-3.1 70B 训练中，仅布局转换就占步时 6.4%，默认 NCCL reduce-scatter 可让并发的 GEMM 变慢 69%。TIDE-AG 借助 NVSwitch 多播和 RDMA 将分片直接送达参数存储，TIDE-RS 则支持 NCCL 不具备的 bf16 转 fp32 混合精度归约。在 64 GPU 上，TIDE 相比 Stock FSDP2 最多降低步时 27.1%，相比 FSDP2+SymmMem 降低 6.0%，在 Llama-3.1 70B、Wan2.2 和 Qwen3-30B-A3B 上均有效，峰值内存最多比 FSDP2+SymmMem 低 4.7 GiB。
The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch
Fully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alone accounts for 6.4% of step time in Llama-3.1 70B training at 8K tokens per GPU. Communication kernels impose a second cost: a default NCCL reduce-scatter can slow a concurrent GEMM that would otherwise complete in a single wave by 69%. We present TIDE, a set of zero-copy collectives built on public NCCL 2.31+ APIs. TIDE preserves FSDP2's per-parameter storage and compute kernels while folding layout placement into collective addressing. TIDE-AG delivers shards directly to parameter storage using NVSwitch multicast and paced recursive-doubling RDMA, with no SMs used for transport or placement. TIDE-RS consumes per-parameter bf16 gradients directly and reduces them into fp32 shards (a mixed-precision mode that NCCL does not provide) using copy engines and short-lived thread blocks (CTAs) for node-local accumulation and NVSwitch for cross-node reduction. A two-phase pipeline separates gradient reads from network progress; Slot-Reuse optionally stages gradients in vacated all-gather slots. On 64 GPUs, TIDE reduces step time by up to 27.1% relative to Stock FSDP2 and 6.0% relative to FSDP2+SymmMem across Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B. It reduces reduce-scatter SM time by factors of 12.5-14.5 and cuts interference with concurrent compute by up to an order of magnitude. Its reduce-scatter buffer remains nearly constant from 16 to 64 GPUs at a fixed per-GPU output size, and its peak memory is up to 4.7 GiB below FSDP2+SymmMem's and, with Slot-Reuse on Wan2.2, below Stock FSDP2's.
