4 GPUs, 2x speed: why PyTorch DDP stalled on a PCIe-only 4x RTX A6000, and two flags that took it to 3.6x

Plain DDP on a rented 4x RTX A6000 gave 2x. Measured all-reduce bandwidth, then bf16 gradients and accumulation took it to 3.6x. Commands inside.