PyTorch Symmetric Memory backend NCCL/NVSHMEM vs PyTorch Distributed backend NCCL

Hi, We are trying to understand the LLM (>70b; two variants: dense, MOE) pre-training communication. To better leverage the communication paradigms available in PyTorch, I explored relatively newer additions, i.e., Symmetric Memory with backends: cuda, nccl, & nvshmem. If I understand correctly, dist backend nccl uses nccl for the control plane and data plane, whereas the symm mem backend nccl uses NCCL for the control plane (like bootstrapping) while giving some additional features than symm ...