A small NCCL probe to distinguish initialization hangs from the first collective
PyTorch Forums
A small NCCL probe to distinguish initialization hangs from the first collective
Disclosure: I maintain TBR8, where the companion tutorial is published in Chinese. I am sharing the essential probe here so it can be reviewed without following a link. The goal is to identify the last checkpoint each rank reaches: entering process-group initialization, initialization returned, or the first all_reduce completed. This helps narrow an investigation; it does not identify the root cause by itself. This version passes device_id to init_process_group to initialize the NCCL communicat...
0 comments
No comments yet.