Distributed Training & Inference: From CPUs and GPUs to a Cluster

An illustrated guide to distributed training: CPUs vs GPUs, CUDA, data, tensor and pipeline parallelism, FSDP vs DeepSpeed ZeRO, Ray, and inference.