How does PyTorch find every gradient by itself?

Autograd, step by step: PyTorch records the forward pass as a graph, then replays it backward