Why Split Prefill and Decode GPUs Lose Time in the Hand-Off, Not the Network

Splitting an LLM server into prompt GPUs and generation GPUs means handing every request's cache across. The bytes take milliseconds. The setup work on the generation server's loop costs far more, because it slows every request and can run the server