Why the KV cache, not the weights, decides how many users an LLM server can hold

A 7B model in fp16 takes 13.5 GB of GPU memory for its weights. On an 80 GB card that leaves about 66...