Stop Guessing Your Context Budget: Practical KV Cache Sizing for Local LLMs
DEV Community
Stop Guessing Your Context Budget: Practical KV Cache Sizing for Local LLMs
KV cache is why a 32K context costs gigabytes of VRAM. The exact arithmetic, the llama.cpp and Ollama settings that control it, why q8_0 is not half, and how to test quality before you commit.
0 comments
No comments yet.