Reserve KV Cache for the Output You Promised, Not Just the Prompt You Received
DEV Community
Reserve KV Cache for the Output You Promised, Not Just the Prompt You Received
Admission control that only measures the prompt lets short requests with big generation caps OOM your LLM server. Reserve for output tokens.
0 comments
No comments yet.