A point release halved your KV cache memory. You did not choose that trade.
DEV Community
A point release halved your KV cache memory. You did not choose that trade.
Two vLLM point releases changed serving defaults in a minor version bump - FP8 KV cache by default, and max_tokens no longer reserved from the context window. A default you did not choose is still a decision you own.
0 comments
No comments yet.