≈ the.bay.news

Halving the KV Cache Doubles Capacity and Buys 1% of Speed

DEV Community
Halving the KV Cache Doubles Capacity and Buys 1% of Speed
A reader found a bug in the script and asked whether an FP8 KV cache halves the long-context penalty. Computed: capacity is exactly 2x at every context, speed is 1% at 4k and 21% at 128k, and the crossover is where the cache read equals the weight read.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.