โ‰ˆ the.bay.news

Ling-3.1-flash and the 7:1 hybrid: how much KV cache does a 560B model need at 1M tokens?

DEV Community
Ling-3.1-flash and the 7:1 hybrid: how much KV cache does a 560B model need at 1M tokens?
A 560B open model keeps full attention in only 9 of 72 layers. Here is the cache and serving math for 1M tokens, computed from its public config.

0 comments

Sign in to join the discussion โ€” your thebay.events account works here.

No comments yet.