Ling-3.1-flash and the 7:1 hybrid: how much KV cache does a 560B model need at 1M tokens?
DEV Community
Ling-3.1-flash and the 7:1 hybrid: how much KV cache does a 560B model need at 1M tokens?
A 560B open model keeps full attention in only 9 of 72 layers. Here is the cache and serving math for 1M tokens, computed from its public config.
0 comments
No comments yet.