Serving Groot-19B-A3B at 256K on one RTX 5090: the memory budget and the thinking trap
DEV Community
Serving Groot-19B-A3B at 256K on one RTX 5090: the memory budget and the thinking trap
Groot-19B-A3B ran its full 262,144-token window on one 32 GB RTX 5090 with vLLM. A 24 GB cap left room for 9,648 tokens. What to check.
0 comments
No comments yet.