โ‰ˆ the.bay.news

Serving Groot-19B-A3B at 256K on one RTX 5090: the memory budget and the thinking trap

DEV Community
Serving Groot-19B-A3B at 256K on one RTX 5090: the memory budget and the thinking trap
Groot-19B-A3B ran its full 262,144-token window on one 32 GB RTX 5090 with vLLM. A 24 GB cap left room for 9,648 tokens. What to check.

0 comments

Sign in to join the discussion โ€” your thebay.events account works here.

No comments yet.