Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU
DEV Community
Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU
llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 16 GB Tesla T4 on Compute Engine, with the same prompts, load sweep and GSM8K and BFCL questions. GSM8K and BFCL are level on every CUDA engine; vLLM with its drafter leads at every load.
0 comments
No comments yet.