Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
DEV Community
Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 8 GB RTX 3050, measured with the same prompts, the same load sweep and the same GSM8K and BFCL questions. GSM8K is level on every engine; vLLM is the fastest up to 16 requests and within 3% of SGLang beyond.
0 comments
No comments yet.