≈ the.bay.news

Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050

DEV Community
Six Serving Engines, One 8 GB GPU: Gemma 4 E2B on an RTX 3050
llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 8 GB RTX 3050, measured with the same prompts, the same load sweep and the same GSM8K and BFCL questions. GSM8K is level on every engine; vLLM is the fastest up to 16 requests and within 3% of SGLang beyond.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.