≈ the.bay.news

Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU

DEV Community
Six Serving Engines on One Tesla T4: Gemma 4 E2B on a Turing GPU
llama.cpp, vLLM, SGLang, mistral.rs, a plain PyTorch engine and LiteRT-LM serve the same 4-bit Gemma 4 E2B on one 16 GB Tesla T4 on Compute Engine, with the same prompts, load sweep and GSM8K and BFCL questions. GSM8K and BFCL are level on every CUDA engine; vLLM with its drafter leads at every load.

0 comments

Sign in to join the discussion — your thebay.events account works here.

No comments yet.