google/gemma-4-E2B-it

Labels

Build

Embedl-optimized

Precision

W4A16

Runtime

TensorRT Edge-LLM 0.10.1 with Embedl optimizations

Context length

4,096 tokens

The smallest Gemma 4 model, with about 2B effective parameters, for on-device use, with images.

Methodology

Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request. Prompt processing is the 2,048 prompt tokens over the time to first token.
Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the device's.
Accuracy: full test sets, greedy, the same model files and settings as served. GSM8K 1319, IFEval 541, MMLU-Pro 1008 questions.
Baseline TensorRT Edge-LLM: the original FP16 model on upstream TensorRT Edge-LLM, same settings, same device.
Vision-language model: measured on text only.

Image	Size
Model docker.io/embedl/model-artifacts:gemma4-e2b.huggingface_int4_embedl-mtp_tensorrt-edgellm.b01f8a285fb11843	7.37 GiB
Runtime docker.io/embedl/models:runtimes.tensorrt-edgellm.arm64.2026101815014ee0	1.01 GiB

2.57× faster

than Baseline TensorRT Edge-LLM

120.0 tok/s

2.57× faster

Baseline TensorRT Edge-LLM: 46.7 tok/s

74.9 %

Baseline TensorRT Edge-LLM: 76.6 %

0.55 J

2.33× lower

Baseline TensorRT Edge-LLM: 1.28 J

8.38 GiB

Every build of Gemma 4 E2B on AGX Thor.

Variant	Precision	Speed	Peak memory	Energy / token	Runtime
Baseline TensorRT Edge-LLM	FP16	46.7 tok/s	12.35 GiB	1.28 J	TensorRT Edge-LLM 0.10.1
Embedl optimized · TensorRT Edge-LLM This page	W4A16	120.0 tok/s	10.03 GiB	0.55 J	TensorRT Edge-LLM 0.10.1 with Embedl optimizations runtimes.tensorrt-edgellm.arm64.2026101815014ee0