nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16

Labels

Build

Embedl INT4

Precision

W4A16

Runtime

TensorRT Edge-LLM 0.10.1 with Embedl optimizations

Context length

4,096 tokens

Embedl's INT4 build of NVIDIA's Nemotron 3.5 Bolt E4B on TensorRT Edge-LLM: the accuracy reference for the FlashHead + MTP build.

Run it

The images are internal: docker login with access to the embedl organisation first. embedl serve runs exactly this.

The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The model image carries the licences and a NOTICE.

Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up. Prompt processing is the 2,048 prompt tokens over the time to first token.
Power: the module's rails, sampled during those requests; idle is the device at rest with the model loaded.
Accuracy: greedy, the build as served, the answer read from its last "Answer:" line, at most 1,024 tokens. GSM8K: all 1,319 test questions. MMLU-Pro: 1,008 questions, 72 of each of its 14 subjects.

Image	Size
Model docker.io/embedl/models:hf.nvidia.nemotron-3-5-bolt-e4b.int4-embedl.thor.2026102421001855	—
Runtime docker.io/embedl/models:runtimes.tensorrt-edgellm.arm64.20261025202937ff	—

Every build of Nemotron 3.5 Bolt E4B on AGX Thor.

Variant	Precision	Speed	Energy / token	Runtime
Embedl optimized · TensorRT Edge-LLM	W4A16	85.6 tok/s Throughput: End-to-end	0.83 J	TensorRT Edge-LLM 0.10.1 with Embedl optimizations runtimes.tensorrt-edgellm.arm64.20261025202937ff
Embedl optimized · TensorRT Edge-LLM This page huggingface_int4_embedl_tensorrt-edgellm	W4A16	51.4 tok/s Throughput: End-to-end	1.38 J	TensorRT Edge-LLM 0.10.1 with Embedl optimizations runtimes.tensorrt-edgellm.arm64.20261025202937ff