Skip to content
Search models or paste a model link. Press Enter to search.

Nemotron 3.5 Bolt E4B

nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16
Left and right arrow keys switch the vendor.
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 architecture graph
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 architecture graph
  • huggingface
  • AGX Thor
Labels
  • Text
  • LLM
  • Generation
Build
Embedl-optimized
Precision
W4A16
Runtime
TensorRT Edge-LLM 0.10.1 with Embedl optimizations
Context length
4,096 tokens
Run Nemotron 3.5 Bolt E4B on AGX Thor

NVIDIA's 4B small language model for fast local assistants, post-trained from Google's Gemma 4 E4B for tool use, instruction following and grounding. Text in, text out.

Licence

The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The model image carries the licences and a NOTICE.

Methodology

  • Throughput: one user, real text from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token; decode is the output tokens after the first over the time they take. For reference, NVIDIA's official release publishes no Jetson AGX Thor figure; on an RTX Spark (32GB) it reports 67.0 tok/s decode with no cached context, 3K in and 50 out, with its Q4_K_M weights on llama.cpp (CUDA), measured with AIPerf at batch 1. The figure on this page is at 2,048 in and 128 out.
  • Accuracy: NVIDIA NeMo Gym (commit e6097044), its own data and scoring, greedy, one answer per item, reasoning on (off for RULER); answers up to 3,500 tokens (MMLU-Pro and MMLU-Redux 7,000, RAGTruth 4,096). Each set is sampled evenly over its items (GSM8K: 1,317 of its 1,319 test questions; IFEval: all 541; RULER: 260 per length over its 13 tasks; IHEval: 300 of a 600-item sample). Measured on an NVIDIA H200 with a 65,536-token context, from the same quantized weights and output layer as this build: Jetson GPUs and the H200 take the INT4 weights in different layouts, so the checkpoint is exported once for each. On this board's context, prompts and answers longer than it do not fit.
  • Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens.
  • Size: the model image's files: engines, tokenizer, serve script and licences.
  • Baseline: NVIDIA's BF16 release (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16), unquantized, as an FP16 TensorRT Edge-LLM engine. Throughput, power and size: measured on this board with the same benchmark, settings and inputs. Accuracy: measured on the same H200 as this build's, through the same harness on the same items.

Files

Image Size

Model

docker.io/embedl/models:hf.nvidia.nemotron-3-5-bolt-e4b.int4-embedl-flashhead-always1k.thor.2026110307432ffe

8.62 GiB

Runtime

docker.io/embedl/models:runtimes.tensorrt-edgellm.arm64.202610302304ff6e

—

Variants

Every build of Nemotron 3.5 Bolt E4B on AGX Thor.

Variant Precision Speed Energy / token Runtime
W4A16 80.7 tok/s
Throughput: Decode
1.04 J
TensorRT Edge-LLM 0.10.1 with Embedl optimizations
runtimes.tensorrt-edgellm.arm64.202610302304ff6e

huggingface_fp16_tensorrt-edgellm

FP16 24.9 tok/s
Throughput: Decode
2.80 J
TensorRT Edge-LLM 0.10.1
runtimes.tensorrt-edgellm.arm64.2026102021218b19