Skip to content
Search models or paste a model link. Press Enter to search.

Nemotron 3.5 Bolt E4B

nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16
Left and right arrow keys switch the vendor.
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 architecture graph
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 architecture graph
  • huggingface
  • AGX Thor
Labels
  • Text
  • LLM
  • Generation
Build
Baseline FP16 (NVIDIA BF16 weights)
Precision
FP16
Runtime
TensorRT Edge-LLM 0.10.1
Context length
4,096 tokens
Run Nemotron 3.5 Bolt E4B on AGX Thor

NVIDIA's 4B small language model for fast local assistants, post-trained from Google's Gemma 4 E4B for tool use, instruction following and grounding. Text in, text out.

Licence

The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The model image carries the licences and a NOTICE.

Methodology

  • Throughput: one user, real text from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token; decode is the output tokens after the first over the time they take.
  • Accuracy: NVIDIA NeMo Gym (commit e6097044), its own data and scoring, on this board, with the model card's settings: reasoning on (off for RULER), temperature 1.0, top_p 0.95, the runtime's fixed seed, one answer per item. IFBench: 300 and RAGTruth 500 items spread evenly over each set; LongBench v2: 154; RULER: 260 per length over its 13 tasks. Measured on the 65,536-token build of the same FP16 export.
  • Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens.
  • Size: the model image's files: engines, tokenizer, serve script and licences.
  • Baseline: NVIDIA's BF16 release (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16), unquantized, as an FP16 TensorRT Edge-LLM engine. This build is the baseline the Embedl-optimized build on this board is compared with.

Files

Image Size

Model

docker.io/embedl/models:hf.nvidia.nemotron-3-5-bolt-e4b.fp16.thor.202610291004f60f

15.25 GiB

Runtime

docker.io/embedl/models:runtimes.tensorrt-edgellm.arm64.2026102021218b19

—

Variants

Every build of Nemotron 3.5 Bolt E4B on AGX Thor.

Variant Precision Speed Energy / token Runtime
W4A16 80.7 tok/s
Throughput: Decode
1.04 J
TensorRT Edge-LLM 0.10.1 with Embedl optimizations
runtimes.tensorrt-edgellm.arm64.202610302304ff6e

huggingface_fp16_tensorrt-edgellm

FP16 24.9 tok/s
Throughput: Decode
2.80 J
TensorRT Edge-LLM 0.10.1
runtimes.tensorrt-edgellm.arm64.2026102021218b19