Skip to content
Search models or paste a model link. Press Enter to search.

Nemotron 3.5 Bolt E4B

nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16
Left and right arrow keys switch the vendor.
Embedl-optimized · TensorRT Edge-LLM
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 architecture graph
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 architecture graph
  • huggingface
  • Orin NX 16GB
Labels
  • Text
  • LLM
  • Generation
Build
Embedl-optimized
Precision
W4A16
Runtime
TensorRT Edge-LLM 0.10.1 with Embedl optimizations
Context length
4,096 tokens
Run Nemotron 3.5 Bolt E4B on Orin NX 16GB

NVIDIA's 4B small language model for fast local assistants, post-trained from Google's Gemma 4 E4B for tool use, instruction following and grounding. Text in, text out.

Licence

The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The model image carries the licences and a NOTICE.

Methodology

  • Throughput: one user, real text from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token; decode is the output tokens after the first over the time they take. Decode, 3,072 in / 50 out: no cached context, 3,072 tokens of new input, exactly 50 out (end of sequence ignored), the same text, 10 requests after 2 warm-up requests, decode as above.
  • Accuracy: NVIDIA NeMo Gym (commit e6097044), its own data and scoring, greedy, one answer per item, reasoning on (off for RULER); answers up to 3,500 tokens (MMLU-Pro and MMLU-Redux 7,000, RAGTruth 4,096). Each set is sampled evenly over its items (GSM8K: 1,317 of its 1,319 test questions; IFEval: all 541; RULER: 260 per length over its 13 tasks; IHEval: 300 of a 600-item sample). Measured on an NVIDIA H200 with a 65,536-token context, from the same quantized weights and output layer as this build: Jetson GPUs and the H200 take the INT4 weights in different layouts, so the checkpoint is exported once for each. On this board's context, prompts and answers longer than it do not fit.
  • Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens.
  • Size: the model image's files: engines, tokenizer, serve script and licences.
  • Baseline: NVIDIA's BF16 release (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16), unquantized, as an FP16 TensorRT Edge-LLM engine. Its accuracy is measured on the same H200, through the same harness on the same items. It does not run on this board: its model image is 15.25 GiB, and on an AGX Thor it peaks at 24.2 GiB of device memory, more than the 16 GB this board has.

Files

Image Size

Model

docker.io/embedl/models:hf.nvidia.nemotron-3-5-bolt-e4b.int4-embedl-flashhead-always1k.orin-nx-16gb.2026110307430269

8.61 GiB

Runtime

docker.io/embedl/models:runtimes.tensorrt-edgellm.arm64.202610302304ff6e

—