Nemotron 3.5 Bolt E4B
nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16 Left and right arrow keys switch the vendor.
- huggingface
- AGX Thor
- Labels
- Text
- LLM
- Generation
- Build
- Baseline FP16 (NVIDIA BF16 weights)
- Precision
- FP16
- Runtime
- TensorRT Edge-LLM 0.10.1
- Context length
- 4,096 tokens
NVIDIA's 4B small language model for fast local assistants, post-trained from Google's Gemma 4 E4B for tool use, instruction following and grounding. Text in, text out.
Licence
The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The model image carries the licences and a NOTICE.
Methodology
- Throughput: one user, real text from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token; decode is the output tokens after the first over the time they take.
- Accuracy: NVIDIA NeMo Gym (commit e6097044), its own data and scoring, on this board, with the model card's settings: reasoning on (off for RULER), temperature 1.0, top_p 0.95, the runtime's fixed seed, one answer per item. IFBench: 300 and RAGTruth 500 items spread evenly over each set; LongBench v2: 154; RULER: 260 per length over its 13 tasks. Measured on the 65,536-token build of the same FP16 export.
- Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens.
- Size: the model image's files: engines, tokenizer, serve script and licences.
- Baseline: NVIDIA's BF16 release (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16), unquantized, as an FP16 TensorRT Edge-LLM engine. This build is the baseline the Embedl-optimized build on this board is compared with.
Files
Variants
Every build of Nemotron 3.5 Bolt E4B on AGX Thor.