- huggingface
- AGX Orin
- Labels
- Text
- LLM
- Generation
- Build
- Baseline FP16 (NVIDIA BF16 weights)
- Precision
- FP16
- Runtime
- TensorRT Edge-LLM 0.10.1
- Context length
- 4,096 tokens
NVIDIA's 4B small language model for fast local assistants, post-trained from Google's Gemma 4 E4B for tool use, instruction following and grounding. Text in, text out.
Licence
The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The model image carries the licences and a NOTICE.
Methodology
- Throughput: one user, real text from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token; decode is the output tokens after the first over the time they take.
- Accuracy: NVIDIA NeMo Gym (commit e6097044), its own data and scoring, greedy, one answer per item, reasoning on (off for RULER); answers up to 3,500 tokens (MMLU-Pro and MMLU-Redux 7,000, RAGTruth 4,096). Each set is sampled evenly over its items (GSM8K: 1,317 of its 1,319 test questions; IFEval: all 541; RULER: 260 per length over its 13 tasks; IHEval: 300 of a 600-item sample). Measured on an NVIDIA H200 with a 65,536-token context, from the same FP16 export of NVIDIA's weights, compiled for the H200. On this board's context, prompts and answers longer than it do not fit.
- Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens.
- Size: the model image's files: engines, tokenizer, serve script and licences.
- Baseline: NVIDIA's BF16 release (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16), unquantized, as an FP16 TensorRT Edge-LLM engine. This build is the baseline the Embedl-optimized build on this board is compared with.
Files
Variants
Every build of Nemotron 3.5 Bolt E4B on AGX Orin.