Nemotron 3.5 Bolt E4B

Methodology

• Throughput: one user, real text from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token; decode is the output tokens after the first over the time they take. Decode, 3,072 in / 50 out: no cached context, 3,072 tokens of new input, exactly 50 out (end of sequence ignored), the same text, 10 requests after 2 warm-up requests, decode as above. For reference, NVIDIA's official release reports 24.0 tok/s decode on a Jetson AGX Orin (32GB) with no cached context, 3K in and 50 out, with its Q4_K_M weights on llama.cpp (CUDA), measured with AIPerf at batch 1.
• Accuracy: NVIDIA NeMo Gym (commit e6097044), its own data and scoring, greedy, one answer per item, reasoning on (off for RULER); answers up to 3,500 tokens (MMLU-Pro and MMLU-Redux 7,000, RAGTruth 4,096). Each set is sampled evenly over its items (GSM8K: 1,317 of its 1,319 test questions; IFEval: all 541; RULER: 260 per length over its 13 tasks; IHEval: 300 of a 600-item sample). Measured on an NVIDIA H200 with a 65,536-token context, from the same quantized weights and output layer as this build: Jetson GPUs and the H200 take the INT4 weights in different layouts, so the checkpoint is exported once for each. On this board's context, prompts and answers longer than it do not fit.
• Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens.
• Size: the model image's files: engines, tokenizer, serve script and licences.
• Baseline: NVIDIA's BF16 release (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16), unquantized, as an FP16 TensorRT Edge-LLM engine. Throughput, power and size: measured on this board with the same benchmark, settings and inputs. Accuracy: measured on the same H200 as this build's, through the same harness on the same items.