- huggingface
- AGX Thor
- Labels
- Text
- LLM
- Generation
- Build
- Embedl INT4 + FlashHead + MTP
- Precision
- W4A16
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 4,096 tokens
Embedl's INT4 build of NVIDIA's Nemotron 3.5 Bolt E4B with FlashHead and MTP speculative decoding, on TensorRT Edge-LLM: 117 tokens a second decode, 1.9× the plain INT4 build, with MMLU-Pro about 8 points lower.
Run it
The images are internal: docker login with access to the embedl organisation first. embedl serve runs exactly this.
Licence
The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The MTP draft is Google's Gemma 4 E4B assistant, Apache 2.0. The model image carries the licences and a NOTICE.
Methodology
- Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up. Prompt processing is the 2,048 prompt tokens over the time to first token. Speculative decoding (MTP): 2.06 tokens per step.
- Power: the module's rails, sampled during those requests; idle is the device at rest with the model loaded.
- Accuracy: greedy, the build as served, the answer read from its last "Answer:" line, at most 1,024 tokens. GSM8K: all 1,319 test questions. MMLU-Pro: 1,008 questions, 72 of each of its 14 subjects.
Files
Variants
Every build of Nemotron 3.5 Bolt E4B on AGX Thor.