Embedl optimized · TensorRT Edge-LLM
- huggingface
- Orin Nano Super
- Labels
- Text
- LLM
- Generation
- Build
- Embedl INT4 + FlashHead + MTP
- Precision
- W4A16
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 4,096 tokens
Embedl's INT4 build of NVIDIA's Nemotron 3.5 Bolt E4B with FlashHead and MTP speculative decoding, on TensorRT Edge-LLM, running in the Orin Nano's 8 GB. Against NVIDIA's published llama.cpp numbers for the Orin Nano: first token 1.6× sooner and 2.5× the decode speed at 3K tokens. MMLU-Pro is about 8 points below Embedl's plain INT4 build on the larger Jetsons.
Run it
The images are internal: docker login with access to the embedl organisation first. embedl serve runs exactly this.
Licence
The model is NVIDIA's under the OpenMDW License Agreement 1.1, post-trained from Google's Gemma 4 E4B under the Apache License 2.0. The MTP draft is Google's Gemma 4 E4B assistant, Apache 2.0. The model image carries the licences and a NOTICE.
Methodology
- Throughput: one user, real text from the public OpenAssistant set (without the chat template, which needs more memory than the device has beside the model), 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up. Prompt processing is the 2,048 prompt tokens over the time to first token. Speculative decoding (MTP): 2.38 tokens per step.
- Power: the module's rails, sampled during those requests; idle is the device at rest with the model loaded.
- Accuracy: greedy, the build as served, the answer read from its last "Answer:" line, at most 1,024 tokens. GSM8K: the first 330 test questions. MMLU-Pro: 252 questions, 18 of each of its 14 subjects.
- Against NVIDIA: NVIDIA's numbers are from research.nvidia.com/labs/nemotron/Nemotron-3.5-Bolt-E4B-4GB (llama.cpp CUDA, Q4_K_M, AIPerf, batch 1). Ours: the same token counts on this device, real OpenAssistant text, exactly that many output tokens, the cached context reused through Edge-LLM's context cache. Time to first token is the mean; tokens per second is the decode rate after the first token. 3K is read as 3,072 and 32K as 32,768 tokens.
- Device: an Orin Nano Super developer kit (8 GB). NVIDIA's table lists the Orin Nano 8GB, the same module; its power mode isn't stated.
- Memory: the model fits because Embedl's runtime keeps the two embedding tables (5.6 GB and 0.67 GB) memory-mapped on the host and reads only the rows each step needs.
- Not on this device: the 32K-context row (the KV cache for 32K tokens does not fit beside the model in 8 GB) and the plain INT4 build (its full output layer does not compile in 8 GB).