Embedl-optimized
- huggingface
- Orin NX 16GB
- Labels
- Text
- LLM
- Generation
- Build
- Embedl-optimized
- Precision
- W4A16
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 4,096 tokens
Alibaba's mixture-of-experts Qwen3.6: 35B parameters, about 3B active per token, for coding and agent work. Text in, text out.
Methodology
- Throughput: one user, real text from the public OpenAssistant set (no chat template), greedy, no prompt reuse. The headline decode is 3,072 tokens in and 50 out, 10 requests after 2 warm-up requests. End-to-end, prompt processing and the second decode are 2,048 tokens in and 128 out, 20 requests after 3 warm-up requests; prompt processing is the prompt tokens over the time to first token.
- Accuracy: greedy, the build as served (its own embedl-serve), the answer read from its last "Answer:" line, at most 2,048 tokens; an answer cut off at that limit counts as wrong. GSM8K: the first 100 questions of the test split. MMLU-Pro: 56 questions of the test split, 4 per subject (of 72 drawn per subject with seed 0).
- Power and memory: sampled on the device during the 2,048-in runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the model process's own at its highest: its GPU buffers, its own allocations and its share of the model files it holds in memory.
- Size: the model image's files: engines, tokenizer, serve script and licences.
- Baseline: none on this board: the 16-bit model as released (BF16, 67.0 GiB of weights) does not fit in the Orin NX's 16 GB. The headline is those 67.0 GiB over this build's 10.6 GiB peak memory.