Qwen3 0.6B

Qwen/Qwen3-0.6B

Labels

Build

Embedl-optimized · FlashHead

Precision

W4A16

Runtime

TensorRT Edge-LLM 0.10.1 with Embedl optimizations

Context length

4,096 tokens

Alibaba's 0.6B Qwen3 model for chat, math and code, with optional step-by-step thinking.

Methodology

Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request. Prompt processing is the 2,048 prompt tokens over the time to first token.
Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the device's.
Accuracy: full test sets, greedy, the same model files and settings as served. GSM8K 1319, IFEval 541, MMLU-Pro 1008 questions.
Baseline TensorRT Edge-LLM: the original FP16 model on upstream TensorRT Edge-LLM, same settings, same device.

Image	Size
Model docker.io/embedl/model-artifacts:qwen3-0-6b.huggingface_int4_embedl-flashhead_tensorrt-edgellm.47f424a7c5d33e15	0.84 GiB
Runtime docker.io/embedl/models:tensorrt-edgellm_arm64_2c081bac0ba4e49edf6a8b0949553a71e6219e35	0.92 GiB

1.63× faster

than Baseline TensorRT Edge-LLM

155.3 tok/s

1.63× faster

Baseline TensorRT Edge-LLM: 95.1 tok/s

46.3 %

Baseline TensorRT Edge-LLM: 49.8 %

0.21 J

1.86× lower

Baseline TensorRT Edge-LLM: 0.39 J

1.76 GiB

Every build of Qwen3 0.6B on AGX Orin.

Variant	Precision	Speed	Peak memory	Energy / token	Runtime
Baseline TensorRT Edge-LLM huggingface_fp16_tensorrt-edgellm	FP16	95.1 tok/s Throughput: End-to-end	1.68 GiB	0.39 J	TensorRT Edge-LLM 0.10.1
Embedl-optimized · FlashHead · TensorRT Edge-LLM This page huggingface_int4_embedl-flashhead_tensorrt-edgellm	W4A16	155.3 tok/s Throughput: End-to-end	0.81 GiB	0.21 J	TensorRT Edge-LLM 0.10.1 with Embedl optimizations tensorrt-edgellm_arm64_2c081bac0ba4e49edf6a8b0949553a71e6219e35