Qwen/Qwen3.5-9B

Labels

Build

Embedl-optimized

Precision

W4A16

Runtime

TensorRT Edge-LLM 0.10.1 with Embedl optimizations

Context length

4,096 tokens

Alibaba's 9B Qwen3.5 model for chat, reasoning and long context, with images.

Methodology

Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request. Prompt processing is the 2,048 prompt tokens over the time to first token.
Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the device's.
Accuracy: full test sets, greedy, the same model files and settings as served. GSM8K 1319, IFEval 541, MMLU-Pro 1008 questions.
Baseline TensorRT Edge-LLM: the original FP16 model on upstream TensorRT Edge-LLM, same settings, same device.
Vision-language model: measured on text only.

Image	Size
Model docker.io/embedl/model-artifacts:qwen3-5-9b.huggingface_int4_embedl-mtp_tensorrt-edgellm.afba33d403358399	10.05 GiB
Runtime docker.io/embedl/models:runtimes.tensorrt-edgellm.arm64.2026101815014ee0	1.01 GiB

2.52× faster

than Baseline TensorRT Edge-LLM

38.0 tok/s

2.52× faster

Baseline TensorRT Edge-LLM: 15.1 tok/s

83.9 %

Baseline TensorRT Edge-LLM: 85.1 %

2.07 J

2.18× lower

Baseline TensorRT Edge-LLM: 4.51 J

11.06 GiB

Every build of Qwen3.5 9B on AGX Thor.

Variant	Precision	Speed	Peak memory	Energy / token	Runtime
Baseline TensorRT Edge-LLM	FP16	15.1 tok/s	20.06 GiB	4.51 J	TensorRT Edge-LLM 0.10.1
Embedl optimized · TensorRT Edge-LLM This page	W4A16	38.0 tok/s	12.46 GiB	2.07 J	TensorRT Edge-LLM 0.10.1 with Embedl optimizations runtimes.tensorrt-edgellm.arm64.2026101815014ee0