Qwen/Qwen3-VL-8B-Instruct

Labels

Build

Embedl-optimized

Precision

W4A16

Runtime

TensorRT Edge-LLM 0.10.1 with Embedl optimizations

Context length

4,096 tokens

Alibaba's 8B Qwen3-VL vision-language model for describing, reading and reasoning about images and video.

Methodology

Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request. Prompt processing is the 2,048 prompt tokens over the time to first token.
Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the device's.
Accuracy: full test sets, greedy, the same model files and settings as served. GSM8K 1319, IFEval 541, MMLU-Pro 1008 questions.
Baseline TensorRT Edge-LLM: the original FP16 model on upstream TensorRT Edge-LLM, same settings, same device.
Vision-language model: measured on text only.

Image	Size
Model docker.io/embedl/model-artifacts:qwen3-vl-8b.huggingface_int4_embedl_tensorrt-edgellm.df2f4d2d4e3f6774	6.78 GiB
Runtime docker.io/embedl/models:runtimes.tensorrt-edgellm.arm64.2026101815014ee0	1.01 GiB

2.47× faster

than Baseline TensorRT Edge-LLM

37.8 tok/s

2.47× faster

Baseline TensorRT Edge-LLM: 15.3 tok/s

81.3 %

Baseline TensorRT Edge-LLM: 82.7 %

1.89 J

2.20× lower

Baseline TensorRT Edge-LLM: 4.16 J

7.79 GiB

Every build of Qwen3-VL 8B on AGX Thor.

Variant	Precision	Speed	Peak memory	Energy / token	Runtime
Baseline TensorRT Edge-LLM	FP16	15.3 tok/s	18.97 GiB	4.16 J	TensorRT Edge-LLM 0.10.1
Embedl optimized · TensorRT Edge-LLM This page	W4A16	37.8 tok/s	9.05 GiB	1.89 J	TensorRT Edge-LLM 0.10.1 with Embedl optimizations runtimes.tensorrt-edgellm.arm64.2026101815014ee0