Embedl-optimized · TensorRT Edge-LLM
- huggingface
- Orin NX 16GB
- Labels
- Image
- Text
- VLM
- Generation
- Build
- Embedl-optimized
- Precision
- W4A16
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 4,096 tokens
Alibaba's 4B Qwen3-VL vision-language model for describing, reading and reasoning about images and video.
Methodology
- Throughput: one user, synthetic prompts (random tokens, as in NVIDIA Jetson AI Lab's benchmarks), 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 50 requests after 3 warm-up requests. End-to-end is the whole request. Prompt processing is the 2,048 prompt tokens over the time to first token. Text prompts only. Measured on an AGX Orin flashed as an Orin NX 16GB (NVIDIA's emulation), power mode MAXN, not MAXN_SUPER.
- Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the device's.
- Accuracy: MMLU-mini, 200 MMLU questions, 0-shot, greedy, run on the device. Not comparable with published 5-shot MMLU scores.
- Size: the runtime container image, compressed as downloaded and unpacked on disk, and the model's files; total size is the model's files plus the container download.