Embedl-optimized · TensorRT Edge-LLM
- huggingface
- Orin Nano Super
- Labels
- Image
- Text
- VLM
- Generation
- Build
- Embedl-optimized
- Precision
- W4A16
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 4,096 tokens
Alibaba's 0.8B Qwen3.5 model for chat, reasoning and long context, with images.
Methodology
- Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request. Prompt processing is the 2,048 prompt tokens over the time to first token. Text prompts only.
- Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the device's.
- Accuracy: full test sets, greedy, the same model files and settings as served. GSM8K 1319, IFEval 541, MMLU-Pro 1008 questions.
- Size: the runtime container image, compressed as downloaded and unpacked on disk, and the model's files; total size is the model's files plus the container download.
- Baseline TensorRT Edge-LLM: the original FP16 model on upstream TensorRT Edge-LLM, same settings; its accuracy measured on the Jetson AGX Thor.