Skip to content
Search models or paste a model link. Press Enter to search.
Log in

Llama 3.2 3B

meta-llama/Llama-3.2-3B-Instruct
Embedl-optimized · FlashHead · TensorRT Edge-LLM
meta-llama/Llama-3.2-3B-Instruct architecture graph
meta-llama/Llama-3.2-3B-Instruct architecture graph
  • huggingface
  • AGX Orin
Labels
  • Text
  • LLM
  • Generation
Build
Embedl-optimized · FlashHead
Precision
W4A16
Runtime
TensorRT Edge-LLM 0.10.1 with Embedl optimizations
Context length
4,096 tokens
Run Llama 3.2 3B on AGX Orin

Meta's 3B Llama 3.2 instruction model for summarizing, rewriting and simple agents.

Methodology

  • Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse. Average of 20 requests after 3 warm-up requests. End-to-end is the whole request. Prompt processing is the 2,048 prompt tokens over the time to first token.
  • Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the device's.
  • Accuracy: full test sets, greedy, the same model files and settings as served. GSM8K 1319, IFEval 541, MMLU-Pro 1008 questions.
  • Baseline TensorRT Edge-LLM: the original FP16 model on upstream TensorRT Edge-LLM, same settings, same device.

Files

Image Size

Model

docker.io/embedl/model-artifacts:llama3-2-3b.huggingface_int4_embedl-flashhead_tensorrt-edgellm.4bf7cb50adf9fb26

 2.91 GiB

Runtime

docker.io/embedl/models:tensorrt-edgellm_arm64_2c081bac0ba4e49edf6a8b0949553a71e6219e35

 0.92 GiB