Embedl optimized · TensorRT Edge-LLM
- huggingface
- AGX Thor
- Labels
- Text
- LLM
- Generation
- Build
- Embedl-optimized
- Precision
- W4A16
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 8,192 tokens
Recursive's Flint: a 40B mixture-of-experts coding model, about 2.5B parameters active per token, with reasoning and tool use.
Methodology
- Throughput: one user, real conversations from the public OpenAssistant set, 2,048 tokens in, 128 out, greedy, no prompt reuse; 20 requests after 3 warm-up requests, the Thor at MAXN. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token.
- Power and memory: sampled on the device during those runs. Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens; peak memory is the drop in the device's available memory from before the model loaded.
- Accuracy: 200 MMLU-mini and 100 GSM8K questions, one sample each, Flint's recommended sampling (temperature 1.0, top-p 0.95, top-k 20), up to 8,192 tokens, as served. The baseline is the BF16 model served by Flint's own SGLang kit (on an H200) on the same questions, four samples each.