Embedl optimized · TensorRT Edge-LLM
- huggingface
- H200
- Labels
- Text
- LLM
- Generation
- Build
- Embedl-optimized
- Precision
- W4A16
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 8,192 tokens
Recursive's Flint: a 40B mixture-of-experts coding model, about 2.5B parameters active per token, with reasoning and tool use.
Methodology
- Throughput: one user, 2,048 tokens in, 128 out, greedy, no prompt reuse; 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token. The baseline is the BF16 model served by Flint's own SGLang kit on the same GPU, same lengths.
- Accuracy: 200 MMLU-mini and 100 GSM8K questions, one sample each, Flint's recommended sampling (temperature 1.0, top-p 0.95, top-k 20), up to 8,192 tokens, as served. The baseline is the BF16 model on the same questions, four samples each.