Embedl's build nvidia/cosmos3-edge of the original model nvidia/Cosmos3-Edge
Embedl-optimized
- nvidia/Cosmos3-Edge
- INT4
- Orin Nano Super
- Labels
- Image
- Text
- VLM
- Generation
- Build
- Embedl-optimized
- Precision
- INT4 weights
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 4,096 tokens
NVIDIA's Cosmos3 Edge reasoner: the 2.4B text-and-image-in, text-out part of NVIDIA's open world model for on-device physical AI.
Methodology
- Throughput: one user, batch 1, greedy, a 1,705-token text prompt in, 128 tokens out, with the build's EAGLE3 drafting (top-k 3, 4 steps, verify tree 16), in MAXN_SUPER. End-to-end is the whole request; prompt processing is the 1,705 prompt tokens over the time to first token (651 ms); decode excludes prefill. With the same engines a held-out 1,705-token text prompt decodes at 111.5 tok/s and a 1,690-token image prompt at 171.3 tok/s.
- Accuracy: greedy, the same INT4 model: MMMU-Pro, AI2D, MMBench and DocVQA 1,500 questions each, RealWorldQA all 765.
- Memory: peak GPU memory during the throughput run.
- Size: the runtime container (download and on disk), the model image, and its TensorRT engines.
- Baseline TensorRT Edge-LLM: the FP16 model on upstream TensorRT Edge-LLM with plain decode, same device and prompt: 26.8 tok/s decode; its end-to-end throughput was not recorded.
- Vision-language model: throughput measured on text prompts only.