Embedl's build nvidia/cosmos3-edge of the original model nvidia/Cosmos3-Edge
Embedl-optimized
- nvidia/Cosmos3-Edge
- INT4
- AGX Thor
- Labels
- Image
- Text
- VLM
- Generation
- Build
- Embedl-optimized
- Precision
- INT4 weights
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
NVIDIA's Cosmos3 Edge reasoner: the 2.4B text-and-image-in, text-out part of NVIDIA's open world model for on-device physical AI.
Methodology
- Throughput: one user, batch 1, greedy, a three-sentence question in, 108 tokens out, with the build's speculative decoding. End-to-end is the whole request (17 ms to the first token); decode excludes prefill, the metric NVIDIA reports on Jetson AI Lab. Prompt processing needs a long prompt and is not measured yet. With speculative decoding, throughput depends on the prompt.
- Accuracy: greedy, the same INT4 model: MMMU-Pro, AI2D, MMBench and DocVQA 1,500 questions each, RealWorldQA all 765.
- Memory: peak GPU memory during the throughput run.
- Size: the TensorRT engines.
- Baseline TensorRT Edge-LLM: the FP16 model on upstream TensorRT Edge-LLM with plain decode, same device: not timed on the AGX Thor yet. Its accuracy is measured here, same questions.
- Vision-language model: throughput measured on text prompts only.