Cosmos3 Edge

Methodology

• Throughput: one user, real conversations from the public OpenAssistant set in the model's chat template, 2,048 tokens in, 128 out, greedy, no prompt reuse, with the build's EAGLE3 drafting (top-k 3, 4 steps, verify tree 16; 2.66 tokens per verify pass). Average of 20 requests after 3 warm-up requests. End-to-end is the whole request; prompt processing is the 2,048 prompt tokens over the time to first token (636 ms); decode excludes prefill. On random tokens (Jetson AI Lab's protocol), where the draft is rarely right: 52.3 tok/s end-to-end, 70.0 tok/s decode.
• Accuracy: greedy, the same INT4 model: MMMU-Pro, AI2D, MMBench and DocVQA 1,500 questions each, RealWorldQA all 765.
• Power and memory: sampled on the device during those runs (VDD_GPU_SOC + VDD_CPU_CV + VIN_SYS_5V0). Idle is the device at rest with the model loaded; energy per token is the run's energy over its generated tokens.
• Size: the runtime container, download and on disk.
• Baseline TensorRT Edge-LLM: the FP16 model on upstream TensorRT Edge-LLM with plain decode, same device: not measured yet.
• Device: an NVIDIA Jetson AGX Orin developer kit flashed as an Orin NX 16GB (NVIDIA's emulation configuration), GPU 918 MHz, memory 3199 MHz, CPUs 1984 MHz.
• Vision-language model: throughput measured on text prompts only.