Cosmos3 Edge

Methodology

• Throughput: one user, batch 1, greedy, a three-sentence question in, 108 tokens out, with the build's speculative decoding (EAGLE3: top-k 3, 4 steps, verify tree 16). End-to-end is the whole request (8.6 ms to the first token); decode excludes prefill. Median of 20 requests after 3 warm-up, served by the build's own embedl-serve on one RTX PRO 6000 Blackwell Server Edition (driver 580.178.04), rented from Hugging Face Jobs. With speculative decoding, throughput depends on the prompt. Text prompts only.
• Accuracy: greedy, the same INT4 model, measured on the NVIDIA Jetson AGX Thor (not yet on the RTX PRO 6000; its engines are compiled for this GPU, so answers can differ in the last digits): MMMU-Pro, AI2D, MMBench and DocVQA 1,500 questions each, RealWorldQA all 765.
• Memory: peak GPU memory during the throughput run, less what the GPU used before the server started.
• Size: the TensorRT engines.
• Baseline TensorRT Edge-LLM: the FP16 model on upstream TensorRT Edge-LLM with plain decode; accuracy only, same questions.