Embedl-optimized · TensorRT
- huggingface
- INT8
- AGX Orin
- Labels
- Image
- Segmentation
- Build
- Embedl-optimized
- Precision
- INT8 where Embedl Deploy quantized (attention probabilities FP16), FP16 elsewhere
- Runtime
- TensorRT 10.16.2.10 (JetPack 7.2), 927 MiB engine
- Input
- One photo at 924×924 and a text prompt of up to 32 tokens
- Upstream
- facebook/sam3; the graph from embedl/sam3 at bb4fe83
Text-prompted segmentation: a photo and a short phrase in, a mask for every object the phrase names out. Meta's SAM 3 image detector as Embedl published it, quantized to INT8 with Embedl Deploy (embedl/sam3), compiled with TensorRT 10.16.2.10 on the NVIDIA Jetson AGX Orin (no re-quantization: INT8 where Embedl quantized it, FP16 elsewhere; its attention rewritten in forms TensorRT 10.16 compiles correctly), with a KServe v2 server (serve.py) that takes the photo and the prompt and answers with the masks.
Methodology
- Latency: on the NVIDIA Jetson AGX Orin as the device farm runs it, batch 1. Engine: GPU time of one execution between CUDA events, copies excluded, mean over 20 s after a 5 s warm-up. Photo and prompt to masks: wall time of serve.py's whole answer (tokenize, resize, the engine with copies, masks to PNGs), median of 10, on a 640×480 COCO photo. No TensorRT baseline: no engine of the unquantized graph is right (on the L4, with FP16 it finds nothing; in FP32 its presence score is wrong).
- Accuracy: the served pipeline on the first 300 COCO val2017 photos with objects: each category present in a photo prompted by its name, every instance scoring above 0.01 (at most 100 per prompt) scored by pycocotools' mask AP. Prompting only the categories present leaves out the prompts with no answer that SAM 3's own benchmarks have, so the numbers compare builds; they are not SAM 3's published results. FP32: Embedl's graph with its Q/DQ stripped, in onnxruntime (CPU), on the same code around it.
- Memory: the serialized engine; FP32 weights: the unquantized graph's ONNX weights; serve.py's peak resident memory, engine loaded and one request answered (a Jetson's GPU memory is the system's).
- Graph: Embedl's published graph (embedl/sam3) with two equivalent rewrites, because TensorRT after 10.3 compiles its attention wrongly: the INT8 rounding of the softmax's output removed (TensorRT's fused INT8 attention kernel is wrong at these shapes), and the float attention's Q·Kᵀ written as an Einsum.
- Check: 6 photo-and-prompt pairs through serve.py on this engine, against the same INT8 graph in onnxruntime (CPU): union-mask IoU mean 0.982, lowest 0.9462; the build fails below 0.9 or if one finds something the other doesn't.