Embedl optimized · TensorRT
- huggingface
- INT8
- L4
- Labels
- Image
- Segmentation
- Build
- Embedl-optimized
- Precision
- INT8 where Embedl Deploy quantized, FP16 elsewhere
- Runtime
- TensorRT 10.3.0, 929 MiB engine
- Input
- One photo at 924×924 and a text prompt of up to 32 tokens
- Upstream
- facebook/sam3; the graph from embedl/sam3 at bb4fe83
Text-prompted segmentation: a photo and a short phrase in, a mask for every object the phrase names out. This is Meta's SAM 3 image detector.
Licences
Derivative of facebook/sam3 by Meta, under the SAM License (LICENSE beside the files; it allows use and redistribution, and has to travel with the model). Embedl's quantized graph is under the Embedl Models Community Licence. The tokenizer's merges are upstream's (facebookresearch/sam3). NOTICE says the same beside the files.
Methodology
- Latency: on the NVIDIA L4, batch 1. Engine: GPU time of one execution between CUDA events, copies excluded, mean. Photo and prompt to masks: wall time of serve.py's whole answer (tokenize, resize, the engine with copies, masks to PNGs), median of 10 after a warm-up, on a 640×480 COCO photo. No baseline: no TensorRT engine of the unquantized graph is right (with FP16 it finds nothing; in FP32 its presence score is wrong); the model card's baseline is PyTorch's.
- Accuracy: the served pipeline on the first 300 COCO val2017 photos with objects: each category present in a photo prompted by its name ("dining table"), every instance scoring above 0.01 (at most 100 per prompt) scored by pycocotools' mask AP over those photos. Prompting only the categories present leaves out the prompts with no answer that SAM 3's own benchmarks have, so the numbers compare builds; they are not SAM 3's published results. FP32: Embedl's graph with its Q/DQ stripped, unquantized, in onnxruntime (CPU), on the same code around it.
- Memory: the serialized engine; FP32 weights: the unquantized graph's ONNX weights.
- Check: 6 photo-and-prompt pairs through serve.py on this engine and on the same INT8 graph in onnxruntime (CPU): union-mask IoU mean 0.992, lowest 0.9842; the build fails below 0.9 or if one finds something the other doesn't.