- huggingface
- W8A8
- AGX Thor
- Labels
- Text
- LLM
- Classification
- Build
- Embedl-optimized
- Precision
- W8A8
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 2,048 tokens
A System One decision model: it answers a typed question about a text — one of up to 16 options, yes or no, or a score from 0 to 5 — with a calibrated probability for every answer, in one pass and without writing any text. JEV-9B is AutoTrust AI's open-weights model (Apache-2.0) in the style of TypeSafe AI's Jev; it is not affiliated with TypeSafe AI.
Methodology
- Latency: one decision at a time through the build's own server on the device (POST /v1/infer), median and p90 over 800 held-out questions with prompts of 80 to 1,050 tokens. A choice of more than 16 options takes several engine passes.
- Accuracy: the same 800 questions (200 each from BoolQ, AG News, MNLI and Banking77, held out from calibration), answered on the device, against the published model read out in FP32 on the host.
- Builds: Embedl-optimized INT8: SmoothQuant W8A8 calibrated on decision prompts, an FP8 embedding table, and the LM head cut to the 24 answer rows, read out in the engine. Baseline TensorRT Edge-LLM: the model unquantized in FP16, exported as is with its full 248,320-row LM head; timed only (it reads out no answer, so it has no accuracy).
- Where the speed comes from: A decision is one prefill pass, bound by compute: W8A8 multiplies on the INT8 tensor cores, FP16 on the FP16 ones. Nothing is generated, so no MTP or FlashHead: the 24-row head is already exact.
Files
Variants
Every build of JEV 9B on AGX Thor.