Embedl's build autotrust/jev-9b of the original model autotrust/JEV-9B
Embedl optimized · TensorRT Edge-LLM
- huggingface
- W8A8
- AGX Thor
- Labels
- Text
- LLM
- Classification
- Build
- Embedl-optimized
- Precision
- W8A8
- Runtime
- TensorRT Edge-LLM 0.10.1 with Embedl optimizations
- Context length
- 2,048 tokens
A System One decision model: it answers a typed question about a text — one of up to 16 options, yes or no, or a score from 0 to 5 — with a calibrated probability for every answer, in one pass and without writing any text. JEV-9B is AutoTrust AI's open-weights model (Apache-2.0) in the style of TypeSafe AI's Jev; it is not affiliated with TypeSafe AI.
| Quantization | Embedl INT8 (SmoothQuant W8A8), the answer read out in the engine |
Methodology
- Latency: one decision at a time through the build's own server on the device (POST /v1/infer), median and p90 over 800 held-out questions with prompts of 80 to 1,050 tokens. A choice of more than 16 options takes several engine passes.
- Accuracy: the same 800 questions (200 each from BoolQ, AG News, MNLI and Banking77, held out from calibration), answered on the device, against the published model read out in FP32 on the host.
- Baseline: the stock TensorRT Edge-LLM INT4 build (ModelOpt AWQ) with the model's full LM head, reading the answers' logits, timed the same way on the same device.