Decisions under test.
Compare model accuracy, calibration, task completion and speed on shared evaluations. Inspect the data and methods behind each result.
English decisions, model by model
Each batch contains 800 natural-language questions and 135 policy or ordinal judgments. Overall accuracy includes both. Choose a batch to inspect the same questions across all four models.
Accuracy and response time
One point per model on the selected batch.
Decoice88.92% · 52.3 ms2,134 / 2,400 correct
Decoice and Prism ran on research HTTP endpoints. Jev includes hosted network time.
Expected and observed mistakes
Among answers accepted at 90% confidence or higher.
Decoice19 wrong / 1,627 accepted
Sarvam Prism66 wrong / 2,080 accepted
Jev 1.14142 wrong / 2,053 accepted
Jev 1.13144 wrong / 2,109 accepted
Bar scale 0–7%. Expected error is the average of one minus each accepted answer’s confidence.
English natural-language questions and policy or ordinal judgments
Share of English natural-language questions answered correctly
Wrong answers given at ≥90% confidence
| Metric | Decoice | Sarvam Prism | Jev 1.14 | Jev 1.13 |
|---|---|---|---|---|
| Overall accuracy | 90.52% | 89.38% | 86.06% | 86.42% |
| Natural accuracy | 88.92% | 87.63% | 84.25% | 84.38% |
| Wrong at ≥90% confidence | 19 | 66 | 142 | 144 |
| Confidence when wrong | 66.05% | 73.98% | 77.78% | 78.10% |
| Calibration error · 10 bins | 0.03347 | 0.01491 | 0.05108 | 0.05283 |
| Calibration error · 15 bins | 0.03419 | 0.01964 | 0.05181 | 0.05266 |
| Brier score | 0.14334 | 0.16021 | 0.21686 | 0.21090 |
| Log loss | 0.25909 | 0.30752 | 0.96356 | 1.07656 |
| Mean latency | 52.26 ms | 75.37 ms | 147.12 ms | 143.87 ms |
| 95th percentile latency | 47.22 ms | 106.58 ms | 188.40 ms | 185.89 ms |
| Completed requests | 2625 / 2625 | 2625 / 2625 | 2625 / 2625 | 2625 / 2625 |
Lower is better for error counts, confidence on wrong answers, calibration error, probability losses and latency. Decoice and Prism used research HTTP endpoints on the GPU host. Jev timings include the hosted network trip. Decoice’s first-batch p95 reached 599 ms.
Where the accuracy comes from
Across all batches, Decoice answered 31 more natural questions correctly than Prism. Its largest lead was in emotion recognition. Prism led in coreference and reading comprehension. In batch 3 both answered 703 of 800 natural questions correctly.
What happens above a confidence threshold?
A threshold changes how many answers an application accepts. Compare the average error probability the models reported with the fraction of accepted judgments they got wrong.
| Model | Accepted | Wrong | Expected error | Observed error |
|---|---|---|---|---|
| Decoice | 1627 | 19 | 2.30% | 1.17% |
| Sarvam Prism | 2080 | 66 | 3.41% | 3.17% |
| Jev 1.14 | 2053 | 142 | 2.11% | 6.92% |
| Jev 1.13 | 2109 | 144 | 1.91% | 6.83% |
Through the production API
The public API answered all 2,625 replayed requests. It scored 2,539 of 2,805 judgments overall and 2,134 of 2,400 natural questions, with 18 highly confident errors. Two individual decisions differed from the four-model run. Across the three batches, latency averaged 96.79 ms and the 95th percentile was 85.31 ms from a client in Google Cloud. Batch 1 had a slower tail, with a 95th percentile of 1,180.66 ms.
Download production accuracy, calibration and timing by batch (JSON)Evaluation method and reliability
The model settings and three disjoint batches were frozen before inference. All four received the same states, questions and options. Two scorers reconstructed the metrics from the recorded responses. The two-sided paired test on pooled natural accuracy gave p = 0.085. Pretraining exposure to source datasets is unknown.
Both Jev endpoints initially rejected 180 policy requests because the client serialized an optional field as null. All 180 succeeded for each endpoint when the client omitted that field. The table uses final predictions and their response times.
Public English benchmarks
Decoice 1.0.0 answered all 4,552 questions on tool-use decisions and English reading comprehension through the production Decisions API.
3,652 test questions · Accuracy
- Decoice 1.0.062.87%
- Sarvam Prism89.81%
- Jev 1.1481.05%
- Jev 1.1380.75%
900 test questions · Accuracy
- Decoice 1.0.080.22%
- Sarvam Prism96.44%
- Jev 1.1496.78%
- Jev 1.1397.22%
3,423 cases retained by Artificial Analysis · Score out of 100
- Decoice 1.0.073.45
- Sarvam Prism93.64
- Jev 1.1487.38
- Jev 1.1387.14
| Test split | Model | Accuracy | Correct | ECE15 ↓ | Wrong at ≥90% confidence ↓ | Measured median / p95 |
|---|---|---|---|---|---|---|
| When2Call | Decoice 1.0.0 | 62.87% | 2,296 / 3,652 | 0.1590 | 99 | 398 / 938 ms |
| When2Call | Sarvam Prism | 89.81% | 3,280 / 3,652 | 0.0365 | 95 | 151 / 264 ms |
| When2Call | Jev 1.14 | 81.05% | 2,960 / 3,652 | 0.0606 | 194 | 148 / 201 ms |
| When2Call | Jev 1.13 | 80.75% | 2,949 / 3,652 | 0.0610 | 196 | 141 / 186 ms |
| Belebele English | Decoice 1.0.0 | 80.22% | 722 / 900 | 0.1573 | 0 | 334 / 362 ms |
| Belebele English | Sarvam Prism | 96.44% | 868 / 900 | 0.0403 | 8 | 84 / 87 ms |
| Belebele English | Jev 1.14 | 96.78% | 871 / 900 | 0.0156 | 8 | 142 / 189 ms |
| Belebele English | Jev 1.13 | 97.22% | 875 / 900 | 0.0163 | 10 | 141 / 192 ms |
When2Call on the Artificial Analysis case list. After applying its 229 published exclusions, Decoice scored 65.82% accuracy and 73.45 / 100 on the published Decision Score formula across 3,423 cases.
Datasets and scoring
When2Call supplies a tool list, a user request and four possible replies. We shuffled the choices deterministically. Belebele English supplies a passage, question and four choices. Each model received the same context, instructions and option order. NVIDIA When2Call · Belebele English
Accuracy is the share of correct choices. ECE15 measures how closely confidence matched accuracy across 15 bins. The review checks dataset hashes and matches every response to a test case.
Timing includes transport and the service path. Decoice used serial HTTPS from its serving VM. Prism ran serially over its research socket. Each Jev version ran up to six concurrent requests through the hosted Decisions API.
How the public tests relate to Prism’s report
Sarvam charts its Jev Decision Index score and its recreation of the Artificial Analysis index. When2Call is one evaluation in Artificial Analysis. The Decision Score above uses its filtered case list and formula with our Decisions API request format. Each published composite also covers other evaluations and uses its own weights. Sarvam’s report · Artificial Analysis methodology