Decisions under test.

Compare model accuracy, calibration, task completion and speed on shared evaluations. Inspect the data and methods behind each result.

English decisions, model by model

Each batch contains 800 natural-language questions and 135 policy or ordinal judgments. Overall accuracy includes both. Choose a batch to inspect the same questions across all four models.

Evaluation batch

Accuracy and response time

One point per model on the selected batch.

80%085%5090%10095%150100%200Response time (ms)

Decoice88.92% · 52.3 ms2,134 / 2,400 correct

Decoice and Prism ran on research HTTP endpoints. Jev includes hosted network time.

Minimum confidence

Expected and observed mistakes

Among answers accepted at 90% confidence or higher.

ExpectedObserved

Decoice19 wrong / 1,627 accepted

2.30%
1.17%

Sarvam Prism66 wrong / 2,080 accepted

3.41%
3.17%

Jev 1.14142 wrong / 2,053 accepted

2.11%
6.92%

Jev 1.13144 wrong / 2,109 accepted

1.91%
6.83%

Bar scale 0–7%. Expected error is the average of one minus each accepted answer’s confidence.

Overall accuracy

English natural-language questions and policy or ordinal judgments

Decoice
90.52%
Sarvam Prism
89.38%
Jev 1.14
86.06%
Jev 1.13
86.42%
Natural-language accuracy

Share of English natural-language questions answered correctly

Decoice
88.92%
Sarvam Prism
87.63%
Jev 1.14
84.25%
Jev 1.13
84.38%
Highly confident errors

Wrong answers given at ≥90% confidence

Decoice
19
Sarvam Prism
66
Jev 1.14
142
Jev 1.13
144
Combined results
MetricDecoiceSarvam PrismJev 1.14Jev 1.13
Overall accuracy90.52%89.38%86.06%86.42%
Natural accuracy88.92%87.63%84.25%84.38%
Wrong at ≥90% confidence1966142144
Confidence when wrong66.05%73.98%77.78%78.10%
Calibration error · 10 bins0.033470.014910.051080.05283
Calibration error · 15 bins0.034190.019640.051810.05266
Brier score0.143340.160210.216860.21090
Log loss0.259090.307520.963561.07656
Mean latency52.26 ms75.37 ms147.12 ms143.87 ms
95th percentile latency47.22 ms106.58 ms188.40 ms185.89 ms
Completed requests2625 / 26252625 / 26252625 / 26252625 / 2625

Lower is better for error counts, confidence on wrong answers, calibration error, probability losses and latency. Decoice and Prism used research HTTP endpoints on the GPU host. Jev timings include the hosted network trip. Decoice’s first-batch p95 reached 599 ms.

Where the accuracy comes from

Decoice
261/300
Sarvam Prism
271/300
Jev 1.14
274/300
Jev 1.13
273/300

Across all batches, Decoice answered 31 more natural questions correctly than Prism. Its largest lead was in emotion recognition. Prism led in coreference and reading comprehension. In batch 3 both answered 703 of 800 natural questions correctly.

What happens above a confidence threshold?

A threshold changes how many answers an application accepts. Compare the average error probability the models reported with the fraction of accepted judgments they got wrong.

Confidence and observed risk
ModelAcceptedWrongExpected errorObserved error
Decoice1627192.30%1.17%
Sarvam Prism2080663.41%3.17%
Jev 1.1420531422.11%6.92%
Jev 1.1321091441.91%6.83%

Through the production API

The public API answered all 2,625 replayed requests. It scored 2,539 of 2,805 judgments overall and 2,134 of 2,400 natural questions, with 18 highly confident errors. Two individual decisions differed from the four-model run. Across the three batches, latency averaged 96.79 ms and the 95th percentile was 85.31 ms from a client in Google Cloud. Batch 1 had a slower tail, with a 95th percentile of 1,180.66 ms.

Download production accuracy, calibration and timing by batch (JSON)
Evaluation method and reliability

The model settings and three disjoint batches were frozen before inference. All four received the same states, questions and options. Two scorers reconstructed the metrics from the recorded responses. The two-sided paired test on pooled natural accuracy gave p = 0.085. Pretraining exposure to source datasets is unknown.

Both Jev endpoints initially rejected 180 policy requests because the client serialized an optional field as null. All 180 succeeded for each endpoint when the client omitted that field. The table uses final predictions and their response times.

Download all batch and family results (JSON)

Public English benchmarks

Decoice 1.0.0 answered all 4,552 questions on tool-use decisions and English reading comprehension through the production Decisions API.

When2Call

3,652 test questions · Accuracy

  • Decoice 1.0.062.87%
  • Sarvam Prism89.81%
  • Jev 1.1481.05%
  • Jev 1.1380.75%
Belebele English

900 test questions · Accuracy

  • Decoice 1.0.080.22%
  • Sarvam Prism96.44%
  • Jev 1.1496.78%
  • Jev 1.1397.22%
When2Call Decision Score

3,423 cases retained by Artificial Analysis · Score out of 100

  • Decoice 1.0.073.45
  • Sarvam Prism93.64
  • Jev 1.1487.38
  • Jev 1.1387.14
Accuracy and confidence on the full public test splits
Test splitModelAccuracyCorrectECE15 ↓Wrong at ≥90% confidence ↓Measured median / p95
When2CallDecoice 1.0.062.87%2,296 / 3,6520.159099398 / 938 ms
When2CallSarvam Prism89.81%3,280 / 3,6520.036595151 / 264 ms
When2CallJev 1.1481.05%2,960 / 3,6520.0606194148 / 201 ms
When2CallJev 1.1380.75%2,949 / 3,6520.0610196141 / 186 ms
Belebele EnglishDecoice 1.0.080.22%722 / 9000.15730334 / 362 ms
Belebele EnglishSarvam Prism96.44%868 / 9000.0403884 / 87 ms
Belebele EnglishJev 1.1496.78%871 / 9000.01568142 / 189 ms
Belebele EnglishJev 1.1397.22%875 / 9000.016310141 / 192 ms

When2Call on the Artificial Analysis case list. After applying its 229 published exclusions, Decoice scored 65.82% accuracy and 73.45 / 100 on the published Decision Score formula across 3,423 cases.

Datasets and scoring

When2Call supplies a tool list, a user request and four possible replies. We shuffled the choices deterministically. Belebele English supplies a passage, question and four choices. Each model received the same context, instructions and option order. NVIDIA When2Call · Belebele English

Accuracy is the share of correct choices. ECE15 measures how closely confidence matched accuracy across 15 bins. The review checks dataset hashes and matches every response to a test case.

Timing includes transport and the service path. Decoice used serial HTTPS from its serving VM. Prism ran serially over its research socket. Each Jev version ran up to six concurrent requests through the hosted Decisions API.

Download English scores and dataset hashes ↗

How the public tests relate to Prism’s report

Sarvam charts its Jev Decision Index score and its recreation of the Artificial Analysis index. When2Call is one evaluation in Artificial Analysis. The Decision Score above uses its filtered case list and formula with our Decisions API request format. Each published composite also covers other evaluations and uses its own weights. Sarvam’s report · Artificial Analysis methodology