Jev is TypeSafe’s System One decision model. Give it an app state object and a typed question, and it gives you back a typed answer, a confidence score, and the probability for each possible option (the Decisions API response type, not the chat endpoints). According to the TypeSafe pitch, you can replace a frontier chat model with a small judgment model like Jev and get significant cuts in both cost and latency, on classification tasks such as this one. How much accuracy are you sacrificing?
We ran 3,080 Banking77 customer support utterances through both Jev 1.13 and Claude Opus 5, the model OpenRouter users spent the most on for classification in the Task spend section of its rankings page as of 22 September 2026. Each utterance gets sorted into one of 77 banking intents, and both models got the same one-line description of each intent.
The chart below puts Jev and Opus side by side, showing how they compare on accuracy, median latency, and cost per thousand requests.
Jev comes in 3.3 points behind Opus on accuracy, but it’s 13 times faster at the median and costs just 1/22 of what Opus does.
Jev vs Claude Opus 5 at a glance
Here’s a closer look at the results. Macro-F1 gives equal weight to every intent.
| Jev 1.13 | Claude Opus 5 | |
|---|---|---|
| Accuracy | 81.0% (79.6 to 82.3) | 84.4% (83.1 to 85.6) |
| Macro-F1 | 80.5% | 83.6% |
| Invalid responses | 0 of 3,080 | 0 of 3,080 |
| Latency p50 | 175 ms | 2,266 ms |
| Latency p95 | 270 ms | 3,004 ms |
| Latency p99 | 353 ms | 3,835 ms |
| Billed cost, full run | $0.34 | $7.44 |
| Cost per 1,000 requests | $0.11 | $2.42 |
| Mean input tokens | 2,605 | 3,750 (3,725 cached) |
| Mean output tokens | 826 | 17 |
The numbers in parentheses are 95% bootstrap confidence intervals, which we calculated using 3,080 examples. Neither model returned a malformed response. Every error was a wrong label, not a parse failure.
How we ran it
Banking77 is an utterance-level customer support intent dataset from PolyAI with the CC BY 4.0 license. The utterances are short, median nine words, and each one is labeled with one of 77 intents. When we ran Jev and Opus on Banking77, we used the full test split, which has 3,080 examples, 40 per intent. Some of the intents are close enough that the models can’t just pick up on a handful of keywords and stop reading. Think card_arrival vs. card_delivery_estimate.
We wrote up a one-line criterion for each intent from just the label name, without looking at the test data at all, and we gave the same exact list of criteria to both Jev and Opus. To score Jev, we sent each utterance as a Decisions API Choice question with the 77 criteria as the options. For Opus, we built a prompt where it saw a system message listing all the criteria, and then a user message with just the utterance. We ran Opus at temperature zero, with reasoning turned off, and using a strict JSON schema response format with an enum of the 77 labels. We turned on prompt caching for the system message, since it was identical on every request. Both models ran from the same machine, and we sent 8 concurrent requests at a time and measured latency as the client-observed round-trip through OpenRouter.
How accurate is Jev?
Let’s talk numbers. On Banking77, Jev is at 81.0%, Opus at 84.4%. A paired bootstrap gives a 95% CI of 2.3 to 4.4 points, so Opus ahead by about three points is unlikely to be noise.
On the upside, these two have a ton of agreement. They agree on 89.3% of the utterances. Where they split, Opus gets 175 right on its own, Jev gets 72.
On the downside, both are still sitting well below the low 90s that encoders fine-tuned on all 10,003 Banking77 training examples report. That’s the cost of relying on one-line criteria instead of doing a full fine-tune.
By class, Opus leads Jev on 35 of the 77 intents, Jev on 15, tied on 27. Opus’s biggest lead is on receiving_money, where it’s at 80.0% compared to Jev’s 52.5%. Meanwhile, Jev shines on compromised_card where it’s at 95.0% while Opus is at 70.0%. Opus tended to mistake stolen card details for an unrecognized payment.
How fast and how cheap is Jev?
Very quickly, Jev’s median round trip was 175 ms and p95 was 270 ms. Opus, on the other hand, without reasoning mode, had a median round trip of 2,266 ms and p95 of 3,004 ms. That means the slowest Jev call (~1.6s) was faster than the fastest Opus call (~1.9s).
The total for Jev was $0.34, or $0.11 per thousand requests. Opus came in at $7.44, or $2.42 per thousand requests. That Opus figure has the 3,700-token system prompt cached, so all the requests were at the cache-read rate instead of the list rate. Without caching, the list rate for Opus would have been about $19 per thousand requests. So if you’re using a label taxonomy with a frontier model, cache that system prompt.
Routing on Jev’s confidence
Jev gives you a confidence score, but it’s not a calibrated probability. Here, it overestimated its own accuracy mid-range. Still, it ranks well. For 58% of utterances with confidence ≥0.99, accuracy is 96.3%. For the 3.5% below 0.5, accuracy is 29.6%.
That ordering is all you need to cascade. Accept Jev above a given threshold, send the rest to Opus. Here’s accuracy and cost per thousand at each threshold, ranging from Jev only down to Opus only.
| Threshold | Handled by Jev | Accuracy | Cost per 1,000 |
|---|---|---|---|
| Jev only | 100% | 81.0% | $0.11 |
| 0.99 | 57.8% | 84.3% | $1.13 |
| 0.95 | 68.1% | 84.2% | $0.88 |
| 0.90 | 75.9% | 84.0% | $0.69 |
| 0.80 | 82.7% | 83.6% | $0.53 |
| 0.70 | 87.3% | 83.2% | $0.42 |
| Opus only | 0% | 84.4% | $2.42 |
At threshold 0.90, 76% of traffic bypasses Opus. In exchange, your accuracy is reduced by at most 0.4 points relative to Opus alone, and your cost cut by a factor of 3.5. Opus got right 175 utterances that Jev missed. The median confidence of those cases was 0.67, and 85% were below 0.90. So Jev usually knows when it’s just guessing.
Caveats
So let’s talk narrowly about the actual test we did: one dataset, one domain, one prompt design, and one fifteen-minute window on one afternoon.
If there is an Opus memorization edge here, its score may be slightly flattered by it, because Banking77 was published in 2020. But you can’t tell from this run alone.
The other point is that both models stumbled on one class, because the criteria were written from label names only, without seeing example messages. In this case, one of the labels was get_physical_card, which covers queries about whether the PIN is delivered separately. It’s practically impossible to guess that from the name. Each model scored 0 out of 40 on that class, routing most messages to change_pin instead. Excluding that class alone, Jev hit 82.1% and Opus 85.5%. But a shipping team would iterate on the criteria using held-out training data, not the test set, and both models would get better.
The cascade table we showed used exactly the same 3,080 examples used for the accuracy measurement, so this is an upper bound. Choose your threshold based on your own traffic.
And lastly, these tests were run with Opus in “reasoning off” mode and structured outputs on. Opus was not tested in “reasoning on” mode, or with other schemas or few-shot examples.
What this means
If three points in accuracy outweigh $2.42 per thousand requests, go with Opus. If you’re dealing with high volume, low latency, or need a fallback slot, Jev alone got within 3.3 points of Opus. Even better, running a cascade on Jev’s confidence score brought us within 0.4 points of Opus, for under 30% of the cost.
The Decisions API reference walks through the basic request shape. The Jev cookbook walks you through a working Choice question in TypeScript. The Jev-verified cascade cookbook has the escalation pattern in code, with a cheap model drafting, Jev checking the draft, and a frontier model handling only what fails the check. The Banking77 test split is a 3,080-line CSV in the PolyAI repository.
Frequently Asked Questions
How accurate is Jev compared to Claude Opus 5 on intent classification?
On the Banking77 test split consisting of 3,080 utterances across 77 intents run on 22 September 2026, Claude Opus 5 achieved 84.4% accuracy and 83.6% macro-F1, while Jev 1.13 achieved 81.0% accuracy and 80.5% macro-F1. The paired gap was 3.3 percentage points on accuracy with a 95% bootstrap interval from 2.3 to 4.4 percentage points. The models agreed on 89.3% of utterances.
How much faster is Jev than a frontier model for classification?
On median and p95 client-observed round trip latency, Jev achieved median 175 ms and p95 270 ms, while Claude Opus 5 with reasoning disabled achieved median 2,266 ms and p95 3,004 ms, about 13 times higher at the median. These numbers are measured from the same client under 8 concurrent requests. They include network time and any queueing in the provider’s system before the actual inference.
How much does it cost to classify text with Jev on OpenRouter?
All 3,080 of the Banking77 requests were billed $0.34 total on typesafe/jev-1.13, which is $0.11 per thousand requests or about $0.0001 per request. Using prompt caching on the 77-label system prompt, the same requests billed $7.44 total on anthropic/claude-opus-5, or $2.42 per thousand. Without caching, Opus would have billed about $19 per thousand at list price.
Can I use Jev’s confidence score to decide when to fall back to a bigger model?
Yes, and in our data it works as a ranking signal. We measured Jev as 96.3% accurate on the 58% of utterances with confidence at least 0.99, and 29.6% accurate on the 3.5% of utterances with confidence under 0.5. Routing everything under 0.9 confidence to Claude Opus 5 recovered 84.0% accuracy, within 0.4 points of Opus alone, at $0.69 per thousand requests. Because the score is not calibrated as a probability, pick the threshold that works for your own data.