OpenAI Decisions API vs Jev: 20 tests, 10,000 calls

English narration · 720p · captions track (en) · MP4

Article: Jev vs OpenAI Decisions, 20 tests and 10,000 calls

I gave Jev and OpenAI's new Decisions API the same set of questions: 20 tests, 10,000 calls across both. Jev won 8 tests, Decisions won 1, and the other 11 were ties. A tie means the gap was too small to call.

A score alone isn't very vivid, so here is one question. It comes from a recorded session on a real website: the task is to book Eiffel Tower tickets on Tripadvisor. The previous step already typed "Eiffel Tower" into the search box; the next step should be clicking Eiffel Tower in the dropdown. Both models report how confident they are when they pick. Jev got it right with 0.98 confidence. Decisions picked the search box again, repeating the step that was already done.

Now that OpenAI has a product of the same kind, is Jev still worth using? This article answers only with numbers I measured: group by group, who won and why that might be; then speed, price and confidence; and finally which one to pick for which job.

1. Two products of the same kind, 14 days apart

Some AI models don't write text. They only answer multiple-choice questions; I call them decision models. Jev is TypeSafe's decision model, released on September 15; its founder used to be a researcher at OpenAI. OpenAI announced the Decisions API at DevDay on September 29, just 14 days later. Decisions is built on GPT-6 Luna, and its public beta opened on October 6. The two are the same kind of product and are used in almost the same way.

OpenAI highlights two selling points: speed, with answers in about 150 milliseconds, and the ability to see images, which Jev lacks. OpenAI published the price but no accuracy numbers. So the only way to know which is more accurate is to test it yourself.

2. What a decision model is

A regular LLM answers with a block of text. A decision model writes nothing; it just returns a probability for each option. Take support-ticket routing: a customer writes that their credit card was charged twice, and you need to send the ticket to either billing or tech. An LLM might reply with a paragraph of analysis that your code then has to parse for the team name. A decision model gives each team a probability. Billing is highest, so the ticket goes to billing.

There are three question types:

• choice: pick one, such as routing a ticket.
• predicate: yes or no, such as whether a piece of code is correct. Jev calls this noul.
• score: a rating, such as the quality of a support reply.

The upside of this output is that the answer is always one of the options, and your code can use the probabilities directly without parsing any text.

3. How I tested

The 20 tests fall into 5 groups by skill: classification and routing, judging right from wrong, reading comprehension, agent decisions, and vision and limits. Each model answered 5,000 questions, and all 10,000 calls succeeded without a single error.

Correct answers come from dataset labels or from programs such as solvers; code questions were checked by running unit tests. No LLM did any grading, because LLMs make mistakes too. Both models got exactly the same state, questions and even option order. Everything ran on October 7 on the same Mac in Tokyo, both models in the same time window, 4 concurrent requests at a time. I had also run Jev once on September 30; a week later, every test differed by about 1 percentage point.

Wins are decided only on the questions where the two answered differently, and only when the gap is too big to be luck, which statisticians call significant. Otherwise it's a tie. In banking support, for example, Jev scored 82% and Decisions 80%; that gap could be luck, so it's a tie. I'll open-source the questions and results on GitHub so anyone can check them.

4. Classification and routing: about even

Routing just means sending a message where it belongs.

• Banking intents (Banking77, pick 1 of 77): Jev 82%, Decisions 80%, tie.
• Chinese and Japanese voice-assistant intents (pick 1 of 60): Jev 84%, Decisions 81%, tie.
• Guessing a product review's star rating (exact match only): Jev 53%, Decisions 49%, tie. Both find exact stars hard.
• Moderating offensive Chinese text: Jev 78%, Decisions 72%, still a tie by the rule. Look closer and Decisions is lenient: of the truly offensive messages, Decisions caught 59%, Jev 79%.
• Whether a SQL query meets its requirement: both 86%.

For simple classification and routing, either works; decide on speed and price.

5. Judging right from wrong: each wins something

This group is all predicates: the answer is yes or no.

Code correctness (HumanEval, half the functions broken in one spot): Jev 86%, Decisions 78%, Jev wins. Decisions loses by being too strict: it judged correct code correct only 62% of the time. One function that converts numbers to another base was completely correct, and Decisions gave it a 0.01 chance of being correct.

Checking math solutions (GSM8K, half with one wrong step): Decisions 88%, Jev 78%. This is the only test of the 20 that Decisions won. In one problem, a party has 40 people, 10 leave, and the solution says 25 remain. Decisions was 100% sure it was wrong; Jev was 0.65 confident it was right.

Fact checking (given evidence, is a claim supported, refuted, or is there not enough info): Jev 82%, Decisions 75%, Jev wins.

Answerability (half the questions have no answer in the passage): Jev 84%, Decisions 75%, Jev wins. In one case the passage is about computational complexity theory, but the question asks what "computational simplicity theory" studies. Decisions was 0.9 sure it could be answered; Jev judged it most likely unanswerable.

For judgments over long text, Jev is steadier; for checking arithmetic step by step, Decisions is stronger.

6. Reading comprehension: Jev is better at spotting contradictions

• Passage retrieval (10 paragraphs from one article, find the one with the answer): both 98%, tie.
• Commonsense continuation (HellaSwag, pick the most sensible of 4 endings): Jev 96%, Decisions 93%, Jev wins.
• Chinese inference (OCNLI, entailment, neutral or contradiction): Jev 83%, Decisions 72%, Jev wins.

On the contradiction questions alone, Jev caught 80% and Decisions only 52%. One sentence says the basket costs one yuan, the other says it sells for two. Jev was 0.99 sure they contradict; Decisions called it neutral.

Both can find answers; Jev is better at finding contradictions.

7. Agent decisions: the biggest gap is the next web action

An agent is an AI program that gets a job done step by step, picking one action from a few at every step.

• Tool selection (pick 1 of 12 functions): both 99%.
• Next web action (Mind2Web, pick which of 10 page elements to click next): Jev 86%, Decisions 70%, Jev wins. This is the biggest gap of all 20 tests, and the Eiffel Tower question above comes from here.
• Kanban task routing (219 real cards into 6 projects): Jev 94%, Decisions 92%, tie.
• Sokoban (generated levels, pick the unique optimal first move out of 4 directions; guessing gets 25%): Jev 46%, Decisions 36%, Jev wins, but both are far from passing.

Picking a tool by name is easy for both; understanding a web page or planning a few moves ahead is harder for Decisions.

8. Vision: only Decisions can see

Seeing images is unique to Decisions; Jev reads text only. So the vision tests work like this: Decisions gets half its questions as images and half as the same content in text, while Jev gets all of them as text. That separates "is seeing hard?" from "is the question hard?"

Mazes (generated, which way next; guessing gets 25%): Decisions 35% from images, 28% from text, Jev 28% from text. All three are close to guessing. Charts (generated bar and line charts, which item is largest): Decisions 100% from images, 98% from text, Jev 100% from text.

Seeing isn't the problem. Mazes are hard because of spatial reasoning, walking the route in your head. Vision is a bonus, but it doesn't solve hard reasoning. If your use case needs images, Decisions is currently the only choice.

9. Limits: more options, and attempts to mislead

Option-count pressure (CLINC, growing the options for the same kind of question from 2 to 150): with 2 options, both score 100%; with 150, Jev still gets 94% while Decisions drops to 86%. Jev wins this one.

Robustness (50 banking questions, 5 variations each): asking the same question twice more changed neither model's answer; reversing the option order changed 1 answer for each; hiding a line in the user message telling the model to pick a wrong answer actually misled each model on 2 questions. Both are steady; tie.

10. Speed, price and confidence

Speed here is latency, the total time from sending a request to getting the answer. In Tokyo, Jev's median was 183 ms and Decisions' was 292 ms. p99 measures the slowest slice of requests (the slowest 1 in 100): Jev 381 ms, Decisions 1,064 ms, so Decisions is occasionally much slower. Requests with images were slower still, with a 423 ms median. OpenAI says about 150 ms; I didn't see that from Tokyo, and my guess is that network distance plays a part.

Price: Jev's 5,000 questions cost $0.16 in total. Decisions published its price on October 6; at that price, the same questions come to about $0.24. Both are cheap enough to ignore.

Confidence is how sure a model says it is. A well-calibrated model that says 90% is right 90% of the time. Looking only at answers given with 90%+ confidence: Jev reached that on 64% of questions and was right on 94% of them; Decisions reached it on 65% and was right on 92%. Both models' confidence is broadly trustworthy.

So in practice: let the model handle what it's sure about, and send the rest to a person. In the arithmetic question above, Jev accepted a wrong solution, but only with 0.65 confidence, so that question would go to a person. Also, Decisions refused to answer 5 questions; Jev refused none.

11. Why (mostly guesses)

The test Decisions won is checking every calculation step in math solutions. It's built on GPT-6 Luna, a general-purpose LLM. My guess is that general LLMs are better at verifying numbers step by step. That's only a guess; we can't see inside it.

Jev is trained specifically for decisions. TypeSafe says its training goal is for the confidence it states to match its real accuracy. My guess is that this makes it steadier at understanding text and spotting contradictions. Again, a guess, not a measured fact.

One more fact: I tested the October 7 version, and Decisions is still in public beta. Later versions may do better.

12. Which one to pick

Across all questions, Jev got 81% right and Decisions 78%. The totals are close; what matters is where each is strong.

Limits first: I tested for one day in Tokyo, and latency will differ elsewhere; I chose the 20 tests, and your use case may not be among them; the questions come from public datasets that either model may have partly seen in training; and Decisions is in beta, so the release version may differ.

• You need images: only Decisions, for now.
• You need to check numbers and calculation steps: Decisions is stronger.
• Simple classification and routing: either works, and Jev is faster and cheaper (183 ms vs 292 ms median).
• Understanding long text, spotting contradictions, or choosing the next web action: Jev is steadier.

The best approach is to take 200 questions with known answers from your own work, run both models, and see which one is more accurate. Once my code is open-sourced, you can adapt it directly. So whether Jev is still worth using depends on which kind of judgment you need.

Which kind of judgment would you hand to a model like this? Let me know in the YouTube comments.

The English version of the video is on YouTube; the Chinese version is on Bilibili. Series: Jev01 · Jev02 · Jev03 · Jev04 · Jev vs Laya · Jev × RAG.