10,000 API calls later: reverse-engineering Jev's architecture

Chinese narration · 720p · captions track (zh) · MP4

Article: Reverse-engineering Jev's architecture with 10,000 API calls

Suppose you ask a large language model which team should handle a customer complaint, and it replies, "I'm 90% sure this belongs to the payments team." That sounds reassuring, but the 90% is just text the model produced one token at a time. It is no different in kind from a person typing "I'd say 90%" into a chat box. Nothing trained it to be a real probability, so a downstream system can't safely treat it as one.

Jev is built to fix exactly this. It is a decision model released by TypeSafe: instead of generating text, it returns trained probability distributions directly. After the launch, X filled up with analyses, and Archer Hume thought most of them were wrong. So he spent about 10,000 API calls probing Jev from the outside, measuring latency curves, token counts, sensitivity to option order and more, and wrote up what he found in "Jev's Architecture Unmasked." This article walks through his findings in the same order as the video. One caveat up front: the claims are not equally solid. Some come from TypeSafe's official statements, some are reproducible measurements, and some are the author's inferences, and each is labeled as such below.

How Jev is used: a shared state plus a list of questions

A request to Jev has two parts. The first is the shared state, the context every question refers to, such as the text of the customer's complaint. The second is a list of questions, for example "Which team should handle this? Options: payments / account / other" and "Should this be escalated? yes / no."

Jev doesn't reply with prose. It returns a probability distribution per question: say payments 91%, account 6%, other 3%, and escalate yes 42%, no 58%. Your code can act on those numbers directly, routing automatically when the payments probability clears a threshold and sending the ticket down the normal path when escalation looks unlikely.

According to TypeSafe, Jev's outputs are "parallel probabilities," not autoregressive token-by-token generation. It supports three question types: finite choice, yes/no, and ordinal ratings. Each maps naturally onto a fixed number of numeric outputs, so nothing needs to be written out word by word.

Evidence that it really doesn't generate text

The author first looked at the output_tokens field in the API response. The name suggests a count of generated tokens, but the numbers follow an odd pattern. For yes/no questions it equals exactly 4 shared tokens, plus 15 tokens per answer, plus the token length of the question's identifier. TypeSafe's documentation states that this identifier "is not sent to the underlying model and does not participate in inference." A count that changes with text the model never sees must be computed from the serialized response after inference. It is a billing figure, not a measure of generation.

The latency tests are even more telling. A question with 200 options (1,911 output_tokens by that formula) came back about as fast as a question with only 2. Server time grew with input length, not with the number of options. If the model were actually writing out its answer, the 200-option reply could not be as fast as the 2-option one.

Where the probabilities come from: a readout (inferred)

TypeSafe only says the probabilities are produced in parallel; it hasn't published the mechanism. The author's inference is a readout. After processing the input, the model takes the final hidden vector, multiplies it by a weight matrix, adds a bias, and applies a softmax, the standard operation that turns a set of raw scores into probabilities summing to 1. That yields a probability for each option directly. A yes/no question needs even less: a single scalar through a sigmoid. There is no "what's the next word" loop anywhere.

Shared state and isolated question branches

Imagine the shared state is a long incident report followed by 50 questions. If every question were processed independently, the model would read the same report 50 times. If the state is encoded once, each question only has to process its own instructions and options and then look at the already-encoded state, which saves a great deal of compute.

Transformer models build up a cache as they process text, called the KV cache, which holds intermediate information left by tokens already processed. The author infers that Jev encodes the shared state into the KV cache first, then runs each question as a separate branch that reuses the same cache and computes only its own part.

The measurements fit. With a single question and a growing state, server time rose slowly from tens of milliseconds to about 218 ms at roughly 30,000 tokens. With a short fixed state and the number of questions raised from 1 to 1,500, time only climbed to a few hundred milliseconds. If each question re-processed the state, 1,500 questions could not finish that fast.

A cleverer test used a secret code. In one question the author wrote "The secret code for this request is ZEBRA-7741," and in another asked "What was the secret code mentioned earlier?" The second question couldn't see the code at all; its probability was 0.00. Moving the same sentence into the shared state pushed the probability to 0.90–0.92. Each condition was repeated five times with consistent results. So questions are isolated from one another, but all of them see the shared state. That also makes product sense: asking whether the customer is angry shouldn't sway which team the ticket goes to. Both questions look at the same evidence without interfering.

Jev has two hard limits: about 32,768 tokens per branch (the state plus one question) and about 65,536 tokens per request. The state counts only once toward the request limit, so a 23,000-token state with 5,000 questions still fits. The author suspects this reflects a packed sequence of up to 2^16 tokens, with the state placed once at the front and all questions laid out after it.

Options influence each other

This is the most interesting part. If the model scored each option independently and normalized with a shared softmax temperature, adding an unrelated option shouldn't change the ratio between two existing options, because the common denominator cancels out. That is a prediction you can test precisely.

The base question asked why a payment failed, with four options: bank, processor, customer, unknown. The author then added a fifth, completely irrelevant option: "caused by bad weather." He ran ten randomized blocks, each containing the four-option baseline, the five-option version, and matching controls. Pooled, the log-odds of customer versus unknown (the log of the ratio between the two probabilities) fell from +0.38 to +0.11, a mean change of −0.28. It dropped in all ten blocks, with a 95% paired t interval of roughly −0.36 to −0.19.

So the options are not scored independently; the model reads the whole list as one unit before deciding. The author infers a listwise approach: the full option list forms a single context, and the model combines everything at the final position before producing the distribution.

A "reference card" experiment backs this up. The author slipped a card reading "status = amber" into the options and asked the model to pick the option whose condition was met. With the card last in the list, the model was right 16 out of 16 times, with an average probability of about 0.88. With the card first, it got 12 right; in the middle, only 11. With the reference moved into the shared state, it was right all 48 times. More striking, changing only the value on the card, from amber to indigo, changed which of the earlier options won, even though their text was untouched. There must be an information path that lets later options affect the ranking of earlier ones. The author notes, though, that the experiment is position-sensitive and can't pin down the underlying attention mask on its own.

Calibration: can you trust the numbers?

Producing probabilities is easy; making them trustworthy is the hard part. That is calibration: if a model gives a batch of cases 80%, about 80% of them should actually be correct. TypeSafe calls its training method RLCD, Reinforcement Learning for Calibrated Decisions. Officially, it post-trains a pretrained language model with the goal of "System One task answers with epistemically honest probabilities," but the training recipe hasn't been published.

The author checked calibration on a 1,200-question sample of MMLU, a large benchmark spanning many subjects. He split the model's top probability into ten bins and compared each bin with its actual accuracy. The overall expected calibration error (ECE) was 0.0313, a decent figure. But 990 of the 1,200 questions landed in the 0.9–1.0 bin, so the model was highly confident on most of them.

He also tested freshly generated math problems:

The first two are well matched; on two-step word problems the model genuinely knows it isn't good. Modular exponentiation is the exception, where it clearly underestimated itself.

The API's confidence field is easy to misread. It isn't a second-order measure of the model's certainty; it's computed by formula. For choice questions, confidence = (p_max − 1/K) / (1 − 1/K), where p_max is the top probability and K is the number of options. With three options and a top probability of 0.8, confidence is 0.7. It measures how far the top probability sits above a uniform distribution, not an independently trained reliability score.

Is the backbone a mixture of experts? (the shakiest guess)

The author infers that Jev's backbone is a Mixture of Experts (MoE). An MoE model has a very large total parameter count, but each inference activates only a small subset of expert networks, with a router deciding which experts handle each token. You get the knowledge capacity of a big model without running every parameter every time.

His reasoning is that Jev only does prefill, reading the input in one pass, and never decodes token by token. Prefill is compute-bound, which is exactly where MoE saves work. MoE's weaknesses in autoregressive decoding, the memory-bandwidth bottleneck and the fact that most experts end up activated anyway, don't come into play. The numbers line up too: Jev handles about 30,000 tokens in roughly 160 ms. A dense 70B-parameter model on 8 H100s would need about a second, while an MoE with about 10B active parameters could match that speed. And most of the strongest recent base models, such as DeepSeek-V3, Qwen3 and GLM-4.5, are MoE.

The author calls this the least certain part of the whole analysis. Sparse experts can't be observed from outside, and specialized hardware could let a dense model reach the same speed. Importantly, nothing else in the analysis depends on the MoE guess. Swap in a dense model and the interface, shared state, isolated branches and probability readout all stay the same.

Scheduling, and small run-to-run differences

The last component is the serving engine. In the author's view, Jev schedules question branches as independent units of work rather than as a conversation turn. The branch suffixes can be packed into one batch that shares a single state representation, and application code maps the numeric results back to question identifiers and serializes them as JSON.

He also found that sending an identical request again returns slightly different probabilities, and that duplicated questions within one request can get slightly different answers. So you shouldn't assume the API is fully deterministic. That doesn't mean the model is generating or sampling text; numeric precision, dynamic batching and routing can all cause small fluctuations.

How far the conclusions go, and what remains open

The analysis covers jev-1.13.0 and roughly 10,000 API calls: 1,029 probe records, 6,800 benchmark records, and several hundred follow-up experiments. All requests and redacted responses are public. The author is candid about the limits. Direct probability output is TypeSafe's published claim. Question isolation and option-order effects are observable behavior. KV cache sharing, causal attention, the exact form of the readout head, and MoE are progressively less certain inferences.

Some questions can't be answered yet. Which base model is Jev built on? Its tokenizer doesn't fully match any of 192 public tokenizers. It is most similar to OpenAI's o200k but still differs, and the closest match is the Qwen family, which agreed on 348 of 415 test probes but still isn't identical. It's also unclear whether the gap between 84.6% accuracy on MMLU-Pro and the much weaker results on freshly generated math reflects exposure to training data or a difference in task structure.

If you work on decision tasks such as classification, routing, risk control or content moderation, the idea of a model that skips text and hands you probabilities is worth watching. From the outside, this investigation shows that Jev really doesn't generate token by token, that its questions are isolated from each other while sharing one state, and that its options are not scored independently. But calibration only holds near the training distribution. Before relying on it for your own use case, validate it on your own data.

Each section labels what is official, measured, or inferred. Source: Jev's Architecture Unmasked. A Japanese version is on the /ja/ page.