Jev and an LLM try to roll a fair die

Jev is interesting:

• It’s not an LLM
• It’s really fast (they’re doing parallel computation rather than autoregressive token-by-token prediction like an LLM).
• It outputs probabilities across questions, given some context.

In other words, Jev is a general-purpose classifier.

This is obviously a different use case from what we typically use LLMs for (even though we do use LLMs as expensive, slow classifiers all the time if we don’t care about – or are usually ignorant of – calibration). Jev is not meant to be a replacement for an LLM, but rather a fast, cheap substitute for taking well-calibrated actions.

But the curious bit is said plainly in the previous sentence: Jev is interesting because its output probabilities are calibrated.

To explain why, let’s take a brief detour into how we can trick LLMs into outputting probabilities.

Because, if you didn’t know, you’ve always been able to directly access the classification “probability” from an LLM (at least when running locally – some API providers like OpenAI do return these for you if you ask for them in the request).On the Responses API you need to pass include=["message.output_text.logprobs"] and a top_logprobs count. You also have to turn reasoning off, which the API requires (but does make intuitive sense too: if the model is generating hidden reasoning tokens first, those vary, and so do the logits that come after. Or maybe there’s a fear that this is a mechanism for extracting the proprietary reasoning tokens). The docs are here: OpenAI docs.

To predict the next token, on the forward pass over the input sequence, an LLM calculates a probability distribution over all possible output tokens. The next token is then sampled from this distribution. This probability distribution is determined entirely from the sequence of tokens provided as input via the forward pass.

If you squint, this is kind of like treating each token as a “class”. We can interpret each token’s probability as the model’s confidence in that token being the next token.

As a result, you can prompt an LLM as follows:

Given the following sentence, classify its sentiment:

"It is sunny today, and man does that make me happy!"

A: positive
B: neutral
C: negative

Respond with only one of the letters corresponding to the sentiment and nothing else.

Then, you can:

• Do a forward pass over the promptThis happens in parallel, by the way! Similar to Jev. .
• Extract the logits of the tokens corresponding to the choices (A, B, C)
• Calculate their softmax over just those letters (so the A/B/C numbers are forced to sum to 1, even if the model also wanted to emit something else).

Et voilà – you are now using an LLM as a general-purpose classifier. These are “among the option letters” probabilities, not P(the next token is A) over the whole vocab.

Let’s make this concrete with a small Qwen3-0.6B model (one forward pass, softmax over the option-letter tokens):

Classify the following email:

"Congrats! You've been selected for a $10,000 prize. Click here to claim your reward before it expires."

A: spam
B: not spam

Respond with only one of the letters corresponding to the answer and nothing else.

A customer wrote:

"I've been charged twice for the same order and nobody has replied in three days. This is unacceptable."

Where should this go?

A: billing
B: technical support
C: shipping

Respond with only one of the letters corresponding to the answer and nothing else.

A customer is requesting a refund:

"I bought this 11 months ago, used it daily, and now I want my money back because I found it cheaper elsewhere."

A: reject the refund
B: approve the refund

Respond with only one of the letters corresponding to the answer and nothing else.

(code is here)

Let’s ignore the accuracy of the logit trick for now (this is coming later). The point I’m trying to illustrate is that we can get classification probabilities directly out of LLMs.

Jev’s claim to fame is that the probabilities are calibrated.

When we say a model is “calibrated”, what we mean is that, if the model classifies something as class A with 90% probability, it will be correct 90% of the time.

A proper check needs lots of predictions, binned against how often they actually happen. A fair die is a looser proxy: if the calibration claim holds any water, asking for the probability of each face should come back close to 1/6 (0.167).

So let’s do that via OpenRouter (useful if, like me, you’re still on the TypeSafe waiting list to get first-party access to Jev).

{
  "model": "~typesafe/jev-latest",
  "state": "A fair six-sided die is about to be rolled. Each face (1–6) is equally likely.",
  "questions": {
    "1": {
      "type": "noul",
      "instructions": "Will the die land on 1?"
    },
    "2": {
      "type": "noul",
      "instructions": "Will the die land on 2?"
    },
    "3": {
      "type": "noul",
      "instructions": "Will the die land on 3?"
    },
    "4": {
      "type": "noul",
      "instructions": "Will the die land on 4?"
    },
    "5": {
      "type": "noul",
      "instructions": "Will the die land on 5?"
    },
    "6": {
      "type": "noul",
      "instructions": "Will the die land on 6?"
    }
  }
}

And here’s what we get back:

(code is here)

So on this proxy, that’s actually pretty close (especially if you think this is a general-purpose model that can do this across any prompt and number of questions).

If we were to compare this to the logit-calculated probabilities from a small local LLM (again using Qwen3-0.6B):

Read the state and answer with a single option letter.

State:
A fair six-sided die is about to be rolled. Each face (1–6) is equally likely.

Question: Will the die land on 1?

Options:
A. yes
B. no

Same prompt for faces 2–6. I’ll also ask the model to write the probabilities as JSON via next-token prediction, then parse whatever it generates.To guarantee valid JSON as output, we use constrained generation by way of XGrammar: at each step we mask out any token that wouldn’t continue as valid JSON with numeric values in [0, 1). We need to do this because a 0.6B model is a little bad at following instructions — left alone it writes 1 / 6, and a looser schema turns that into 1. Constrained decoding is a topic for another day — here it’s just a trick to keep the output parseable. The logit probability column is softmax of P(A) (which corresponds to the “yes” answer).

The JSON numbers aren’t class probabilities in the logit sense — they’re what the model wrote. That’s still how we’d typically use an LLM as a classifier: assume those outputted probabilities are a reliable proxy for class probabilities. (It’s also usually around 10x slower, since you’re generating token by token, whereas the logit trick is a single parallel forward pass.)

Six yes/no forwards: 0.225s. Six JSON generations: 2.757s. Jev, by comparison, is 0.460s — though that’s a round trip to OpenRouter, and the Qwen times are local (six separate forwards, no network).

(logits code is here; JSON code is here)

The logits are not calibrated at all! They’re super-biased towards saying YES to every question. If we take the JSON as a proxy for class probabilities, it does look like a fair die: {"A": 0.1667, "B": 0.8333} on every face. So it knows 1/6. That knowledge just doesn’t show up in the A/B logits.

But Jev seemingly isn’t quite calibrated when it comes to multiple choice

So done and dusted? Jev is well-calibrated and in the same ballpark as a forward pass on a tiny LLM?

Well, not quite.

There seems to be an unexpected failure mode.

The die example above was six independent yes/no questions (nouls).

But Jev also has a choice type: one question with several mutually exclusive options.

This is much closer to the A/B/C logits trick from earlier. So let’s ask for the same fair die as a single choice:

{
  "model": "~typesafe/jev-latest",
  "state": "A fair six-sided die is about to be rolled. Each face (1–6) is equally likely.",
  "questions": {
    "face": {
      "type": "choice",
      "instructions": "Which face will the die land on?",
      "criteria": {
        "1": "The die lands on 1",
        "2": "The die lands on 2",
        "3": "The die lands on 3",
        "4": "The die lands on 4",
        "5": "The die lands on 5",
        "6": "The die lands on 6"
      }
    }
  }
}

This time, Jev is not calibrated at all! It returns choice=1 with confidence 0.80:

(code is here)

A reminder again that a fair probability would be 0.167.

This looks a lot more like our LLM logit probability results from earlier.

For a fair comparison, let’s also run our LLM over a similar multiple-choice prompt (rather than the noul-equivalent six yes/no questions):

A fair six-sided die is about to be rolled. Each face (1–6) is equally likely.

A: The die lands on 1
B: The die lands on 2
C: The die lands on 3
D: The die lands on 4
E: The die lands on 5
F: The die lands on 6

Respond only with the letter corresponding to the answer and nothing else.

Qwen3-0.6B, logits (softmax over A–F) vs asking it to emit JSON:

(logits code is here; JSON code is here)

One forward pass: 0.042s. Generating the JSON object: 1.157s. Jev: 1.111s.

Same shape as Jev on the logits side, but still very uncalibrated. This is the “latching” we see with LLMs — not the independent-yes mess from the noul-style prompts.

This does make me curious about the underlying model that Jev is using. Is it trained on top of a small LLM? Maybe? It seems to suffer from the same “latching” behaviour. Or is it an entirely new architecture? Open questions for now.

What happens if we change the order of the statements?

Let’s reverse them: 6 first, 1 last.

{
  "model": "~typesafe/jev-latest",
  "state": "A fair six-sided die is about to be rolled. Each face (1–6) is equally likely.",
  "questions": {
    "face": {
      "type": "choice",
      "instructions": "Which face will the die land on?",
      "criteria": {
        "6": "The die lands on 6",
        "5": "The die lands on 5",
        "4": "The die lands on 4",
        "3": "The die lands on 3",
        "2": "The die lands on 2",
        "1": "The die lands on 1"
      }
    }
  }
}

Nope. Still choice=1, still ~82% on face 1:

(code is here)

So it’s not (just) picking the first option. It really wants 1. That’s quite weird! The shuffle seems to make no difference.

This implies that Jev computes nouls independently, but falls back to LLM-esque behaviour across choices.

What about for an LLM?

Same shuffle: A is now 6, F is now 1.

A: The die lands on 6
B: The die lands on 5
C: The die lands on 4
D: The die lands on 3
E: The die lands on 2
F: The die lands on 1

Qwen3-0.6B, logits vs JSON:

(logits code is here; JSON code is here)

The logits actually do move a bit here. Most of the mass still jumped to A (now face 6, rather than 1) – which is a common bias I see in LLMs.Usually called position bias: models often prefer the first MCQ option. Zheng et al. (2024) treat the broader pattern as selection bias (a prior on option IDs like A), with position as one piece of it.

So LLMs are no better. But (surprisingly!) Jev is no better either. At least not when using choice. So keep this in mind when building with Jev!

Aside: are bigger LLMs more calibrated?

Maybe Qwen3-0.6B is just too small to output accurate logits, and a larger model’s logit probabilities would actually look like a fair die? Let’s get the logits for OpenAI’s gpt-5.6 with reasoning={"effort": "none"} (logprobs over A–F) class=“sidenote-num”>OpenAI doesn’t return logprobs for many of its latest models. Astra won’t even take the parameter, and you can’t turn the reasoning off.:

A: The die lands on 1
B: The die lands on 2
C: The die lands on 3
D: The die lands on 4
E: The die lands on 5
F: The die lands on 6

On the 1–6 prompt, it actually does look a little more balanced (but still far from being calibrated)A 0.00 here can just mean the letter wasn’t in top_logprobs. The API lets you ask for up to 20; I asked for 20, and gpt-5.6 still only returned 4 tokens on 1–6 and 1 after the shuffle. Missing letters get treated as 0.:

(code is here)

Still not a fair die, but it’s not “always A” either. If we shuffle, though, we’re back to the same biases we saw earlier:

(code is here)

So, anyway

On noul, Jev’s pitch seems to hold water. Six independent “will it land on N?” questions about a fair die come back with a probability of ~0.15–0.16 each, in 0.460s, all in one go. That’s what you expect a calibrated classifier to output.

The LLM equivalent is the logit trick: one parallel forward pass, softmax over the option-letter tokens. The outputs are completely uncalibrated (unless we go for JSON and take those written numbers as a reliable proxy for class probabilities — which looks more accurate here, sure, but far slower).

Surprisingly (to me anyway), big models fare a little bit better, but still nowhere near as calibrated as Jev.

The catch seems to be everything that isn’t noul. If we reorganize the same request instead as a choice, Jev behaves more like an LLM again. Weird!

Regardless, now that Jev is on OpenRouter, I’ll be playing a lot more with it and seeing if I can find interesting use-cases. The demo on TypeSafe’s website is an excellent showcase, so do go check it out.

Appendix: every run

A fair die is 0.167 on each face. Independent yes/no (noul) probabilities do not have to sum to 1. Multiple-choice distributions do. Parentheses are the error vs 1/6.

Qwen times are wall-clock inference after a warmup pass, excluding model load (Apple MPS, float16). Jev times are warm OpenRouter round trips (network included). JSON runs are greedy generate with XGrammar constraining the output to a JSON object of option-letter keys and numeric values in [0, 1). No numeric example in the prompt. max_new_tokens is 96, except 192 for the questions choice run (it likes very long 0.1666… literals).

Independent yes/no — P(yes) per roll

Multiple choice — P(roll)

Each column is the die face, not the option letter or its position. roll 1 is P(lands on 1) on every row, including 6–1. JSON rows are uniform 0.167 after requiring decimal numbers in [0, 1).

gpt-6-astra is omitted: it rejects logprobs, so there is nothing to put here.