Qwen3-0.6B rolls a d20

Spurred on by this idea, I wanted to push it a little bit further and explore what happens when we try to roll a single-sided die (d1) through to a twenty-sided die (d20).

Fortunately, for us, the Qwen tokenizer treats letter A-T as single-tokens, so we don’t need to do anything fancy to map each token to a corresponding face (but I did check!).If you don’t remember why this is important, I recommend checking out my previous post about this.

We setup the problem as we did before: some state, a question, and then we list each of the options (corresponding to each face of the die), where we map each single-token letter to a possible outcome. We then ask the model to respond with only with the letter corresponding to the answer:

A fair 6-sided die is about to be rolled. Each face (1–6) is equally likely.

Which face will the die land on?

A: The die lands on 1
B: The die lands on 2
…
F: The die lands on 6

Respond only with the letter corresponding to the answer and nothing else.

Of course, we’d expect a fair die to have a uniform probability of 1/n for any side. But, as we saw previously, that isn’t necessarily the case for an LLM.

This is a little useless, but a lot interesting, so I want to explore this further to satisfy my own curiousity across a number of different scenarios.

Let’s look at the simplest scenario: each die’s faces are listed in order (1, 2, 3, 4, 5, 6, and so on…) and mapped to an incrementing corresponding letter (A: 1, B:2, C:3, etc.) which is also mapped to a single token in the LLM.

We then take the prompt, one for each d1 to d20, and do a single forward pass and take the softmax over the logits corresponding to each letter (which is mapped to a single token inside Qwen’s tokenizer). The softmax operation is what converts the logits values into a probability distribution.

Let’s plot the probability of each face for this arrangement, as well as the “doubling” error.

(You can toggle between the probabilities and the relative error using the buttons)

For this first example, we immediately see that the logits are heavily biased towards the first option: a classic case of “ordering bias” (a known phenomenon in LLMs).

To explore this, let’s see what happens if we reverse the order of the faces in the prompt, so that we list the faces from last to first (eg. T:20 -> A:1 for a d20):

Wow! Again, we clearly see some of the “positioning bias” at play: the final face of each die has a much larger share of the probability distribution (reminder: since we reversed the order of the die faces, they appeared first in the prompt).

Curiously, however, notice that the face corresponding to the letter A (face 1) still got a very large chunk of the probability mass across most dice. That’s peculier – is there something about the letter A, or the number 1, or both, that makes the model biased towards that token’s logit?

If we shuffle the order of the faces shown to the model, does this change things at all? Let’s shuffle each die’s face order 40 times, Although, if I had infinite compute, I would’ve loved to compute every possible variant, but I’m a little too impatient to wait for 20! possibilities to run :).and compute the average probability for each face, we get the following:

Randomizing the order gives us a slightly more uniform distribution (emphasis on slightly), but we still see that the A token (mapped to face 1) is still heavily biased by the model!

Maybe it has something to do with the joint association of the face number (1) with the first letter of the alphabet (A)? There might be a strong attentive association there. To investigate this, let’s keep the faces in the same order, but shuffle the letter assigned to each face (thus breaking the A->1 relationship). We again do this for 40 shuffles and compute the average predicted logit probablility, and get the following:

Shuffling the symbols and holding the faces still, the preference for the first option comes back. Face 1 is the mode on every die, despite that face no longer always being mapped to A.

This means there is a strong positioning bias at play with the first option that is presented to the model.

Let’s try shuffling both the order of the faces and the assigned letters. After yet another 40 shuffles per die, we get the following average:

Huzzah! That’s much closer to a uniform distribution – but we had to break both the positioning bias, and the joint association between A and 1. The numbered buttons are the forty shuffles, thinking off. Pick one and a die to see that menu in the order it was listed, each letter followed by the face it was bound to.

Qwen3-0.6B’s next-token logits are biased toward the first option, and toward the letter A. The ordered menu and the symbol shuffle both put the mode on the first line. The shuffled face order is the menu where A takes the mode instead. Shuffling the order and the letters together spreads both of those across the faces.

What happens if we turn on thinking?

Those six menus were scored with thinking turned off. Qwen3’s template then writes an empty think block, and the next token is the answer. With thinking left on, the assistant turn stops before that block. Almost all of the next-token probability sits on <think>. The letters hold essentially none of it. The heatmaps are the same letter softmax as above, taken at that first assistant token.

Listed in order, face 1 is still the first line.

Once F is on the menu, it takes the mode. The dice too small to list F put the mode on A, except the 5-sided die, which puts it on E.

The same menu, listed from the last face back to the first.

Listing the menu backward leaves F where it is. The mode stays on F for every die that lists it.

I shuffled the menu and took the mean softmax.

Shuffling the face order leaves the same spike. F stays the mode on every die that lists it.

The faces stay in order, 1 through n. The letters are reassigned onto those faces.

Redealing the letters, F is no longer fixed to face 6, and the face means spread out. No single face collects the mass the way the first option did with thinking off. The numbered buttons are the forty redeals. Pick one, then a die, and the letter-to-face mapping and the greedy thinking trace for that prompt appear under the chart. Each redeal still spikes, on the face that drew F.

With the think block left open, the next-token probability sits on <think>. The letter slice sits on F for every die that lists it, and shuffling the menu leaves that slice in place.

Thinking off again. No die, no menu, and no question. The user turn is the same 400 characters of punctuation and digits on every row. The heatmap is still the softmax over the letters that die would have listed.

The junk on its own leaves a spike. Mean entropy is 0.131 and mean distance is 0.806, near the bare ordered menu at 0.031 and 0.854. Dice with 2 through 7 sides put the mode on A. The 8-sided die splits A at 0.523 and H at 0.330. From 9 sides through 20, I holds 0.997 or more of the slice.

The same slice, with an empty system turn and an empty user turn.

An empty system turn and an empty user turn still flatten the small dice, and I still takes the large ones. Mean entropy is 0.342 and mean distance is 0.666. Dice with 2 through 4 sides put the mode on B. The 2-sided die is B at 0.598 and A at 0.402. From 5 through 7 sides the mode is E. The 8-sided die puts 0.656 on H. From 9 sides through 20, I holds 0.985 or more of the slice.

Written as JSON

The letter softmax is one forward pass over the menu. This pass asks the model to write the probabilities instead. Same menu, thinking off, one greedy generation per die, no shuffles. XGrammar masks any token that would break a JSON object whose keys are the letters on the menu and whose values are numbers in [0, 1).

About half the dice write a fair share on every face, and those numbers sum to one. The 6-sided die and the 20-sided die are in that group.

A 1-sided die writes a value well below one. The grammar’s upper bound excludes 1, so the fair value for that die is not a legal token sequence. A 3-sided die writes that same large value on each face, and the numbers sum to more than one. A 9-sided die repeats the 10-sided answer. An 18-sided die repeats the 6-sided answer on every face, and the row sums to far more than one.

A 12-sided die, a 14-sided die, and a 17-sided die start near a fair share and fall away toward the last faces. An 11-sided die starts near a fair share and then flattens. A 13-sided die is nearly fair, with a little extra on the first face.

The ordered menus took a second. The random text, twenty forwards, took about a second. The empty prompt took under a second. The face shuffle and the letter shuffle, 800 forwards each, took about 30 seconds (Apple MPS, float16). Shuffling both took about 27 seconds. With thinking on, the same two passes took about 36 and 42 seconds. The twenty JSON generations took about 59 seconds. The heatmap script is qwen_die_heatmap.py and the JSON script is qwen_die_json.py, both in other/scripts/die-heatmap/.

Appendix A: Results of every experiment

^† Normalized mean entropy. For a die with \(n\) faces and probabilities \(p_i\),

$$H = \frac{-\sum_i p_i \log p_i}{\log n}.$$

The column is the mean of \(H\) over the 2-sided through 20-sided dice. A fair row scores 1: the closer the values are to 1, the more uniform the distribution. A row that puts all of its mass on one face scores 0. We exclude the 1-sided die from this calculation.

^‡ Mean distance. The total variation distance from a fair die is

$$\mathrm{D} = \frac{1}{2}\sum_i \left|p_i - \frac{1}{n}\right|.$$

We mean the distance over all dice. It is the share of probability you would have to move to reach \(1/n\) on every face. 0 is fair: values closer to 0 are better. We exclude the 1-sided die from this calculation.

^‖ The returned JSON rows are not guaranteed to be a distribution. As a result, before scoring, we divide every row by its sum to normalize them to a distribution.