Softmax, Sampling, and Temperature

How a language model turns scores into a choice — and what that choice does and does not tell you

Prerequisites: basic probability and algebra; no machine-learning background assumed. A little familiarity with statistics or discrete-choice modeling helps with one optional section, but is not required.

1. The problem being solved

At each step of generating text, a language model must choose one token from a fixed, finite vocabulary. Complete texts are open-ended only because that choice is repeated, one token at a time, with each chosen token added to the context before the next choice. Two requirements have to be met at the same time:

  • the model needs a complete probability distribution over every token in the vocabulary, not just a single best guess, because a single guess cannot express how much probability the model assigns to each candidate, or represent the fact that several continuations may be almost equally plausible;
  • that distribution has to be built from whatever internal scores the network happens to produce — plain real numbers, positive or negative, large or small, with no built-in guarantee that they are nonnegative or sum to one.

Softmax is the fixed function that solves exactly this problem. It takes any list of real-valued scores and turns it into a valid probability distribution — every entry nonnegative, all entries summing to one — while preserving the order of the scores.

2. A small next-token example

Consider a language model partway through the sentence:

“Given the traffic data, the agency decided to ___ the bridge.”

Pretend, for this toy example, that the model’s entire vocabulary contains only these five tokens:

repair, monitor, replace, close, ignore

The model’s job at this position is to assign a probability to each of the five that reflects how plausible a continuation it is, given everything in the context so far. A real model has a much larger vocabulary; the arithmetic below is the same, just with a much longer list.

3. Pre-softmax scores, and what “logit” means here

Before softmax, the model produces one raw score for each possible next token — a plain real number that can be positive, negative, large, or small, with no requirement that the full set of scores be nonnegative or sum to anything in particular. Conventionally the score for candidate \(i\) is written \(z_i\). For the toy vocabulary in §2:

Token Score
repair 2.0
monitor 1.0
replace 0.5
close −0.5
ignore −3.0

These five numbers are not yet probabilities. What softmax uses is their differences, not their absolute values — the next section makes that exact.

Machine-learning literature usually calls these raw scores logits. This use is related to, but not identical with, the statistical logit \(\operatorname{logit}(p) = \log\!\big(p / (1-p)\big)\), which takes a probability you already have and returns a real number. In a multiclass softmax, an individual score \(z_i\) is not \(\operatorname{logit}(p_i)\). The exact relationship becomes clearer once we see the softmax equation. [S-09]

4. The softmax equation

Softmax turns the list of scores \(z = (z_1, \ldots, z_V)\) from §3 into a probability distribution over all \(V\) candidates — in the toy example, \(V = 5\):

\[ p_i = \frac{\exp(z_i / T)}{\sum_{j=1}^{V} \exp(z_j / T)} \]

For now, set the temperature \(T = 1\) — §6 below is entirely about what \(T\) does. Three properties of this equation are worth internalizing right away:

  • every \(p_i > 0\). Softmax itself never assigns zero probability to a candidate with a finite score, however low that score is.
  • softmax is invariant to a constant shift. Adding the same constant \(c\) to every score leaves every \(p_i\) unchanged, because \(\exp(z_i + c) = \exp(c)\exp(z_i)\), and the \(\exp(c)\) factor cancels between the numerator and the denominator. Scores have no natural zero point; only their relative values matter. [S-01]
  • differences of scores are log probability ratios. For any two candidates, \(\log(p_i / p_j) = (z_i - z_j)/T\), which at \(T = 1\) is simply \(z_i - z_j\). A score gap of 1 between two tokens means the higher one is \(e \approx 2.7\) times as probable, whatever the other scores are. This is why the magnitudes of the gaps matter, not just the ranking. [S-01]

Applying this to the five scores from §3 (still at \(T=1\)) gives \(p \approx (0.60, 0.22, 0.13, 0.05, 0.004)\) for (repair, monitor, replace, close, ignore) — nonnegative, summing to one, and in the same order as the raw scores. With a real vocabulary, every additional token’s score enters the sum in the denominator, so the probabilities of these five would all be smaller than shown here.

The name contrasts softmax with a hard winner-take-all choice based on the argmax — selecting the single top-scoring token and giving it everything. Softmax returns a whole distribution instead; as \(T \to 0\) that distribution approaches the argmax’s all-or-nothing choice (derived in §6). It is also smooth, so gradients pass through it during training (see Neurons, Layers, and Learning for why that matters), which the hard choice does not allow. [S-10]

5. The connection to multinomial logit (optional bridge)

This section is a mathematical bridge for readers who already know discrete-choice models from statistics or economics. It is not required to understand the rest of this page, and the main teaching thread continues in §6 if you skip it.

If the softmax equation in §4 looks familiar, it should to anyone who has used a multinomial or conditional logit model: given a list of scores, exponentiate and normalize — the same recipe. The role the score \(z_i\) plays here is the same role a linear index \(x_i'\beta\) plays in a conditional-logit choice model. [S-02]

The third property in §4 is the familiar odds-ratio property associated with independence of irrelevant alternatives (IIA): for a fixed set of scores, the ratio of any two probabilities depends only on those two scores. It holds here algebraically, exactly as it does in a standard conditional-logit model. [S-03] This identity also shows the connection to statistical log-odds: with exactly two candidates, it reads \(\operatorname{logit}(p_1) = \log\!\big(p_1/(1-p_1)\big) = (z_1 - z_2)/T\), and only at \(T = 1\) does this reduce to \(\operatorname{logit}(p_1) = z_1 - z_2\). [S-09]

The resemblance stops there, and it is worth being precise about where.

The language model as a whole is not a multinomial-logit model. [S-02] Only the final step — turning scores into probabilities — shares multinomial logit’s arithmetic. Everything upstream of that step (the network that produced the scores) is not linear, low-dimensional, or estimated the way a discrete-choice model’s index function is.

The algebraic property is not a behavioral one. The odds-ratio property should not be given a behavioral, random-utility, or welfare interpretation here, because the scores are outputs of a large, context-dependent network rather than estimated utility indices with structural meaning: \(z_i\) changes with every token added to the prompt, and no structural model of “utility” is being estimated. [S-03] Softmax borrows multinomial logit’s arithmetic, not its econometrics.

6. Temperature and sampling

Temperature \(T\) rescales every score from §3 — dividing by \(T\) — before the softmax step in §4 is applied. Lower temperature concentrates more probability on the highest-scoring tokens. Higher temperature flattens the distribution and gives lower-scoring tokens relatively more probability. For any \(T > 0\) the order of the tokens is unchanged. The name comes from the Boltzmann distribution in physics, which has the same form with a score playing the role of a negative energy. [S-11]

Two different questions get asked about the same probability distribution from here on, and it matters which one you’re asking:

  • Argmax is deterministic and introduces no variation; sampling can produce different continuations from the same probability distribution. Argmax asks “what is most likely?” Sampling asks “draw one token according to these probabilities” — a genuinely random step.
  • As \(T \to 0\), softmax converges to argmax when there is a unique highest score — probability mass concentrates entirely on the top candidate. (If two or more scores are exactly tied for the maximum, the limiting distribution instead splits mass among the tied candidates; with real-valued network outputs, exact ties are rare in practice.) [S-04]
  • As \(T \to \infty\), the distribution approaches uniform across all candidates — every token becomes roughly equally likely regardless of its score. [S-04]
  • Changing \(T\) leaves the model’s underlying parameters completely untouched. The same trained model, given the same prompt, can produce different continuations on different runs purely because of how its output distribution is sampled — not because anything about the model itself changed. [S-07]

7. The interactive

The laboratory below uses the same five scores from §3. Move the sliders, change the temperature, and compare a single draw against 100 or 1,000 draws. The empirical frequencies (red markers) should track the theoretical probabilities (blue bars) more and more closely as the number of draws grows — the same law-of-large-numbers behavior that governs any repeated random draw from a fixed distribution.

8. A common misconception

Lowering the temperature does not make answers more truthful.

Temperature changes how concentrated the output distribution is; it does not check whether a continuation is true. A model can assign high probability to unsupported or false text, and deterministic decoding — \(T \to 0\), always taking the argmax — can select it too. Sampling changes variability across runs, not grounding. [S-08]

9. Check your understanding

If you add the same constant \(c\) to every score, how does the probability distribution change? It does not change. Softmax is invariant to a constant shift applied to every score (§4): the factor \(\exp(c)\) introduced by the shift appears in every term of both the numerator and the denominator and cancels exactly.
As \(T \to 0\) with a unique highest score, what does the distribution converge to? A point mass on the single highest-scoring token — the distribution converges to argmax (§6).
(Optional bridge, §5.) Why shouldn’t a score \(z_i\) be called “the utility of token \(i\),” the way \(x_i'\beta\) is called a utility index in a discrete-choice model? Because \(z_i\) is the output of a large, context-dependent neural network, not a linear index in a small set of observed covariates with estimable, interpretable coefficients. Treating it as a utility index imports a behavioral or welfare interpretation the model was never estimated to support (§5).
True or false: lowering the temperature to make the model’s output distribution more concentrated also makes that answer more likely to be true. False. Temperature controls how concentrated the output distribution is over continuations the model already considers plausible; it says nothing about whether the training process ever checked those continuations against truth. Deterministic decoding can also produce unsupported answers (§8).

10. Continue learning

  • 3Blue1Brown, Transformers, the tech behind LLMs (video; ~5 min for the relevant section, starting at 22:22, “Softmax with temperature”; full video is part of a longer series). Introductory/visual level. A clear animated account of why softmax needs a normalizing sum and how temperature reshapes it — the visual counterpart to §4 and §6 above. Limitation: does not discuss the multinomial-logit connection or its limits (§5 above). https://www.3blue1brown.com/lessons/gpt

  • Andrej Karpathy, Deep Dive into LLMs like ChatGPT (video; the segment beginning at 26:01, “inference,” runs roughly 5 minutes; the full video is 3.5 hours). Introductory level, plain language. Shows sampling in the context of a full autoregressive generation loop. Limitation: no equations — purely conceptual; pair it with §4–§6 above for the mathematics. https://www.youtube.com/watch?v=7xTGNNLPyMI&t=1561s

See Resources for the complete list of sources used in this part of the course, with full source-map entries.