In this lesson
Logits, temperature, and the token that comes out
The final vector becomes a bet over the whole vocabulary, a decoding policy picks the token - and the untrained toy model has a surprise in store.
By the end, you can
- Trace the path from the last output vector to logits, probabilities, and a sampled token.
- Distinguish what the model computes from what the decoding policy chooses.
- Demonstrate how a finite context window removes early tokens from later forward steps.
Before you start: The Transformer block, the floor a model stacks
At the top of the tower sits one output vector - two numbers, in our toy. It comes from the last position, the only causal stream that has seen the whole context. And a reader is waiting for a word.
The answer hides in a table you already used, read backwards. This is the step chapter 6 left open: turn the vector into a bet over the whole vocabulary, then place the bet.
It is also the chapter where our hand-built toy gets caught pretending to know English.
Observe the table read backwards
To score a word, the model takes a dot product between the context vector and that word’s embedding row from chapter 3. Lookups on the way in, scorers on the way out: the same rows, read the other way.
Many production models share those two tables. This common design choice is called weight tying; the studio uses it so one row can reveal both jobs.
From here on, weights means learned model numbers. Chapter 4’s attention weights were different: temporary shares recomputed for each sentence.
Each raw vocabulary score is a logit. After the forward pass, the decoding runtime can adjust temperature; softmax - chapter 4’s accountant - turns logits into shares of 100%. Whether they land anywhere sensible is another question.
Predict the toy’s first word
The context is the course trace, “eat an”. Every number in this machine was written by hand for teaching, and none of it was ever trained.
Your bet
After “eat an”, which noun does our hand-built model actually bet on?
Commit before manipulating the instrument. This choice is final.
The top of the tower produces a vector; the reader wants a word. Watch the vector become a bet, then let the decoding policy choose.
Adjust k and p
Live missions
- Break the cat loop with a non-greedy policy
- Move the tied row until orange wins
From vector to bet
Current context → output vector h = [3.661, 1.974]
| Word | Logit h · E[word] | Share (after T) | Final share |
|---|---|---|---|
| cat · #314kept | 4.424 | 46.1 % | 100.0 % |
| dog · #271cut | 4.052 | 31.8 % | 0.00 % |
| queen · #409cut | 2.993 | 11.0 % | 0.00 % |
| king · #408cut | 2.943 | 10.5 % | 0.00 % |
| bicycle · #734cut | -0.864 | 0.23 % | 0.00 % |
| car · #733cut | -0.886 | 0.23 % | 0.00 % |
| orange · #901cut | -1.621 | 0.11 % | 0.00 % |
| apple · #612cut | -2.224 | 0.06 % | 0.00 % |
The loop, for real
Every draw reruns everything: causal block → logits → policy → token appended to the context.
eatan
Greedy policy: the die is ignored, the maximum always wins.
Manipulate the policy
There it is: cat, at roughly 46% - and orange near the bottom at 0.1%, with only apple faring worse. Hold that 0.1% in mind. Chapter 8 will return to it.
Slide temperature down to 0.5 and read the Share (after T) column: the winner’s share balloons. Raise it to 2 and that distribution flattens; underdogs breathe.
Watch the ranking: it does not move. Final share stays at 100% for cat throughout, because the policy below is still greedy and keeps exactly one candidate.
Now compare policies. Set Decoding policy to Greedy (argmax): it always takes the maximum. Draw a few tokens and the toy writes “eat an cat cat cat…” - the kind of repetition called degeneration.
Top-k and top-p cut the tail before drawing; full sampling cuts nothing and draws from the whole distribution. The die carries a number, and that number is the seed: same seed, same text.
Move orange’s two coordinates. Its input lookup and output scorer are the same tied row, so this control can change the ranking itself.
Make orange win, then reset it. In chapter 8, training makes that same move on its own, from examples rather than by hand.
Shrink the context window to two tokens and draw again. Early chips fade as new ones arrive. They remain in the conversation display, but the next forward pass receives only the visible tail.
Production chat systems also run on finite context budgets, so their wrappers may truncate, summarize, or retrieve older material. A long conversation does not become permanent memory in the weights.
Generation needs a stopping mechanism too. A model can bet on a learned end-of-sequence token. The outer application can also stop at a token budget, a delimiter, a tool call, or a safety rule.
The model supplies one-token bets. The orchestrator - the application code around it - decides whether to request another.
Challenge Primary mission: make orange top the table. Try temperature first, then move orange’s tied row. Explain why only one control can change the ranking.
Optional stretch: stop the “cat cat cat” loop by selecting a sampling policy. Name what changed - the model’s logits or the runtime’s decoding rule.
Compare your answer
Temperature divides every logit by the same positive number. It changes the gaps and the probability shares, but never their order. Orange cannot reach the top that way.
Dragging orange changes its tied row, so its logit can overtake cat’s. Switching away from greedy changes only the runtime policy: the model’s logits stay put, but another token can win a sample.
Explain the division of labor
The pipeline has a strict division of labor:
- The model computes: context vector → logits, one raw score for every word. Here, the learned weights shape its current bet.
- The decoding runtime acts next: it adjusts temperature, turns logits into probabilities with softmax, then applies a policy - greedy, top-k, top-p, or full sampling - without modifying the model.
Temperature belongs to the second half. One positive divisor, applied to every logit, stretches or shrinks the gaps between them. It reshapes the bet; it cannot reorder it.
Chapter 4’s score-sharpness slider was a teaching lens we bolted onto attention. This one ships in production: generation temperature.
- The tower outputsh = one vector
- The table scoreslogit(word) = h · E[word]
- The policy picksnext token → back into the context
Now the cat. Why does a machine with every mechanism in place bet on cat after “eat an”? Because mechanisms alone carry no learned regularities.
Every matrix in this fixture was authored to be legible, not to be right. The chapter 1 table, crude as it was, at least learned from data. Our beautiful tower has seen none.
Working machinery and a good bet are separate achievements.
Scale check. This toy bets over 8 nouns in 2 dimensions. A production model may bet over roughly 100,000 tokens represented in thousands of dimensions.
Its function words live in the same vocabulary as everything else. The path you traced survives; scale and training supply vastly more numbers.
Reflect on what temperature can never do
Checkpoint
Greedy decoding picks “cat” here at every temperature. Why can't temperature change that?
Pick an answer first.
Checkpoint
After several draws, the first token fades outside a two-token context window. What has actually happened?
Pick an answer first.
Connect the loop, then break the spell
The loop is closed: text → tokens → vectors → attention → block → logits → probabilities → chosen token → back into context.
Yet the toy still says “eat an cat.” Every mechanism works; its numbers are simply bad. So one question remains: where do good weights come from?
Dragging one row fixed one word; a production vocabulary has roughly a hundred thousand rows. Training takes over in the next chapter, wrong bet after wrong bet, until this same wager flips to orange.
Sources and scope
- Press & Wolf (2017) shows that sharing the embedding and unembedding weights - the tying this lab makes visible - improves language models.
- Holtzman et al. (2020) documents degeneration under greedy and beam decoding and proposes nucleus (top-p) sampling.
- The vocabulary is the eight handcrafted chapter 3 rows; the prefix’s function words exist only as input fixtures. No number here comes from a trained model - that gap is the point of the chapter.
- Content and claims reviewed on July 28, 2026.