In this lesson
Training, the machine that fixes its own numbers
Loss, gradient, small step, repeat. Free exactly two numbers, watch gradient descent drag a row of the map, and see chapter 7's absurd bet flip.
By the end, you can
- Explain training as loss measurement, gradient computation, and repeated small parameter updates.
- Explain why learned geometry serves prediction rather than human intuition.
- Compare exact counting, backed-off counting, and a shared neural representation on an unseen context.
Before you start: Logits, temperature, and the token that comes out
Nobody wrote the numbers in a real language model. Authors chose the structure; training set every value. In chapter 7, our lovingly hand-built tower, given “eat an,” bet 46% on cat.
Can a computed update repair that bet?
The loop has three moves: measure the error with a loss, compute the gradient, then update the parameters. Repeat.
Observe what gets to move
A parameter is an adjustable number. To keep the update visible, this lab freezes the tower and frees exactly two: the coordinates of orange’s embedding row.
The context vector h for “eat an” stays fixed. The loss prices the bet, the gradient points uphill, and the update steps the other way: a pull on the row. Everything that follows hangs on the strength of that pull.
Predict the shape of learning
The loss starts at 6.8 - the price of giving the right answer roughly a 0.1% chance.
Your bet
At learning rate 0.1, what will the loss curve look like as you train?
Commit before manipulating the instrument. This choice is final.
The whole tower is frozen except two numbers: orange’s row in the chapter 3 table. Watch gradient descent drag it.
The model’s bet
| Word | Share |
|---|---|
| cat | 46.1 % |
| dog | 31.8 % |
| queen | 11.0 % |
- Loss −log p(target)
- 6.819
- p(target)
- 0.1 %
The model still bets wrong. Keep training.
Loss curve
The map, during training
Seven rows stay fixed; orange’s row (in orange) is dragged step by step. The trail shows its path.
cos(orange, cat) -0.272
Chapter 1 promise
Generalization lab
Same training corpus, new context. All three methods must bet on “peel an,” a row none of them saw.
Visible training corpus
eat an → orange × 6draw an → apple × 3spot an → dog × 2drive a → car × 4ride a → bicycle × 4pet a → cat × 3
Held-out test
peel an → orange ?never shown during trainingExact counting
Looks up the whole row.
No row. No bet.Backed-off counting
Drops the first word and counts after “an.”
- p(orange)
- 54.5 %
- rank
- #1
- examples used
- 11
Tiny neural model
Shares one 2D map across contexts.
- p(orange)
- 96.5 %
- rank
- #1
- examples used
- 22
Exact counting has no row. Backoff recovers a weaker clue. Near “eat,” the shared representation ranks orange first.
Live missions
- Flip the bet to orange
- Make transfer lose its useful clue
Real computation, handcrafted 2D map. The softmax classifier learns its weights by gradient descent on the 22 visible examples. A real model learns its representation too, across thousands of dimensions.
Inspect the gradient
| Context vector h (frozen) | [3.661, 1.974] |
|---|---|
| Pull strength (1 − p) | 0.999 |
| Current gradient | [-3.657, -1.972] |
| Current orange row | [-0.680, 0.440] |
gradient = −(1 − p(target)) × h; update: row ← row − rate × gradient
Manipulate the descent
Train ten steps at rate 0.1, then read the curve: this settles your bet. Watch the loss fall, p(orange) rise, and the orange row move.
The same computed gradient drives all three displays. Within a few steps the top prediction flips; catch the exact step.
Reset and compare the computed paths. Rate 0.01 needs 38 steps to cross the lab’s threshold; rate 1 crosses in one. Across these three runs one rule holds. Bigger step, earlier crossing.
Rate 5 is about to complicate it. Fair warning first. That path is hand-authored, not produced by this two-parameter gradient calculation.
Select it and read the oscillating curve as an illustration: a step too large for its landscape can fly past the bottom instead of settling into it.
Optionally, switch to Beat the optimizer and move the row yourself. A few hand-placed moves make the computed direction concrete; they add nothing to the evidence for gradient descent.
Challenge
After the top prediction flips, keep training and watch the pull strength (1 − p) in the inspector. Explain why the updates shrink as the model assigns more probability to the target.
Compare your answer
The gradient magnitude contains (1 − p). When the target has little probability, that factor is large. As p approaches 1, the factor approaches 0.
With a fixed learning rate and context vector, the shrinking factor makes each later update smaller.
Inspect the final map. The row for orange ends beside cat and leaves its former fruit neighbors. Explain which objective moved it, and why human categories did not constrain that move.
Explain the three-step loop
For this one target, the cross-entropy loss is −log p(target). It approaches zero as the target probability approaches 1 and grows without bound as that probability approaches 0.
Giving the target 0.1% costs about 6.9; giving it 90% costs about 0.1. Confident mistakes pay the steepest price.
Over many bets, in the ideal case, this loss rewards calibrated probabilities - honest odds, where a stated 70% comes true 70% of the time. Whether any trained model earns that is a separate question, and one trace cannot settle it.
The gradient of the target row, with every other number frozen, has a formula short enough to read aloud:
gradient = −(1 − p(target)) × h
Math pause: a gradient as a slope arrow
A gradient points toward faster increase in loss. Gradient descent subtracts it, so the update moves downhill.
Here the sign makes the update move the target row toward h. The factor (1 − p) makes the pull large after a surprise and small after a confident success.
So the pull has a direction and a strength: here it drags the target row toward the context vector, scaled by (1 − p) and the learning rate. A surprised model pulls hard; a confident one barely tugs.
The update is row ← row − rate × gradient. Recomputing the loss and gradient after each update produces the curve you observed.
- Measureloss = −log p(orange)
- Differentiategradient = −(1 − p) × h
- Step and repeatrow ← row − rate × gradient
A full training step runs backpropagation, the chain rule walked backward through the network, and out comes a gradient for every trainable parameter, measured over one batch of examples. An optimizer turns those gradients into updates.
The two-parameter formula is specific to this frozen output stage. The general loop survives at scale: measure a batch loss, backpropagate its gradients, update, and repeat.
In this fixture, optimizing one context moves orange beside cat: their cosine climbs from −0.27 to 0.998. The fruit neighborhood was not consulted. The loss rewards the prediction, not a human taxonomy.
At real scale, many examples pull on the same rows at once, and the geometry that survives is the compromise that serves the loss. Here you watched one pull, deliberately alone.
This two-parameter loss has no finite minimum along its improving direction, so its computed path does not overshoot. The oscillation at rate 5 is authored, as disclosed before the control.
Oversized steps can destabilize neural-network training. Optimization research supplies that claim; this lab merely borrows it.
Scoped transfer check.
The lower panel switches to “peel an,” a held-out context: it was kept out of training. All three methods receive the same 22 context-and-next-token examples, and none contains it.
Exact counting has no row. Backoff discards peel and gives orange 6 of 11 suffix counts, enough for first place but only weak evidence. Neither shares evidence through a learned representation.
The neural toy places peel near eat. Shared weights reuse what it learned from “eat an → orange” and give orange more than 90%, far stronger than suffix backoff.
Move peel away from eat. The weights remain fixed, but the prediction changes. Within this toy, representation geometry determines which contexts can share evidence.
The demonstration is scoped: shared representations let nearby vectors lend each other evidence. Nothing here says the neighbors deserve that trust, and softmax will not check for them.
Reflect on what generalization needs
Checkpoint
Backoff and the neural toy both rank orange first for unseen “peel an.” Why is the neural toy's probability much stronger?
Pick an answer first.
Checkpoint
After training, orange's row sits right beside cat and far from the fruit it used to neighbor. What does that reveal about where the learned map's shape comes from?
Pick an answer first.
Connect the repaired machine to the real ones
The toy now ranks orange first after “eat an.” The chapter 7 bet is settled: loss priced the mistake, the gradient chose the direction, and small repeated steps did the rest.
At model scale the same three moves run over billions of parameters at once, and none of the three comes cheap.
The next lesson asks how parameters, training tokens, and compute grow together, and what the bill comes to.
Sources and scope
- Rumelhart, Hinton & Williams (1986) describe backpropagation and its use to adjust a network’s weights.
- Bengio, Ducharme, Vincent & Jauvin (2003) jointly train distributed word representations and a next-word probability model with gradient-based optimization.
- This lab differentiates two free parameters analytically; no backpropagation through the tower is performed, and the convex-bowl behavior of the learning rates does not generalize to real loss landscapes.
- Content and claims reviewed on July 28, 2026.