In this lesson
The game a language model plays
No neurons yet. Count what follows each word in a tiny corpus, bet on the next one, and run the generation loop by hand.
By the end, you can
- Explain next-token prediction without crediting the model with intent or lookup.
- Describe how a longer context changes the probabilities of what comes next.
Better late than ___. You filled that in before finishing the sentence. Your phone’s keyboard plays the same trick above the space bar, and a large language model plays it too - at every single step. That is the entire game: given some text, bet on what comes next, write it down, and bet again. Everything else in this course is machinery for playing that game well.
Before touching a single neuron, it is worth playing the game with the dumbest machine that can play it at all: a table of counts.
Observe a corpus become counts
The instrument below holds a complete corpus of eight short sentences - small enough to read in ten seconds and to count by hand. From it, one rule builds a model: for each word, tally every word that was ever seen right after it. “The cat” is followed by sleeps twice and eats once, so after “the cat,” the table bets 2-to-1 on sleeping.
Predict the winner
The corpus contains eat an orange, peel an orange, and eat an apple. The counting model has just read the word an.
Your bet
In this corpus, which word wins the bet right after “an”?
Nobody grades this. Commit, and let the instrument settle it.
The whole corpus fits on screen. Every percentage is a count divided by a total: check it by hand.
The complete corpus
- thecatsleepsonthemat
- thecatsleepsunderthetable
- thecateatsamouse
- thedogwatchesthecat
- thedogsleepsonthemat
- eatanorange
- peelanorange
- eatanapple
Occurrences of the context are underlined; the counted word right after it is boxed.
What follows this context
| Next word | Count | Share |
|---|---|---|
| orange | 2 / 3 | 66.7 % |
| apple | 1 / 3 | 33.3 % |
P(word | context) = count / total
The generation loop
The model rereads its last words, draws according to the shares, appends the word, and repeats.
an
Manipulate the loop
Pick different contexts and watch the shares change. Then switch the context length from 1 word to 2 words: the alone is vague, but the dog narrows the future to a coin flip between watches and sleeps. A longer context is information, and information reshapes the bet.
Now run the loop. Press Draw a word: the model samples a word according to the shares, appends it, rereads its own last words, and bets again. That self-feeding cycle is called autoregressive generation, and it is exactly how an LLM writes: one bet at a time, each bet conditioned on everything written so far.
Challenge Two missions. One: find a context whose winning share is exactly 100 %, then explain why that certainty is a fact about this corpus, not a fact about English. Two: keep drawing words until the loop halts, and say precisely why the counting model went silent.
Explain the betting machine
The table implements one formula, visible under the bars:
P(word | context) = count / total
Read it as “the probability of a word, given the context.” The vertical bar is the whole trick: probabilities are always conditioned on something, and changing the context changes the distribution. This is why the same model can continue “eat an” and “call an” differently - no understanding required, just different rows of the table.
- You readeat an ___
- The corpus countsorange: 2 · apple: 1
- The model betsorange (66.7 %)
But you also found the fatal flaw. The moment a context was never observed - even once - the table has no row, no counts, and no opinion. Longer contexts make this worse: with two-word contexts, most word pairs that could exist never occur in any corpus, however large. Counting cannot generalize. The rest of this course builds the machine that can: it will turn words into geometry so that similar contexts can share evidence, instead of requiring identical ones.
Prediction is not retrieval. The model did not look up an answer or consult a fact. It continued text according to a conditional distribution. When a full LLM answers a question, it is playing this same game with vastly better machinery - which is why it can be fluent and wrong at the same time: fluency is the game; truth is not the scoring rule.
Reflect on what a percentage means
Checkpoint
The instrument shows “orange: 66.7 %” after the context “an.” What does that number actually report?
Connect the game to the machine
You now know the game: predict, append, repeat. You also know why a counting table loses it - unseen contexts leave it speechless. To play better, a model must first stop treating words as indivisible symbols. The next chapter starts exactly there, with a question that sounds absurd until you meet a tokenizer: how many r’s are in strawberry?
Sources and scope
- Shannon (1948) introduces n-gram models of text and the generation-by-sampling experiment this lesson reenacts.
- Bengio, Ducharme, Vincent & Jauvin (2003) states the n-gram generalization problem and proposes learned word vectors as the remedy - the road this course follows.
- The eight-sentence corpus is an original teaching fixture; its counts are designed to be checkable by hand, not representative of real language statistics.
- Content and claims reviewed on 2026-07-18.