The game a language model plays

Better late than ___. You filled that in before finishing the sentence. A language model plays a higher-stakes version of the same game: given some text, it bets on the next piece, writes it, then bets again.

That piece is called a token. Here our tokens are whole words; the next lesson shows why real tokenizers cut text differently. First, play the game with the dumbest bookmaker in the business: a table of counts.

Observe a corpus become counts

The instrument holds a complete corpus of eight short sentences, small enough to count by hand. One rule builds the model: for each context, tally every word seen immediately after it.

“The cat” is followed by sleeps twice and eats once. After “the cat,” the table therefore bets 2-to-1 on sleeping. It has already priced the bet you are about to make.

Predict the winner

The corpus contains eat an orange, peel an orange, and eat an apple. The counting model has just read the word an.

Your bet

In this corpus, which word wins the bet right after “an”?

Commit before manipulating the instrument. This choice is final.

Probability lab / counting model

The whole corpus fits on screen. Every percentage is a count divided by a total: check it by hand.

Context length1 word2 words

Observed context

Live missions

Make the loop go silent
Make apple win the bet after “an”

The complete corpus

the cat sleeps on the mat
the cat sleeps under the table
the cat eats a mouse
the dog watches the cat
the dog sleeps on the mat
eat an (Observed context)orange (Next word)
peel an (Observed context)orange (Next word)
eat an (Observed context)apple (Next word)

Occurrences of the context are underlined; the counted word right after it is boxed.

Edit the corpus

Rewrite the sentences, one per line. Counts, bars, and the loop recompute live.

What follows this context

Next word given “an”
Next word	Count	Share
orange	2 / 3	66.7 %
apple	1 / 3	33.3 %

P(word | context) = count / total

The generation loop

The model rereads its last words, draws according to the shares, appends the word, and repeats.

die #1913

Manipulate the loop

Pick different contexts and watch the shares change. To compare longer context, select the, switch to 2 words, then choose the dog; it selects a different row, with a coin flip between watches and sleeps.

Select the context the, then press Draw a word several times. The model samples from the shares, appends its choice, rereads its latest context, and bets again.

That self-feeding cycle is autoregressive generation. An LLM runs the same loop over tokens: one bet at a time, conditioned on whatever pieces it can still see.

Now open Edit the corpus. Rewrite the eight sentences, one per line, and watch the counts, bars, and loop recompute. At first, orange beats apple 2-to-1 after “an.” One well-aimed sentence can turn the underdog into the favorite.

Challenge

Primary. Make apple win after “an” by editing the corpus. Explain which count changed and why that changed the odds.

Stretch. Keep drawing until the loop halts. Name the missing row that made the counting model go silent.

Compare your answer

Primary. Replace one “an orange” continuation with “an apple.” The counts become 2 for apple and 1 for orange. The odds move because the row changed, not because the table learned botany.

Stretch. The loop halts after a sentence-final word such as “mat,” “mouse,” or “orange.” That word has no continuation in the corpus, so the table has no row to bet from.

Explain the betting machine

The table implements one formula, visible under the bars:

P(word | context) = count / total

Read it as “the probability of a word, given the context.” The vertical bar marks the condition. Change the context and you may select different counts, so the distribution - the full set of shares - changes too.

That is why one table can continue “eat an” and “peel an” differently. No understanding is required here, only different rows of counts.

You read… an ___
The corpus countsorange: 2 · apple: 1
The model betsorange (66.7%)

The fatal flaw was hiding in the stretch mission: a context with no row, seen or unseen, leaves the bookmaker no counts and no odds to post. Longer rows can be more specific, but they are also easier to miss entirely.

A table of exact matches cannot generalize beyond its rows. The rest of the course builds a machine in which related tokens and contexts can share evidence instead of requiring identical text.

Prediction is not retrieval. This model did not look up an answer or consult a fact. It continued text from a conditional distribution.

A full LLM plays the same next-token game with far better machinery. It can therefore be fluent and false: the training target is fluent continuation, and nothing in that target checks for truth.

Reflect on what a percentage means

Checkpoint

The instrument shows “orange: 66.7%” after the context “an.” What does that number actually report?

From broad regularities of English usage, the model estimates that “orange” is the continuation a fluent speaker would most often produce.Two of the three observed continuations of “an” in this corpus are “orange.”The model has understood that the topic is food.

Pick an answer first.

Checkpoint

Select “the,” switch the context length to two words, then choose “the dog.” What happens to the bet?

Nothing changes: “dog” only adds detail a table already keyed on “the” can ignore.A longer context always drives one word to 100%, because more words mean more certainty.The distribution changes: “the dog” selects a different row, which happens here to contain two continuations.