In this lesson
Tokenization, or how a model reads
Neither letters nor words: a model receives tokens. Bet on the best split, then put it to the test.
By the end, you can
- Explain why a model turns text into tokens before any computation.
- Compare the trade-offs between characters, words, and subwords.
Before you start: The game a language model plays
How many r’s are in strawberry? A language model can produce pages of fluent prose and still lose this tiny spelling bet. The puzzle begins before the first neural layer, in the cutting room.
A tokenizer cuts text into tokens and maps each token to an ID. Those IDs are the network’s explicit input units, not a parallel row of letters. If strawberry is one token, its three r’s are not separately exposed at input.
That does not make letter counting impossible; later computation may recover spelling clues. The cut still determines the raw pieces and the bill: how many token positions later layers must process.
Observe three cuts and their bills
Take “eat an orange.” The cutting rule could work at several scales:
- letter by letter:
e,a,t,a,n,o… (eleven units for three words); - word by word:
eat,an,orange(three units, case closed); - or something in between: pieces of words.
The middle cut looks strange. Does it buy the best balance? Place your bet.
Predict the winning split
Three candidates: characters, words, subwords. The mode producing the shortest sequence is easy to guess. The real question lies elsewhere: what will each mode do with the invented word orangebike, a 👋, or 3.14?
Your bet
Which mode covers orangebike without losing its spelling or stretching the whole sentence?
Commit before manipulating the instrument. This choice is final.
Live missions
- Raise the characters-per-token ratio above 3 in Subwords
- Use an emoji or invented word to drive the ratio below 1.5
The same text, counted by all three methods at once.
A common compromise: familiar chunks stay whole; rare words, symbols, and emoji fall back to small units.
Manipulate the cutting bill
Every cut sends a bill downstream. Switch to Characters: this lab preserves unfamiliar text but bills one token per character, emoji included.
Try Words. Entries in this tiny vocabulary stay compact; missing words collapse to <UNK>, taking their spelling with them.
Return to Subwords and settle your bet: enter Orangebike 👋 costs 3.14!. Familiar fragments can stay together while smaller fallback units preserve every symbol. Many production tokenizers use variants of this compromise.
The counters show the price. The vocabulary still needs a fallback for the unfamiliar, but fewer tokens for the same text mean fewer positions for later layers to process.
Now type strawberry. Its r’s may hide inside different pieces: the network receives only token IDs and must rebuild the spelling before it can count.
Primary. In Subwords mode, make the chars / token ratio climb. Use the small training corpus as your clue: familiar chunks earn fewer cuts.
Stretch. Add an emoji or invented word and drive the ratio back toward 1. Explain why the bill grew.
Compare your answer
Primary. “eat an orange” keeps familiar chunks together, so each token carries several characters and the ratio rises well above 1.
Stretch. An unfamiliar 👋 or orangebike falls back to small pieces. More cuts cover the same text, so the token bill rises and the ratio falls toward 1.
Explain the subword recipe
Subword tokenizers learn a vocabulary of recurring fragments. In the BPE (Byte Pair Encoding) family, our teaching version repeats one rule:
Find the most frequent adjacent pair of fragments, merge it, then repeat.
That is the cutting rule. The playback shows only merges that change the sample word; each badge keeps that merge’s rank in the full training run.
BPE learns across the whole corpus. This playback shows only merges that change the sample word; it skips the rest.
Frequent chunks become larger tokens; unfamiliar text falls back to smaller units. This lab merges Unicode characters so every cut stays visible.
Production tokenizers often begin with bytes and add pre-splitting rules. The same bill remains: coverage against sequence length.
If C non-space characters produce T visible tokens, the displayed compression ratio is C / T. In Subwords mode a high ratio means the vocabulary already held long chunks of your text.
It measures neither writing quality nor model intelligence, and in Words mode an unknown word collapses to one <UNK> chip, so the ratio can spike on text the vocabulary never saw.
Misconception to avoid. A token is not guaranteed to be a unit of meaning: its boundary depends on the tokenizer and its vocabulary. Change the tokenizer, the language, or the writing system, and the same-looking word splits differently.
Reflect before moving on
Checkpoint
What reaches the first neural layer after the tokenizer cuts orangebike into pieces?
Pick an answer first.
Connect the numbers to what comes next
Before leaving, type eat an orange in Subwords mode. Ignore the separate space chips. This course’s tiny vocabulary keeps eat, an, and orange whole.
For later labs, those teaching tokens receive addresses #120, #18, and #901. All three are course fixtures; no production tokenizer assigned them.
- You typeeat an orange
- BPE keepseat · an · orange
- The course carries#120 · #18 · #901
Those numbers are vocabulary addresses; their size says nothing about meaning. In the next chapter, each address opens a row of learned coordinates and places its token on a map. Can a map make #901 mean orange?
Sources and scope
- Sennrich, Haddow & Birch (2016) describes BPE-style subword units as a way to represent rare and unknown words.
- The lab uses a deliberately readable Unicode-character variant. Production tokenizers may operate on bytes and apply different pre-splitting rules.
- Content and claims reviewed on July 28, 2026.