Tokenization, or how a model reads

How many r’s in strawberry? For years, models that could pass a bar exam kept fumbling this grade-school question. The reason is not mysterious: the neural network inside a language model never sees letters. A tokenizer first breaks text into tokens and maps each token to a numerical identifier. The network receives those identifiers, nothing else. If strawberry arrives as a single block, there is nowhere to count r’s.

There are several ways to cut it up: some preserve every tiny detail, some keep the sequence short, and some fail on an unfamiliar word. Your first job is to find the useful compromise.

Observe three ways to break a sentence

Take “the cat sleeps.” Anything goes when turning it into units:

• letter by letter: t, h, e, c, a, t… (over a dozen units for three words);
• word by word: the, cat, sleeps (three units, done);
• or something in between: pieces of words.

The third option looks like the strange one. Whether it is also the best one is exactly what you are about to bet on.

Predict the winning split

Three candidates: characters, words, subwords. The shortest is easy to guess. The real question lies elsewhere: what will each mode do with the invented word moonbike, a 👋, or 3.14?

Your bet

Which mode handles moonbike without losing its spelling or stretching the whole sentence?

CharactersWordsSubwords

Nobody grades this. Commit, and let the instrument settle it.

Tokenizer

Your textTokenization turns language into numbers.41 / 240 characters

Split by

CharactersWordsSubwords (BPE)

Tokenization↵␣turns↵␣language↵␣into␣numbers↵.

25tokens

37characters

1.5chars / token

A common compromise: familiar chunks stay whole; rare words, symbols, and emoji fall back to small units.

ResetCopy link

Manipulate the bill

The strange third option turns out to be precisely the compromise most production tokenizers make - and the counters show what it buys them. Switch to Characters: almost nothing is unknown, but the sequence becomes long. In Words, entries known to this tiny vocabulary stay compact while missing words collapse to <UNK>: the original spelling is lost. Finally, return to Subwords and try Moonbike 👋 costs 3.14!. Every symbol should remain visible, while familiar fragments can stay together.

The counters expose the trade-off: fewer tokens usually mean less work in later layers, while the vocabulary still needs a fallback for inputs it has never seen.

Explain the subword recipe

Subword tokenizers learn a vocabulary of recurring fragments. In the BPE (Byte Pair Encoding) family, a readable version of the algorithm repeats one rule:

Find the most frequent adjacent pair of fragments, merge it, then repeat.

That is the whole recipe. The playback below shows only the merges that change the sample word; the numbered badge keeps each merge’s rank in the full training run:

Tokenizer · BPE merge playback- English

BPE learns across the whole corpus. This playback shows only merges that change the sample word; it skips the rest.

One word from the corpus, across the merges

tokenization

Start: every character is its own token.

PlayPreviousNext stepReset

Visible merge 0 of 9Merges learned in the corpus: 80Vocabulary size: 26

Frequent chunks become larger tokens; unfamiliar text falls back to smaller units. (This teaching variant works on Unicode characters and keeps punctuation, whitespace, and emoji visible; production tokenizers usually work on bytes with extra pre-splitting rules - the central trade-off is the same.)

If C non-space characters produce T visible tokens, the displayed compression ratio is C / T. It describes this split only; it measures neither writing quality nor model intelligence.

Reflect before moving on

Checkpoint

Why are subwords a useful compromise for an unfamiliar word such as moonbike?

Why are subwords a useful compromise for an unfamiliar word such as moonbike?

AThey guarantee that every word always becomes exactly one token.BThey use one token per character for every sentence.CThey keep frequent chunks compact and fall back to smaller units for the rest.

Check answer

Connect the numbers to what comes next

Before leaving, type eat an orange in Subwords mode. Ignore the separate space chips: this course’s tiny learned vocabulary keeps eat, an, and orange whole. We now assign those teaching tokens the addresses #120, #18, and #901 for the labs ahead. These are course fixtures, not identifiers from a production tokenizer.

1. You typeeat an orange
2. BPE keepseat · an · orange
3. The course carries#120 · #18 · #901

Be careful with those numbers: they are only addresses in a vocabulary, and their magnitude says nothing about semantic proximity. In the next lesson, each address opens a locker holding a learned vector.

Sources and scope

• Sennrich, Haddow & Birch (2016) describes BPE-style subword units as a way to represent rare and unknown words.
• The lab uses a deliberately readable Unicode-character variant. Production tokenizers may operate on bytes and apply different pre-splitting rules.
• Content and claims reviewed on 2026-07-18.