In this lesson
Embeddings: meaning becomes geometry
Each identifier opens a locker holding a learned vector. Drag words around a toy map and measure what resembles what.
By the end, you can
- Explain how an embedding table turns an identifier into a vector.
- Interpret cosine similarity without mistaking it for a measure of truth.
Before you start: Tokenization, or how a model reads
The previous lesson ended with three addresses: #120, #18, #901. An address can distinguish orange from cat, but its size carries no meaning: token #901 is not “more orange” than token #314. To compute with useful relationships, the model needs more than that locker number. It needs a vector. A vector is nothing scarier than a short list of numbers - two in this lab - drawn as an arrow from the center of a map. First, play with the geometry it will use.
Observe a vocabulary become a map
The map below places cat and dog pointing in nearly the same direction, while apple sits off on its own. This is not yet a map learned by a model; it is a hand-built scene for isolating the geometry.
Predict the cat’s big move
The instrument’s ranking scores how much every word resembles the observed token. Now suppose you grab cat and drag it all the way over to apple, on the far side of the map.
Your bet
If cat moves in with apple, what happens to its similarity with dog?
Nobody grades this. Commit, and let the instrument settle it.
This small map is handcrafted. Grab a point and drag it (sliders and arrow keys work too), and watch the cosine ranking follow.
Live missions
- Make car closer to queen than bicycle
- Make a cosine cross zero
| Token | ID | x | y | cosine |
|---|---|---|---|---|
| #314 | 0.82 | 0.72 | 1.00 | |
| #271 | 0.74 | 0.68 | 1.00 | |
| #408 | 0.34 | 0.86 | 0.89 | |
| #409 | 0.30 | 0.96 | 0.85 | |
| #901 | -0.68 | 0.44 | -0.27 | |
| #612 | -0.78 | 0.32 | -0.44 | |
| #733 | 0.20 | -0.82 | -0.46 | |
| #734 | 0.12 | -0.66 | -0.51 |
Teaching toy: these eight points did not come from a trained model. A real embedding is learned and often has hundreds of dimensions.
Manipulate directions
Grab a point and drag it; the map and table update together. Watch the ranking rather than the screen distance: cosine cares about direction from the origin, not absolute position.
Challenge Make car more similar to queen than to bicycle, in as few moves as you can. Then break something: find a single-point move that flips a cosine from positive to negative. The reset button forgives everything.
Compare your answer
Car and bicycle both start pointing toward the bottom of the map while queen points up, so one move settles the first mission: drag car upward until its arrow lines up with queen’s direction. Cosine judges direction from the origin, not screen distance, so car now resembles queen more than bicycle. That very move also breaks a sign: with car swung up and bicycle still pointing down, the angle between them passes 90 degrees, so their dot product, and thus cos(car, bicycle), crosses zero and turns negative.
These eight points are handcrafted, not projected from a trained model. Real embedding tables are learned and often have hundreds or thousands of dimensions. Their axes do not arrive with human labels such as “animal” or “royal”; useful regularities are distributed across many directions.
Explain the machinery of the table
In 2013, researchers showed that some learned word-vector offsets captured recurring relationships. The famous shorthand was king − man + woman ≈ queen. It is a striking example, not an algebraic law: the result depends on the model and its training data.
- Token identifierorange · #901
- Table lookupE[901]
- Starting vector[−0.68, 0.44]
An embedding table is a matrix E - a grid of numbers - with one row per token. For identifier i, E[i] returns vector eᵢ. During training, the values in E are adjusted whenever doing so helps the model predict the next token.
The course’s workhorse for comparing and scoring vectors is the dot product: multiply matching slots, then add. For [a, b] and [c, d], that is ac + bd, and it grows large when two arrows point in the same direction. Chapter 4 uses this operation to score which tokens match; chapter 7 uses it to score every word in the vocabulary.
To compare two vectors, the instrument uses cosine similarity: a score that cares only about direction, never about length. A value near 1 means “similar directions” in this learned space; it does not mean “identical words” or “true statement.” At [0, 0], direction does not exist, so cosine similarity is undefined - the lab says so instead of inventing a zero.
The same score, in formulas
cos(a, b) = (a · b) / (||a|| × ||b||)
The dot product a · b grows when directions align; dividing by both lengths removes scale, so only the angle survives.
Math pause: vectors without the fog
A vector here is only an ordered row of numbers. [0.8, 0.7] means move 0.8 along the horizontal axis and 0.7 along the vertical one. A dot product multiplies matching coordinates and adds them: [a, b] · [c, d] = ac + bd. The real model uses thousands of coordinates, but the operation is still multiply matching slots, then add.
Context is still missing. For the same orange token identifier, the table returns the same row in “eat an orange” and in a new phrase we lean on from here: “paint it orange.” (Chapter 1 peeled its oranges; painting one forces the second meaning.) The initial embedding carries learned regularities, including imperfections and biases from training; it does not yet express the precise role of this occurrence. Keep those two phrases in mind: they open chapter 4.
Reflect on what a cosine says
Checkpoint
Rows E[271] and E[314] have different lengths but point in nearly the same direction. What can you conclude?
Connect the fixed vector to its sentence
We now have one starting vector per token, but each lookup blissfully ignores the surrounding sentence: the vector for orange has no idea whether it is fruit or paint. In the next lesson, that same initial vector enters two contexts and leaves with two different outputs. Time to let the tokens talk to each other.
Sources and scope
- Mikolov, Yih & Zweig (2013) reports recurring syntactic and semantic offsets in learned word-vector spaces.
- The lab’s eight 2D points are handcrafted to teach lookup and cosine geometry; they are not measurements from a trained language model.
- Content and claims reviewed on July 18, 2026.