Embeddings: meaning becomes geometry

Token #901 is not “more orange” than token #120. An address says nothing about what it holds, and the tokenizer handed over nothing but addresses. Yet the model needs useful numbers before it can bet.

An embedding table supplies them by lookup. An ID opens one row; that row is a vector, an ordered list of coordinates. In this 2D lab, the vector becomes an arrow on a map. Start by redrawing the map yourself.

Observe a vocabulary become a map

The eight points here are hand-built to isolate the geometry; no trained model produced them. The map places cat and dog in nearly the same direction, while apple sits apart.

Predict how the map redraws the ranking

The ranking compares every arrow with the observed token. Before touching the map, bet on what happens when cat moves all the way over to apple, across the origin from dog.

Your bet

If cat moves in with apple, what happens to its similarity with dog?

It stays high, since cat and dog are still animals, whatever their exact positions.It collapses, and may even drop below zero once cat sits across from dog.It cannot change, because vectors are drawn once and stay fixed forever after.

Commit before manipulating the instrument. This choice is final.

Vector map / 2D toy

This small map is handcrafted. Grab a point and drag it (sliders and arrow keys work too), and watch the cosine ranking follow.

Observed tokencat · #314dog · #271king · #408queen · #409apple · #612orange · #901car · #733bicycle · #734Dimension 1

0.82

Dimension 2

0.72

Live missions

• ○Make car closer to queen than bicycle
• ○Make a cosine cross zero

Closest neighbour: dog (cosine 1.00).

Values and similarities

Token	ID	x	y	cosine
cat	#314	0.82	0.72	1.00
dog	#271	0.74	0.68	1.00
king	#408	0.34	0.86	0.89
queen	#409	0.30	0.96	0.85
orange	#901	-0.68	0.44	-0.27
apple	#612	-0.78	0.32	-0.44
car	#733	0.20	-0.82	-0.46
bicycle	#734	0.12	-0.66	-0.51

Teaching toy: these eight points did not come from a trained model. A real embedding is learned and often has hundreds of dimensions.

ResetCopy link

Manipulate directions

Start with your bet: select cat, drag it onto apple, and read the dog row in the cosine column.

Drag one point. The map and the table update together, and the ranking re-prices every bet. Trust the ranking over screen distance: cosine compares each arrow’s direction from the origin.

A real embedding table is learned and often runs to hundreds or thousands of dimensions against this lab’s two. Its map carries no north labelled “animal” or “royal”; each useful regularity is spread across many directions at once.

Explain the machinery of the table

First comes lookup. An embedding table is a matrix E, one row per token. Identifier i opens row E[i] and returns starting vector eᵢ.

1. Token identifierorange · #901
2. Table lookupE[901]
3. Starting vector[−0.68, 0.44]

During training, the coordinates in E change when that helps the model predict the next token. The resulting row places its token on the learned map before sentence context is mixed in.

Next comes comparison. The course’s workhorse is the dot product: multiply matching coordinates, then add. For [a, b] and [c, d], the result is ac + bd.

The dot product tends to grow when arrows align. Chapter 4 uses it to score token matches; chapter 7, to score vocabulary rows for the next bet.

The instrument uses cosine similarity, which removes length and compares direction. Near 1 means aligned directions in this space, not identical words or a true statement.

At [0, 0], no direction exists, so cosine is undefined. The map says so instead of quietly inventing a zero.

Lookup and comparison in hand, you can now scope the famous example. A 2013 study found recurring offsets in some learned word vectors, summarized as king − man + woman ≈ queen.

That pattern is evidence from particular models and data, not an algebraic law. A learned map can carry useful regularities without giving every direction a clean human meaning.

The same score, in formulas

cos(a, b) = (a · b) / (||a|| × ||b||)

The dot product a · b grows when directions align; dividing by both lengths removes scale, so only the angle survives.

Math pause: vectors without the fog

A vector here is only an ordered row of numbers. [0.8, 0.7] means move 0.8 along the horizontal axis and 0.7 along the vertical one.

A dot product multiplies matching coordinates and adds them: [a, b] · [c, d] = ac + bd. A real model may use thousands of coordinates, but the operation stays the same.

Reflect on what a cosine says

Checkpoint

The tokenizer emits ID #901 for orange. What does the embedding layer do first?

The tokenizer emits ID #901 for orange. What does the embedding layer do first?

AIt treats 901 as a coordinate, so larger IDs begin farther across the semantic map.BIt searches every row for the word closest to orange, then returns that row.CIt uses 901 as an address and returns row E[901], the token’s starting vector.

Check answer

Pick an answer first.

Connect the fixed vector to its sentence

Each lookup now places a token on the starting map, blissfully unaware of its sentence. The orange row cannot yet tell the fruit bowl from the paint can.

Next, that same starting vector enters two contexts and leaves with two different outputs. What operation lets the other tokens redraw its position?

Sources and scope

• Mikolov, Yih & Zweig (2013) reports recurring syntactic and semantic offsets in learned word-vector spaces.
• The lab’s eight 2D points are handcrafted to teach lookup and cosine geometry; they are not measurements from a trained language model.
• Content and claims reviewed on July 28, 2026.