verbatome
Verbatome / Chapter 05II · Inside the Transformer9 min read · ~15 min hands-on

In this lesson

Word order enters the machine

Same tokens, different order, different meaning. Run an exact permutation experiment and give every slot its own geometric stamp.

By the end, you can

  • Demonstrate why content-only attention cannot distinguish a token moved to another position.
  • Explain how a position signal makes order available to every later computation.

Before you start: Attention, where tokens start talking

“Dog bites man” and “man bites dog” contain the same pieces. Only their places changed, yet the event reversed. Attention compares token content through queries and keys. A dot product sees numbers, not first, second, or last. This lesson gives order one job of its own before the full Transformer floor adds anything else.

Observe the missing coordinate

The lab sends orange through the same unmasked block in two phrases: eat an orange and orange an eat. With position vectors off, every operation is either applied to one token or compares token content with other token content. Nothing reads a slot number.

Predict what reversing can change

Your bet

With position vectors off, what happens to orange's output when it moves from last to first?

Nobody grades this. Commit, and let the instrument settle it.
Position lab / give order a coordinate

Reverse the same three tokens. Without a position signal, the tracked token follows its content unchanged. Add position vectors and its path splits.

Live missions

  • Track orange across two position stamps
  • Make the unmoved middle token change through attention

Original order

eatanorange

slot
p2
position stamp
none
same block output
[-1.225, 1.360]

Reversed order

orangeaneat

slot
p0
position stamp
none
same block output
[-1.225, 1.360]

Same vector. Content moved, but the computation received no clue that its place changed.

Real block computation with handcrafted 2D embeddings and additive position vectors.

Manipulate the position stamp

First track each of the three tokens with positions off. Its output follows the token unchanged when the phrase reverses. The block is permutation-equivariant: reorder the inputs and the matching outputs reorder with them.

Now turn positions on. Slot p0 adds [0.4, 0]; slot p2 adds [0, 0.4]. These are tiny coordinates attached to places, not words. When orange moves, its content row stays [-0.68, 0.44], but the sum entering attention changes. The rest of the block can finally react to order.

Challenge Two missions. One: with positions on, predict which stamp ([0.4, 0] or [0, 0.4]) orange receives in each phrase, then check. Two: explain why adding a position vector is enough for attention to react to order even though attention never reads a written slot number.

Compare your answer

Orange receives the stamp of the place it occupies: [0, 0.4] when last and [0.4, 0] when first. Adding that stamp changes the numbers from which Q, K, and V are computed, so all later comparisons can react to place without needing a separate “slot” instruction.

Explain why positions work

A Transformer does not receive word order as invisible metadata. Order must enter as numbers. This toy adds a vector to every token. Many production models instead rotate queries and keys according to position. The mechanisms differ, but the contract is the same:

token representation + position signal → position-aware representation

  1. Same token roworange · [−0.68, 0.44]
  2. Different slot stamp+ p0 or + p2
  3. Different input to Q, K, Vattention can react to order

The position signal does not say what a sentence means. It only lets later learned weights discover useful patterns such as “the subject came earlier” or “this closing bracket matches that opener.”

Reflect on the two experiments

Checkpoint

Why can the content-only block not notice that orange moved?

Why can the content-only block not notice that orange moved?

Connect order to a complete floor

Tokens now carry both content and place. Order is no longer something the course asked you to take on faith. You proved the requirement yourself: no stamp, no order. The next lesson keeps the stamps on and walks the rest of the floor - normalization, the MLP, and the quiet trick that lets a hundred floors stack.

Sources and scope
  • Vaswani et al. (2017) introduces the Transformer’s added position encodings.
  • Su et al. (2021) describes rotary position embedding (RoPE), which encodes position by rotating queries and keys.
  • The lab uses handcrafted 2D additive stamps and removes the causal mask solely to make the permutation experiment exact.
  • Content and claims reviewed on July 20, 2026.