Verbatome
Chapter 05II · Inside the Transformer5 min read · ~8 min hands-on

II · Inside the Transformer

In this lesson

Word order enters the machine

Same tokens, different order, different meaning. Run an exact permutation experiment and give every slot its own geometric stamp.

By the end, you can

  • Demonstrate why content-only attention cannot distinguish a token moved to another position.
  • Explain how a position signal makes order available to every later computation.

Before you start: Attention, where tokens start talking

“Dog bites man” and “man bites dog” contain the same pieces. Reverse their order and the whole event flips.

Chapter 4’s head, with its mask off, never saw which one came first. The lab strips attention to content alone: no position signal and no causal mask. Will orange notice when it moves, or will the computation simply follow its content?

Place that bet before giving each slot a geometric stamp. In this lab, everything attention will ever know about order has to fit inside that stamp.

Observe the missing coordinate

The lab sends orange through the same unmasked attention block - one head plus the small learned network wrapped around it - in two phrases: eat an orange and orange an eat.

With position vectors off, every operation acts on token content, and nothing marks which slot a token occupies. The block gets the pieces of the sentence; whether it gets the order is your bet.

Predict what reversing can change

Your bet

With position vectors off, what happens to orange's output when it moves from last to first?

Commit before manipulating the instrument. This choice is final.
Position lab / give order a coordinate

Reverse the same three tokens with the position signal off, then add the position vectors and watch what changes.

Second bet. Once positions are on, which token will still produce identical outputs in both orders?

Live missions

  • Track orange across two position stamps
  • Call the second bet, then make the unmoved middle token change

Original order

eatanorange

slot
p2
position stamp
none
same block output
[-1.225, 1.360]

Reversed order

orangeaneat

slot
p0
position stamp
none
same block output
[-1.225, 1.360]

Real block computation with handcrafted 2D embeddings and additive position vectors.

Manipulate the position stamp

First track each token with positions off. When the phrase reverses, does a single digit of orange’s output move? Compare the two cards before you call it.

orange an eat is nonsense to you and three respectable tokens to the block. The block is permutation-equivariant: reorder the inputs and the matching outputs reorder with them.

Before you switch the stamps on, the lab demands a second bet. Which token will still produce identical outputs in both orders? Commit; the verdict lands at once.

Now turn positions on. Slot p0 adds [0.4, 0]; slot p2 adds [0, 0.4]. These coordinates belong to places, not words.

When orange moves, its content row stays [−0.68, 0.44], but the sum entering attention changes. The block can now react to its place.

Challenge Two missions. First, with positions on, predict which stamp ([0.4, 0] or [0, 0.4]) orange receives in each phrase, then check.

Then set Tracked token to an and watch the token that never moved change anyway: its neighbours’ stamps reach it through attention. Explain why the stamp lets attention react to order even though it never reads a written slot number.

Compare your answer

Orange receives the stamp of its slot: [0, 0.4] when last and [0.4, 0] when first.

For the unmoved middle token, attention mixes neighbours whose position-stamped vectors changed. The stamp changes the numbers used to compute Q, K, and V, so every later comparison can react to place without reading a separate slot label.

Explain why positions work

A Transformer does not receive word order as invisible metadata. Order must affect the numbers it computes.

This toy adds a vector to every token. Many production models instead rotate queries and keys by position. The mechanisms differ, but the contract is the same:

token representation + position signal → position-aware representation

  1. Same token roworange · [−0.68, 0.44]
  2. Different slot stamp+ p0 or + p2
  3. Different input to Q, K, Vattention can react to order

The signal does not say what a sentence means; it lets later layers learn patterns such as “the subject came earlier” or “this bracket closes that one.”

Why remove the causal mask? A causal mask is already asymmetric: the first position can see one token, the next can see two, and so on.

A model can exploit that pattern to smuggle in some position information without an explicit signal. This lab removes the mask so only the slot stamp can break the symmetry.

Reflect on the two experiments

Checkpoint

Why does the content-only block fail to notice that orange moved?

Why does the content-only block fail to notice that orange moved?

Pick an answer first.

Connect order to a complete floor

Tokens now carry both content and place. In this unmasked experiment, exact to the digit, you proved the result yourself: no stamp, no order.

The next lesson keeps the stamps on and walks the rest of the floor - normalization, the MLP, and the quiet trick that lets a hundred floors stack.

Sources and scope
  • Vaswani et al. (2017) introduces the Transformer’s added position encodings.
  • Su et al. (2021) describes rotary position embedding (RoPE), which encodes position by rotating queries and keys.
  • Haviv et al. (2022) shows that causal language models can infer position without an explicit position encoding, likely by exploiting the causal mask.
  • The lab uses handcrafted 2D additive stamps and removes the causal mask solely to make the permutation experiment exact.
  • Content and claims reviewed on July 28, 2026.