The Transformer block, the floor a model stacks

Here is the secret behind the intimidating diagrams: a Transformer is mostly one floor, photocopied. Attention, which you built in chapter 4, is one of its rooms. The quiet trick chapter 5 promised is on this floor, twice.

Observe the running stream

Start with the residual stream: one vector per position, already carrying its token and the slot stamp chapter 5 added.

RMSNorm rescales that stream, attention mixes information across positions, and the mixture meets the stream it came from at a small + sign.

A second RMSNorm, then the MLP (multi-layer perceptron), a small learned network working each position alone, then another +. Twice per floor, one question: what happens to everything the stream already carries?

Predict what a residual does

Positions begin on. Chapter 5 won them. Your bet is stage 8, + residual = block output, the floor’s final +.

Your bet

When the MLP contributes a correction, what happens to the incoming residual stream?

Commit before manipulating the instrument. This choice is final.

Block explorer / one complete floor

Step one word through all eight stages of one floor. At each residual addition, check what the addition kept.

Position vectorsChapter 5 recall: add p₀, p₁, p₂ before the blockTracked word

Live missions

Track the middle word and claim the first stage where its two traces diverge

1 / 8 · Input embedding

Original order

eatanorange

position: p2
vector added: [0.000, 0.400]
Input embedding: [-0.680, 0.440]

Reversed order

orangeaneat

position: p0
vector added: [0.400, 0.000]
Input embedding: [-0.680, 0.440]

Same vectors at this stage: the position stamps have not changed anything the block has read yet. Step forward to find where they start to matter.

The full floor (original order)

orange · #901
Stage	Vector
Input embedding	[-0.680, 0.440]
+ position vector	[-0.680, 0.840]
RMSNorm (before attention)	[-0.890, 1.099]
Attention mixture	[-0.143, 0.680]
+ residual (after attention)	[-0.823, 1.520]
RMSNorm (before MLP)	[-0.673, 1.244]
MLP (2 → 3 → 2, ReLU)	[0.000, 1.244]
+ residual = block output	[-0.823, 2.764]

Inspect attention weights (original order)

Attention weights (unmasked, in this lab)
Q ↓ / K →	eat	an	orange
eat	0.339	0.480	0.181
an	0.261	0.401	0.338
orange	0.053	0.055	0.892

Each displayed row sums to 1.000. A real decoder keeps the causal mask from chapter 4; it is removed here so the order experiment is exact.

Manipulate the floor

Keep Position vectors on. Step from the positioned stream through RMSNorm, then attention. At + residual, read that row against + position vector and Attention mixture before you decide what the addition kept.

Continue through the second RMSNorm and the MLP. This lab expands 2 features to 3, applies ReLU, then contracts to 2. The first MLP output coordinate reads 0.000 because ReLU clipped the hidden activations that feed it.

That is a skyscraper’s whole ceremony, performed on three hidden numbers.

At stage 8, + residual = block output, your bet settles: incoming stream plus MLP correction, nothing discarded.

Now set Tracked word to an and step through the stages. The panel runs both sentence orders side by side, so you can find the first stage where the two orders diverge.

An keeps the same slot in both orders, so the split begins only when one operation reads its reordered neighbors.

When you think you have that stage, press This is the first divergence. The instrument tells you whether your claim lands too early, too late, or right on it.

Open Inspect attention weights if you want to see the shares behind that mixture.

Toggle positions only as a recall check from chapter 5. The quiet trick it promised is this floor’s new idea, the residual stream, a continuous workspace that every sublayer reads and amends.

Challenge With positions on, track the middle word an. Reversing the phrase leaves it in the same slot with the same position vector. Claim the first stage where its two traces diverge, then explain why.

Compare your answer

The traces first diverge at attention. Before it, an has the same token vector, slot, position vector, and normalized vector in both runs.

Attention reads across positions. Reversing the phrase changes the neighboring keys and values that an mixes, so its attention correction changes.

Optional stretch.

Inspect the attention matrix and explain why every row sums to 1. Chapter 4’s accountant, the softmax, is still on duty: it balances each row of unmasked scores to that exact total.

Explain the floor plan

Here is the floor in order, and why each room earns its place:

Position vectors break the symmetry in chapter 5’s unmasked, content-only experiment by adding slot information to each token vector.
RMSNorm rescales each vector to a standard size before it enters a sublayer, so values do not drift huge or tiny as floors stack.
Attention is the only step where positions exchange information; you built it in chapter 4.
The MLP complements attention: it never looks sideways. It applies the same learned transformation to each position independently and often holds a large share of a dense block’s parameters, its adjustable learned numbers.
Residual additions preserve the incoming vector and add a correction: output = input + correction, twice per floor. This running total is the residual stream.

This floor is missing a wall, chapter 4’s causal mask, which separates allowed history from forbidden future. Order here comes from the slot stamps and nothing else.

Why photocopy the floor? On the next copy, attention reads vectors the previous floor already contextualized, so each round of cross-position mixing and per-position transformation builds on earlier corrections.

Depth enables that composition. The residual stream gives every block a continuous path to the accumulated state.

The two normalizing and transforming steps have exact formulas, but you can cross the floor without them; open the panel below when you want the mechanics.

Chapter 4 builtattention: tokens talk
This floor addspositions · norm · MLP · residuals
A real model stacksN × this floor

The same floor, in formulas

In the lab’s simplified form, RMSNorm rescales x to x / √(mean(x²)). Its root-mean-square becomes 1 while its direction is preserved. Implementations also use a learned gain and a small stabilizing term.

The lab MLP computes W₂ · ReLU(W₁x + b₁) + b₂, with ReLU(z) = max(0, z). ReLU clips negative activations to zero.

Here the MLP expands each position from 2 to 3 dimensions, then contracts to 2. Dense Transformer MLPs typically expand the hidden width by a fewfold before projecting back.

One pre-norm floor, written as amendments to the residual stream h:

h ← h + Attention(RMSNorm(h)) h ← h + MLP(RMSNorm(h))

This lab adds hand-authored position vectors. Models may instead learn position encodings or rotate Q and K by position with RoPE. These methods inject order through different operations.

No single operation is “where the model thinks.”

Attention moves information between positions; the MLP transforms each position independently. They alternate block after block.

A decoder normally leaves the causal-mask wall standing. This lab removes it so the unmasked order experiment from chapter 5 remains exact.

Reflect on what the stream keeps

Checkpoint

Why do both sublayers use residual additions?

They force every correction to have a positive sign.They preserve a running signal while attention and the MLP add learned corrections.They convert vectors back into vocabulary IDs after every floor.

Pick an answer first.

Checkpoint

Why does stacking many identical floors buy more than a single floor?

Each new floor introduces brand-new tokens into the sentence.The residual stream is reset between floors, so each floor starts from nothing.Each floor reads vectors the previous floors already contextualized, so the amendments compose.