What scale buys · Verbatome

The toy you trained holds about sixty adjustable numbers - just enough to learn that orange follows “eat an.” Llama 3.1 405B holds 405 billion of them, in the same three places. The difference is one word: scale.

A bigger kitchen arrives with a bill: shelf room for parameters, groceries to feed them, oven time to run the training. Before you read it, place a bet: where did all those parameters go?

No forged receipt from a secret model here. Every number below is published, computed in front of you, or flagged as an estimate or an illustration.

Observe the parameter budget

A parameter budget divides the model’s learned numbers among parts you already met: the embedding table, the attention projections, and the per-token MLP inside every block.

Vocabulary size controls the embedding pile. Width (the length of every token vector) and depth (how many floors stack) grow the attention and MLP piles.

Both grow roughly with the square of the width, so change the shape and the balance tips.

The explorer keeps only these dominant terms and omits biases, normalization, and weight tying. Its totals still preserve the shape of the budget.

Predict where the parameters pile up

Your bet

In the explorer's small-open dense preset, which part holds the most parameters?

Commit before manipulating the instrument. This choice is final.

Scale explorer / where the parameters live

A bigger model is the same recipe with bigger numbers. Set the shape, watch where the parameters pile up, then see what a real training run of that size actually costs.

Load a shapeOur toythe model you built by handSmall open model~8B, weights you can downloadPublished 405B modelLlama 3.1 reference shape

Layers (depth) 12Width (model dim) 768Vocabulary (tokens) 48,000

Live missions

Shrink the vocabulary until the attention projections outweigh the embedding table
Load the published 405B reference and read its training budget

Parameter budget

Embedding table30.3 % 37 M
Attention projections23.2 % 28 M
Per-token MLP46.5 % 57 M

Total parameters122 M

What a run of this size costs

Custom shape. The parameter count is computed; training tokens, GPU-hours, and dollars are only known for the real models above - load one to read them.

Where the training data comes from

The tokens the chapter 8 loop grinds through are not lying around clean. A web-scale corpus is crawled, then most of it is thrown away.

Raw web crawl1000 T
Language + quality filter150 T15 % kept
Near-duplicate removal45 T30 % kept
Tokens actually trained on15 T33 % kept

T = one trillion (a thousand billion) tokens.

Loss versus compute

Pour in more compute - bigger model, more tokens - and the loss falls along a curve. Each tenfold of compute buys a smaller drop, bending toward a floor it never reaches.

FLOP = one floating-point operation; a training run uses billions of billions of them.

irreducible floor (1.85)

The warm marker couples the current parameter count to the 2022 Chinchilla estimate of about 20 training tokens per parameter. It is illustrative. The ~8B and ~400B dots represent published runs.

On this illustrative curve, each tenfold increase buys a smaller loss reduction. The direction follows published scaling laws; the plotted values are not measurements.

Deeper: double the model or double the data?

At fixed compute, parameters and data must stay balanced. Hoffmann et al. (2022) estimated about 20 training tokens per parameter in their setting. This is dated, not a timeless law.

Simplified dense-Transformer formula, not exact Llama GQA/SwiGLU. Presets use public dimensions, tokens, and GPU-hours; dollars, data funnel, and loss curve are labeled estimates. No training runs here.

Manipulate the shape

Load the small open model preset and settle your bet. Read the three shares before touching a slider: in this dense shape (every block runs for every token), which part holds the largest pile? Does it match your pick?

Raise width. Attention and the MLP grow roughly with its square, so their shares pull ahead. Lower vocabulary and only the embedding table shrinks. Two models with the same total can spend it in very different places.

Now load Published 405B model. The cost card prices one documented run: shelves (parameters), groceries (training tokens), and oven time, billed in GPU-hours - one H100 chip running for one hour.

Under the card, one question asks for those GPU-hours within a factor of ten. Answer from the card, not from memory; the mission chip turns green only on a correct read.

Read Where the training data comes from and watch the token count shrink at each stage: crawl, filter, deduplicate. The stages are illustrative - Llama 3.1’s real recipe is private - but the funnel is FineWeb’s, and that one is published.

Finally, read the loss-versus-compute curve. Nobody measured that slope on Llama; it is hand-traced from the published shape, where each tenfold of compute buys a smaller drop and the curve flattens toward a floor.

Challenge Primary mission: reset the explorer, keep depth fixed, then reduce vocabulary until attention outweighs the embedding table. Explain the flip using vocabulary × width versus width².

Optional stretch: load Published 405B model and state the orders of magnitude for parameters, training tokens, and GPU-hours. Flag the dollar figure as an estimate; nobody published the bill.

Compare your answer

The embedding table grows with vocabulary times width. Attention grows roughly with width squared per layer, so shrinking vocabulary can make attention overtake embeddings while depth stays fixed.

Hundreds of billions of parameters, tens of trillions of tokens, tens of millions of H100 GPU-hours. The dollar total is published GPU-hours times an illustrative cloud rate, which is why it stays an estimate.

Explain what scale actually multiplies

Scaling couples three budgets. Parameters buy capacity, data supplies the examples, and compute pays for the arithmetic that turns those examples into better parameters.

A bigger parameter budget without enough data leaves capacity underused. More data without enough compute leaves examples unprocessed. The kitchen grows only when its shelves, groceries, and oven time grow together.

A scaling law is a published empirical relationship between those budgets and model loss. Kaplan et al. measured smooth power-law trends; later work showed that model size and training data must be balanced under a fixed compute budget.

Deeper: a dated balance, not a universal constant

Hoffmann et al. (2022) estimated roughly 20 training tokens per parameter for the models and compute regime they studied.

That ratio belongs to the regime that produced it. What survives is the coupling: parameters without matching data and compute are shelves paid for and left half empty.

For a public anchor, Llama 3.1 reports a dense 405-billion-parameter model trained on about 15 trillion tokens. Its paper puts that run at roughly 31 million H100 GPU-hours.

At the explorer’s illustrative rate, those GPU-hours give roughly $60–$120 million. The range is an estimate built from that rate, and all of it is spent before the model answers a single question.

Reflect on the coupled budget

Checkpoint

Which three budgets must grow together in a large training run?

Parameters, training data, and compute.The number of distinct algorithms the model runs at once.The tokenizer rules, giving each language its own private model.

Pick an answer first.

You also just priced one real training run. Carry the orders of magnitude forward; the exact digits belong to that one paper.

Checkpoint

For the published 405B model shown in the explorer, which set of numbers has the right order of magnitude?

Billions of tokens, thousands of GPU-hours, and a few thousand dollars.Trillions of tokens, millions of GPU-hours, and roughly $60–$120 million at the stated illustrative rate.Trillions of parameters, with a fresh full training run for every question you ask.