In this lesson
From raw continuer to assistant
Walk one architecture through supervised fine-tuning and preference tuning. See how training steers its bets, what response-only loss does, and what remains unguaranteed.
By the end, you can
- Explain how supervised fine-tuning and preference tuning steer a base model’s weights.
- Explain why response-only SFT concentrates loss on replies without hiding the prompt from the model.
- State why these later training phases change behavior without installing a truth checker or rule engine.
Before you start: What scale buys
Fresh from pretraining - chapter 8’s next-token diet at chapter 9’s scale - a model can meet a question with another question. It is not a broken assistant but a successful text continuer, betting on what usually follows the visible text.
To make an assistant, trainers send that same tower to finishing school - no obedience box on the syllabus, just more training, until answer-like continuations win the bet more often.
Observe one tower, three schools
The lab compares one architecture at three authored snapshots: base, supervised fine-tuning (SFT), and preference tuning. Attention, blocks, and softmax stay in place; only the trained weights differ.
Every continuation and token share is hand-authored to make those stages legible. What is real is the claim they demonstrate: changed weights can move bets. Start where the tower starts, before any school has touched it.
Predict the base fixture’s reaction
Give the base stage one instruction: Translate “cat” into French.
Your bet
In this authored fixture, what does the base stage do with the translation instruction?
Commit before manipulating the instrument. This choice is final.
One architecture, three sets of numbers. Guess first, then compare.
Continuations, tells, and token shares are authored for this fixture. They illustrate training stages, not measured behavior from a real model.
Read the continuation, then name the school that produced it - before the reveal.
Translate “cat” into French.
“chat.”
Live missions
- Attribute 6 of 8 correctly
- Score a flawless 8 of 8
Compare side by side opens on the translation prompt. Which column never answers, dealing worksheet instead of a translation? Read it first, then settle your bet.
Manipulate the three stages
Open Guess the school. The clues are deliberately bold: more worksheet at base, a direct reply after SFT, then fuller framing or a refusal after preference tuning.
Learn this fixture’s codebook before treating it as evidence.
Switch to Compare side by side and choose the insult prompt. Track the refusal-starting token: an authored 5% long shot, barely 8% after SFT, then a 64% favorite - and not one layer of the architecture changed.
That climb is the mechanism to keep: further training can shift probability mass onto a behavior.
Open Inspect the SFT mask and watch which positions carry loss: only the reply is scored. The prompt earns no loss term, yet every reply gradient flows back through the attention that read it.
Primary: trace the refusal token across all three stages. Name what changes, what stays fixed, and what this authored movement can actually demonstrate.
Stretch: inspect the SFT mask. Explain why the prompt still shapes the reply even though the optimizer never scores it.
Compare your answer
The authors set the refusal share higher to illustrate a possible outcome when preference data rewards declining and redirecting. The fixture does not compute that optimization.
In response-only SFT the prompt is pure context that the optimizer never scores. Gradients born on reply positions pass back through everything the prompt conditioned - unscored does not mean unused.
Without the mask, training would also spend loss on the user and system text, and the model would learn to write prompts as well as answer them.
Explain the finishing school
- SFT applies chapter 8’s cross-entropy to curated dialogue demonstrations. In this response-only recipe, the prompt is input while the reply is the scored target.
- Preference tuning learns from comparisons between candidate replies. RLHF (reinforcement learning from human feedback) and DPO (direct preference optimization) move probability toward the selected behavior rather than one reference reply.
- Pretraining buildsa continuer of text
- SFT reinforcesuseful reply patterns
- Preferences reinforceselected kinds of reply
These stages can teach behavior, formats, and some task content. Their common mechanism is ordinary training: gradients change weights, and changed weights reshape later bets.
Finishing school installs neither a rule engine nor a truth test. A refusal, a citation-shaped sentence, and a confident date are all generated continuations.
Better steering can reduce failures without certifying any token. The school grades manners, and a wrong date can have impeccable ones.
Published evaluations find that human and preference-model judgments can favor replies that echo a user’s view over a correct disagreement. The papers measure a tendency; whether a given preference run inherits it is an empirical question.
Reflect on what actually changed
Checkpoint
What does response-only masking do during the SFT example?
Pick an answer first.
Checkpoint
A preference-tuned assistant states a wrong date with perfect aplomb. What follows?
Pick an answer first.
Connect the trained assistant to runtime context
Finishing school changed weights that are now frozen. Nothing you type tonight will move them.
Yet the next lesson steers this assistant anyway, without retraining a thing. That leaves one question to carry into it: where does a request live, if not in the weights?
Sources and scope
- Ouyang et al. (2022) documents an SFT-then-RLHF pipeline and reports both instruction-following gains and remaining mistakes.
- Rafailov et al. (2023) presents Direct Preference Optimization for learning from chosen and rejected responses.
- Sharma et al. (2023) measures sycophancy and studies how human and preference-model judgments can reward agreement over correctness.
- All continuations and token shares are authored teaching fixtures. They illustrate probability steering inside one toy, not measured stage behavior from a real model.
- Content and claims reviewed on July 28, 2026.