In this lesson
Confident, wrong, and augmented
Make a fluent model answer a fictional fact incorrectly, supply retrieved evidence, and see both the value and limits of grounding.
By the end, you can
- Explain hallucination as plausible next-token prediction without a built-in truth test.
- Demonstrate how retrieval and tools add evidence through context while remaining fallible.
Before you start: The prompt is the program
Ask the toy below where the Aster Museum is, and it answers Paris - instantly, fluently, at temperature zero, and wrongly. Nothing malfunctioned. The model was optimized to continue text, and in eight chapters of machinery you never built a step that opens the world and checks a claim. Fluent-and-false has a familiar name: hallucination. This chapter makes the toy commit one, then repairs the answer from outside.
Observe a confident mistake
The Aster Museum is fictional, and this lab decides what is true about it: the museum’s registry says Lyon. The model has never seen that registry, while its base logits confidently prefer Paris. No sampling is needed; greedy decoding chooses the wrong city. An external system can retrieve the registry and place it before the question, changing the evidence available in context.
Predict what retrieval guarantees
Your bet
A retrieval system adds a document before the question. What does that guarantee?
Nobody grades this. Commit, and let the instrument settle it.
The toy must answer a question about a fictional fact. Change what the external system retrieves, then inspect how that text shifts the same model bet.
Live missions
- Make relevant evidence correct the bet
- Make distracting evidence strengthen the mistake
- Correct the second question with relevant evidence
- Expose a citation the model cannot support
External system
- retrieve text
- insert into context
- call the model
- check or cite the answer
No document was retrieved.
user: Where is the fictional Aster Museum?
Answer distribution
Fluent and wrong. Greedy decoding did not prevent the error.
Retrieval changes context, not weights. Good evidence can reduce error; missing, misleading, ignored, or misread evidence can still produce a hallucination.
Handcrafted evidence effects, real logits and softmax. This teaching setup demonstrates the mechanism, not a factual claim about a real place or institution. The citation check is a simple rule keyed to the evidence source, not a real quote-verification algorithm.
Manipulate the evidence path
With no source, Paris wins. Add the registry and Lyon wins because the relevant text adds a strong contextual clue. Then choose the distracting travel guide: Paris becomes even more confident. Retrieval is a pipe, not a truth machine. Its output may be absent, stale, irrelevant, adversarial, or simply misunderstood by the model.
Now switch the fictional fact to the Corvid Archive and run the repair a second time: only the relevant source pulls the answer to 1958, and correcting this second question is its own mission. Then flip on Require a citation. A pass/fail line appears - the answer passes only when it quotes the supplied source, and fails when there is no source or the text never backs the claim. Force a failing line and you have watched a citation check catch an unsupported answer. Here it is only a simple source-keyed rule, not real quote verification, but the shape is the point.
The wrapper can improve the system further: search multiple sources, rerank passages, require citations, verify a calculation with a tool, or abstain when support is weak. Each step is external orchestration around the same next-token engine.
Challenge Two missions. One: retrieve something that makes the model more confidently wrong, then explain why “more evidence” hurt. Two: with the relevant registry selected, name the two failure points that remain between retrieval and a trustworthy answer.
Compare your answer
The distracting travel guide repeats Paris, so it reinforces the model’s wrong preference: retrieval quality matters more than the mere presence of text. Even with the registry, the model can misread or ignore good evidence, and the evidence itself can be stale, hostile, or wrong. Verification and abstention remain orchestration jobs.
Explain hallucination without mysticism
The model produces logits for plausible continuations. It has no separate internal operation named isTrue(answer). Post-training may reward admitting uncertainty, and good context may contain the answer, but neither turns probability into proof.
Sampling can trade reliability for variety, yet it is not the root cause. Greedy decoding can confidently select the highest-scoring false continuation. Conversely, a sampled answer can be true. The important boundary is between language likelihood and external verification.
Retrieval-augmented generation, often called RAG, follows a visible loop:
question → retrieve evidence → add evidence to context → call model → inspect or cite output
- External systemfind and rank evidence
- Runtime contextplace evidence before the question
- Model + wrappergenerate, inspect, cite, or abstain
Tools follow a similar pattern. The model proposes a structured call; an orchestrator executes it; the result returns as new context.
Reflect on the guarantee
Checkpoint
Why can a grounded assistant still hallucinate?
Checkpoint
You retrieve the museum registry and place it before the question. What has retrieval actually guaranteed?
Connect three assistant levers
You can now locate three different ways a chatbot changes. Weights store behavior learned during pretraining and post-training. Runtime context carries instructions, history, examples, and evidence for one request. Orchestration retrieves documents, runs tools, chooses decoding, stops generation, and calls the model again. The capstone will ask you to trace both the inner model and this outer loop without mixing them together.
Sources and scope
- Lewis et al. (2020) combines retrieval with generation by supplying retrieved passages to a language model.
- The Aster Museum and all logits are authored teaching data. They demonstrate how evidence can alter a real softmax distribution, not a claim about an actual museum.
- Retrieval lowers some factual risks but cannot guarantee source quality, faithful use, or a verified final answer.
- Content and claims reviewed on July 20, 2026.