Lab A — The knowledge-cutoff experiment
Duration: 20 minutes · Format: pairs · Sensitive data: none
Ask one factual question whose correct answer changed recently, first on a surface with no retrieval, then on a retrieval-enabled surface. Record both answers verbatim, then audit the citations. The goal is not to catch the model out; it is to see the failure mode change shape.
| # | What to record | Your observation |
|---|---|---|
| 1 | Answer from the no-retrieval surface (verbatim) | |
| 2 | Does it hedge, or state a stale fact confidently? | |
| 3 | Answer from the retrieval surface (verbatim) | |
| 4 | How many sources are cited? | |
| 5 | Open source 1: does it contain the specific claim? | |
| 6 | Open source 2: does it contain the specific claim? | |
| 7 | Claims supported / unsupported / unverifiable | |
| 8 | Publication date of the newest cited source |
Worked solution
The expected pattern: the no-retrieval surface answers fluently from stale parameters, often without hedging — fluency is not calibration. The retrieval surface is usually more current but rarely fully verifiable: a typical result is that one or two claims are directly supported by the cited pages while at least one is a plausible-sounding synthesis appearing in no single source. That unsupported claim is the entire lesson. Retrieval moved the risk from "the model is out of date" to "the summary is unfaithful to its sources", and only opening the pages exposes it. If every claim checks out, note the publication dates: currency is what retrieval genuinely bought you.
Lab B — Delegated computation: read the program, not the number
Duration: 20 minutes · Format: pairs · Sensitive data: none
Give a code-executing surface a small table of numbers and ask two questions: one purely arithmetic, one requiring an extrapolation. Then read the emitted program before looking at the result.
- Paste the 20-40 row dataset and ask for a straightforward aggregate (a mean, a total, or a per-category count).
- Before reading the answer, read the emitted program. Write down which rows it selected and how it handled missing or malformed values.
- Recompute one value by hand or with a spreadsheet and compare.
- Now ask the surface to project the trend three periods forward.
- Read that program too, and write down the single assumption that the printed number hides.
Worked solution
On the aggregate, the arithmetic is normally exact — that is the genuine advantage of leaving the token stream. The instructive failures are upstream of the arithmetic: silently dropped null rows, a filter that excluded a category, or a type coercion that turned a malformed value into zero. Any of these produces a confident, well-formatted, wrong number. On the extrapolation, the program is usually a linear fit; the hidden assumption is that the trend continues unchanged beyond the observed range, which the data cannot support. State the assumption next to the figure whenever you report it. The transferable rule: the program is the artifact under review, and the number is only its output.
Lab C — Context budget and the persistence audit
Duration: 15 minutes · Format: individual · Sensitive data: none
Estimate what you are actually paying for on a long-context turn, then audit what your surface has silently remembered about you.
- Open the tokenizer interactive linked from the learner course page and paste a document you would realistically attach.
- Record the approximate token count, and note that this cost is paid on every turn the document stays in the window.
- Identify the smallest excerpt that would still answer your question, and estimate its token count.
- Open the memory or custom-instructions panel of a surface you use regularly.
- List what it has stored, and mark each entry: still useful, stale, or should never have been stored.
Worked solution
The token estimate is deliberately approximate — real tokenizers differ, which is exactly why the interactive states its heuristic openly. The reliable finding is the ratio: the excerpt that actually answers the question is usually a small fraction of the full document, and that difference is billed on every subsequent turn. The persistence audit almost always surfaces at least one stale entry, and often one that was never intentionally saved. Neither result is a vendor indictment; both are the cost of convenience features that default to on. The habit worth keeping: bound what you paste, and review what persists.
Open the interactive — context budget Open the interactive — tokenizer