Lab A — Weight-memory estimation and the gap to reality
Duration: 20 minutes - Format: pairs, one spreadsheet per pair - Sensitive data: none
Fill the table for a 7B-parameter model at three precisions using parameters x bits per weight / 8. Then load the model your machine can actually run, read the resident memory reported by the runtime, and write the gap in the third column. Report both numbers, never the estimate alone.
| Precision | Weight-memory estimate | Measured resident memory |
|---|---|---|
| 16 bits (FP16/BF16) | ||
| 8 bits (INT8) | ||
| 4 bits | ||
| 4 bits, context 8k, 1 sequence | ||
| 4 bits, context 8k, 8 sequences |
Worked solution
For 7 billion parameters the estimates are about 14 GB at 16 bits, about 7 GB at 8 bits, and about 3.5 GB at 4 bits, since the formula is parameters x bits per weight / 8. Measured resident memory is always higher than the estimate: expect roughly 10 to 30 percent above it at short context for weights plus runtime buffers and allocator overhead, and considerably more once the KV cache fills. The last two rows are the point of the lab: holding weights at 4 bits, going from one sequence to eight at 8k context adds KV-cache memory that can approach or exceed the 3.5 GB of weights, depending on layer count, KV heads, head dimension, and cache precision. Pairs who report only the 3.5 GB figure have sized a service that will fail under concurrency.
Lab B — Measuring two precisions under a fixed workload
Duration: 25 minutes - Format: pairs at a terminal, results posted to the shared pad - Sensitive data: none
Compare a higher-precision build with a quantized build of the same model on the same machine. Everything except precision must be held fixed. Post your five numbers to the shared pad so the group can compare spreads across hardware.
- Write down, before loading anything: the baseline build, the fixed prompt set, the fixed output length, the concurrency level, and the quality rollback threshold you will accept.
- Run the baseline build and record time to first token, inter-token latency, output throughput, and peak memory, discarding the first run as warm-up.
- Run the quantized build with the identical prompt set, output length, concurrency, and runtime version, and record the same four numbers.
- Score both builds on the same small set of representative task outputs using the same rubric, and note any answer where the quantized build degraded.
- Record hardware, runtime version, and the artifact hash of each build next to the numbers, then state whether the candidate passes your threshold.
Worked solution
The expected shape of the result is that the quantized build uses clearly less memory for weights, roughly half at 8 bits and roughly a quarter at 4 bits relative to 16 bits, while the latency result is not predictable in advance. On hardware with well-supported kernels for the format, decode throughput often improves because less weight data is read per token. On hardware without those kernels, or where the runtime dequantizes on the fly, the smaller build can be slower than the baseline, and at least one pair in a mixed-hardware room usually observes exactly this. Quality typically degrades slightly and non-uniformly: short factual answers often look identical while longer reasoning or code outputs show the difference first. A pair that cannot state its baseline, its fixed variables, and its threshold has produced an anecdote, not a measurement.
Lab C — Writing the release gate
Duration: 10 minutes - Format: individual, on paper, then two volunteers read theirs aloud - Sensitive data: none
You are asked to ship a 4-bit build of a model currently served at 16 bits, for an internal assistant handling around 20 concurrent users with contexts up to 8k tokens. Write the release gate that must pass before the swap.
- Name the baseline explicitly: which build, which revision, which runtime version it will be compared against.
- List the representative tasks and the quality rubric, and set the rollback threshold as a number before any measurement.
- List the performance measurements required at the real concurrency and context distribution: TTFT, inter-token latency, throughput, peak memory.
- State the environment record: hardware, runtime version, artifact hash for both builds.
- State the rollback procedure and who decides, including what happens if quality passes but peak memory does not.
Worked solution
A passing gate names the 16-bit build at a pinned revision as the baseline, fixes a task set drawn from real traffic rather than a generic benchmark, and sets a quality threshold in advance, for example no more than a stated small drop on the rubric with no regression at all on the safety-relevant subset. It requires TTFT, inter-token latency, throughput, and peak memory measured at 20 concurrent sequences with 8k contexts, not at concurrency 1, because the KV cache at that load is what decides whether the box holds. It records hardware, runtime version, and artifact hash for both builds so the run is reproducible. Finally it names an owner and a rollback path, and treats a peak-memory failure as blocking even when quality passes, since an out-of-memory event under load is a worse outcome than a slightly weaker answer.
Open the interactive — CPU vs GPU Open the interactive — quantization