Learning objectives
- Describe, in one sentence each, what a CPU and a GPU are respectively good at for LLM inference, and name the workload property that decides between them.
- Separate a request into its prefill phase and its decode phase, and explain why the first is parallel and the second is sequential across token positions.
- Compute a weight-memory estimate with parameters x bits per weight / 8 for any model size and precision, without a calculator error.
- List at least five memory consumers that this estimate excludes: KV cache, activations, allocator overhead, runtime buffers, metadata, unquantized layers.
- Write a quantization release gate that names the baseline, the representative tasks, and the measurements required before promotion.
- Explain why a smaller model file does not guarantee lower latency, citing at least two concrete mechanisms.
- State that KV-cache precision and weight precision are independent choices, and predict the memory consequence of confusing them.
Materials
- One machine per pair with a local inference runtime already installed and at least one small model already downloaded (do not download during the session).
- The CPU/GPU interactive and the quantization interactive linked from the learner course page.
- A printed or projected reference card of common precisions: FP32, FP16/BF16, INT8, 4-bit, with bytes per weight.
- A whiteboard for the prefill/decode timeline and the memory-budget stack diagram.
- A shared spreadsheet or pad for pairs to post their measured numbers so the group can compare spreads.
- Offline fallback: a pre-recorded terminal capture and a CSV of measurements from the same model at 16, 8, and 4 bits, so both labs still run without network or GPU access.
Minute-by-minute plan
| Time | Duration | Segment |
|---|---|---|
| 0:00 | 10 min | Framing: hardware fit is a property of the workload, not a ranking of chips. |
| 0:10 | 20 min | Concept 1: CPU and GPU roles. Prefill vs decode on the whiteboard. |
| 0:30 | 20 min | Lab A: weight-memory estimation at 16, 8, and 4 bits, then compare with resident memory. |
| 0:50 | 10 min | Concept 2: what the estimate excludes. Build the memory-budget stack. |
| 1:00 | 5 min | Break. |
| 1:05 | 25 min | Lab B: measure TTFT, inter-token latency, and peak memory at two precisions. |
| 1:30 | 15 min | Concept 3: the quantization release gate. Debrief Lab B against it. |
| 1:45 | 10 min | Lab C: write the gate for a stated deployment, on paper. |
| 1:55 | 5 min | Review: the two counter-intuitive truths, and what to measure tomorrow. |
Key messages to land
- Hardware fit depends on workload shape. CPUs excel at general control flow, preprocessing, and modest workloads; GPUs expose many parallel arithmetic units and high memory bandwidth for dense tensor operations.
- Prefill is highly parallel and compute-bound; autoregressive decode stays sequential across token positions and is usually bound by memory bandwidth. The two phases have different bottlenecks and must be measured separately.
- A first weight-memory estimate is parameters x bits per weight / 8. Treat it as a floor, not a budget.
- That estimate excludes the KV cache, activations, allocator overhead, runtime buffers, metadata, and sometimes layers that were left unquantized. Real resident memory is always higher.
- A smaller model file does not guarantee lower latency. Kernel support, dequantization cost, workload shape, and hardware can make a smaller artifact slower.
- Weight quantization does not automatically shrink the KV cache. KV-cache precision and weight precision are separate choices, and one is not evidence for the other.
Common pitfalls
Learners conclude that the GPU is simply the better device and stop thinking about fit.
Present a counter-example on the spot: a low-volume, single-user, short-prompt service where a CPU meets the latency target and the GPU adds cost, scheduling complexity, and idle capacity. Ask which one you would deploy and why.
The memory estimate is treated as the deployment budget, so the service is sized to fail under load.
Draw the budget as a stack on the whiteboard: weights at the bottom, then KV cache, activations, runtime buffers, allocator overhead. Refuse any lab answer that reports only the weight figure.
A pair reports that the 4-bit build is faster without having controlled prompt, batch, runtime, or hardware.
Stop the debrief and ask for the fixed variables. If they cannot list them, the measurement is not evidence. Re-run one case with the prompt and concurrency pinned.
Quality regression from quantization goes unnoticed because nobody defined a baseline before testing.
Require the higher-precision baseline and the rollback threshold to be written down before any quantized build is loaded. Enforce this at the start of Lab B, not at the debrief.
Learners assume quantizing weights also quantized the cache, and are then surprised by memory growth at long context.
Make it explicit with numbers: hold weights at 4 bits and show that the KV cache at long context and high concurrency can rival or exceed the weight footprint.
The session drifts into a comparison of specific vendor cards and quantization formats.
State up front that formats and kernels change on a monthly cadence, so any specific comparison is dated. Teach the measurement protocol; leave the current leaderboard out.