Hardware fit depends on workload shape. Quantization can reduce model-weight memory, but compatibility, latency, and task quality still require measurement.
What you will be able to do
2-hour session
- Describe, in one sentence each, what a CPU and a GPU are respectively good at for LLM inference, and name the workload property that decides between them.
- Separate a request into its prefill phase and its decode phase, and explain why the first is parallel and the second is sequential across token positions.
- Compute a weight-memory estimate with parameters x bits per weight / 8 for any model size and precision, without a calculator error.
- List at least five memory consumers that this estimate excludes: KV cache, activations, allocator overhead, runtime buffers, metadata, unquantized layers.
- Write a quantization release gate that names the baseline, the representative tasks, and the measurements required before promotion.
- Explain why a smaller model file does not guarantee lower latency, citing at least two concrete mechanisms.
- State that KV-cache precision and weight precision are independent choices, and predict the memory consequence of confusing them.
CPU and GPU roles
CPUs excel at general control flow, preprocessing, and modest workloads. GPUs expose many parallel arithmetic units and high memory bandwidth for dense tensor operations. Prefill is highly parallel; autoregressive decode remains sequential across token positions.
Memory estimate
A first weight-memory estimate is parameters × bits per weight ÷ 8. It excludes KV cache, activations, allocator overhead, runtime buffers, metadata, and sometimes unquantized layers.
Quantization release gate
Compare the quantized candidate with a higher-precision baseline on representative tasks. Record quality, time to first token, inter-token latency, throughput, peak memory, hardware, runtime, and artifact hash.
Open the interactive — CPU vs GPU Open the interactive — quantization
Quick check
Does a smaller model file guarantee lower latency?
Show answer
No. Kernel support, dequantization, workload, and hardware can make a smaller artifact slower.
Does weight quantization automatically shrink KV cache?
Show answer
No. KV-cache precision and weight precision are separate choices.
Practise this
Work the three labs, then check yourself against the 10-question quiz.