1/18
KV-cache sizing and optimization
The cache that makes decoding fast is the same cache that decides how many users you can serve.
Presenter notes
Open with the trade in one sentence: speed bought with memory. Promise that by the end everyone can size a cache from a configuration sheet.
2/18
Why the second token is cheap
The key and value tensors of every processed token are kept, so decoding does not recompute attention state for the whole prefix.
Presenter notes
Draw the prefix growing token by token and shade the reused state. Keep this drawing up for the rest of the session.
3/18
Three units of memory
- Per token: keys and values for every layer and KV head
- Per sequence: per-token cost times the tokens held in context
- Per deployment: per-sequence cost times the concurrent sequences
Presenter notes
Insist on the third line. Most sizing mistakes come from reasoning per sequence and forgetting concurrency.
4/18
The sizing estimate
2 x layers x KV heads x head dimension x cached tokens x bytes per element, multiplied by active sequences when caches are independent.
Presenter notes
Write it once, then immediately say what it excludes: weights, activations, fragmentation, runtime overhead. It is an order of magnitude.
5/18
Where the factor 2 comes from
One tensor for keys, one for values — per layer, per KV head.
Presenter notes
This is a common misreading. Ask the room before answering; someone will propose precision or redundancy.
6/18
KV heads, not query heads
The cache scales with KV heads, which is why grouped-query and multi-query attention shrink it without changing parameter count much.
Presenter notes
Point at both numbers on the printed configuration cards. Require the KV-head figure to be circled before Lab A begins.
7/18
Lab A — size it by hand
80 layers, 8 KV heads, head dimension 128, 16-bit elements. Compute per token, per sequence, per deployment.
Presenter notes
25 minutes in pairs. Circulate and check units early; byte-to-gibibyte slips are the main source of wrong answers.
8/18
Order of magnitude, stated assumptions
| Context (tokens) | Concurrent sequences | Approx. KV memory |
|---|---|---|
| 4096 | 1 | ~1.3 GB |
| 8192 | 1 | ~2.7 GB |
| 8192 | 8 | ~21 GB |
| 32768 | 8 | ~86 GB |
| 131072 | 1 | ~43 GB |
Presenter notes
Assumes 80 layers, 8 KV heads, head dimension 128, 2 bytes per element. Say the assumption aloud every time you point at this table.
9/18
Doubling behaves the way you fear
Under the linear estimate, doubling the active context or doubling concurrency roughly doubles KV memory.
Presenter notes
This is the single sentence to leave on the board during the break. It is the practical takeaway of the arithmetic.
10/18
Where the estimate stops being linear
- Allocator fragmentation and block granularity
- Sliding-window or hybrid attention layers
- Runtime-specific layouts and reserved headroom
Presenter notes
Be explicit that the formula is a planning tool, not an accounting statement. Always verify against the runtime you actually deploy.
11/18
The levers
- Bound context and batch deliberately
- GQA or MQA when the model allows it
- Paged allocation of cache blocks
- Verified prefix reuse and cache precision
Presenter notes
Preview all four, then take them one at a time. Learners want a ranking; give them the ranking only after the costs.
12/18
Bounded context first
Cached tokens are the term you control at the application level, and it enters the estimate linearly.
Presenter notes
Connect back to session 04: the excerpt that answers the question is usually far shorter than the document attached.
13/18
Paged attention
Allocating the cache in fixed-size blocks removes the contiguous worst-case reservation per sequence, so more sequences fit.
Presenter notes
Read the runtime documentation page here rather than paraphrasing. The block vocabulary matters when reading dashboards later.
14/18
Cache precision is its own decision
Lower-precision cache is not the same trade as lower-precision weights, and it needs its own evaluation.
Presenter notes
Name the failure mode: degradation appears first in long-context and multi-turn behaviour, which short benchmarks miss.
15/18
Prefix reuse has preconditions
- The prefix must be byte-identical, not merely similar
- Reuse must stay inside one trust boundary
- The saving is prefill, not decode
Presenter notes
Force a byte-level comparison of two real prompts in Lab B. Templates that differ by one system line share nothing.
16/18
Measuring the service
| Metric | What it exposes | Read it at |
|---|---|---|
| Time to first token | Prefill cost and queueing | p50 and p95 |
| Inter-token latency | Decode loop health | p50 and p95 |
| Cache occupancy | Headroom before eviction | Over the whole run |
| Eviction / recompute rate | Wasted prefill under pressure | Over the whole run |
Presenter notes
Ask which metric would move first if the context distribution shifted longer. The answer is TTFT, and it is the one people watch least.
17/18
Benchmark what you will actually serve
A single fixed prompt length at a single concurrency predicts nothing about production behaviour.
Presenter notes
Require a context-length distribution and at least two concurrency levels before accepting any Lab B measurement plan.
18/18
Provider prompt cache is not your KV cache
- Run-time KV cache is per-request state in the serving process
- Provider prefix reuse has its own eligibility and persistence rules
- Billing and cache lifetime are defined by the provider, not by you
Presenter notes
Close by reading the provider page aloud and listing only the rules it actually states. Do not let the room fill the gaps from memory.