KV-cache sizing and optimization

10 questions, one correct answer each. The aim is to check that you can size a KV cache from an architecture sheet, choose a lever for a stated constraint, and read the service metrics that expose cache pressure.

1. What does the KV cache actually store?

A) The generated output tokens
B) The attention key and value tensors for tokens already processed
C) A copy of the model weights for the active layer
D) The tokenizer vocabulary used by the request

Show answer

B — It holds the per-layer key and value tensors computed for tokens already in context, so that decoding a new token does not recompute attention state for the whole prefix.

2. In the common sizing estimate, which quantity multiplies the cache size linearly alongside layers and head dimension?

A) The number of attention query heads
B) The number of KV heads
C) The vocabulary size
D) The number of model parameters

Show answer

B — The cache stores keys and values, which exist per KV head. Query heads do not add cache entries, which is exactly what GQA and MQA exploit.

3. A deployment holds 8k tokens per sequence. Traffic doubles the active context to 16k with the same concurrency. Under the simplified linear estimate, KV memory…

A) Stays the same
B) Roughly doubles
C) Roughly quadruples
D) Falls, because longer prompts batch better

Show answer

B — Cached tokens enter the estimate linearly, so doubling the active context roughly doubles the KV memory for that sequence.

4. Why does the factor 2 appear at the front of the estimate?

A) Because keys and values are stored separately
B) Because the model runs two forward passes
C) Because the cache is mirrored for fault tolerance
D) Because attention is bidirectional

Show answer

A — One tensor for keys and one for values, per layer and per KV head. The factor has nothing to do with precision or redundancy.

5. What does paged attention primarily improve?

A) The numerical accuracy of attention
B) Memory fragmentation and utilization, by allocating the cache in fixed-size blocks
C) The knowledge cutoff of the model
D) The tokenizer compression ratio

Show answer

B — Block-based allocation removes the need to reserve a contiguous worst-case region per sequence, so more sequences fit in the same memory. It does not change attention arithmetic.

6. Which statement about cache quantization is honest?

A) If weight quantization is acceptable, cache quantization is automatically acceptable
B) It must be evaluated separately, because its quality effects differ from those of weight quantization
C) It has no quality effect at all
D) It only affects the prefill phase

Show answer

B — The two touch different tensors and degrade differently, often first in long-context and multi-turn behaviour. A separate evaluation set is required.

7. Which metric most directly exposes prefill cost as context grows?

A) Inter-token latency
B) Time to first token
C) Output tokens per second at steady state
D) Model parameter count

Show answer

B — Time to first token covers prompt processing, so it rises with prompt length. Inter-token latency reflects the decode loop instead.

8. A serving system reports a rising eviction and recompute rate at fixed traffic. What is the most likely reading?

A) The model weights are corrupted
B) Cache demand exceeds the memory budget, so sequences are being dropped and re-prefilled
C) The tokenizer changed
D) The network is saturated

Show answer

B — Eviction followed by recompute is the symptom of cache pressure: freed blocks must be rebuilt by re-running prefill, which shows up as latency and wasted compute.

9. How does a provider prompt cache differ from the run-time KV cache of a single request?

A) They are the same mechanism under two names
B) The provider cache is a prefix-reuse feature with its own eligibility, persistence and billing rules
C) The provider cache stores the generated answer instead of the prompt
D) The run-time cache persists across accounts

Show answer

B — Run-time KV cache is per-request state inside the serving process. Provider prompt caching reuses a matching prefix across requests under conditions the provider defines, and those conditions are not the same as in-process caching.

10. You must halve KV memory without changing model weights or the served model. Which lever is the most direct first move?

A) Increase the batch size
B) Bound the maximum context retained per sequence
C) Add more attention query heads
D) Raise the sampling temperature

Show answer

B — Cached tokens are a linear term you control at the application level. Increasing batch size raises total cache demand, and the other two options do not affect cache memory.

Scoring

8-10 correct: you can size a cache from a configuration sheet and defend a lever choice under a stated constraint. 5-7: revisit the levers section, especially the difference between KV heads and query heads and between cache and weight precision. Below 5: redo Lab A by hand for a single architecture before reading anything else; the arithmetic is the foundation for the rest of the session.

Learner course · Go to the exercises