Rafael Lopes2026-06-30
The Core Claim
The KV cache is the single structural trick that makes autoregressive decoding affordable: it stores the Key and Value projections of every processed token so each new token costs attention against a lookup, not a recomputation of the whole sequence — "You don't modify it during the LLM inference. You just append to it, with every processed token" [Source 40]. The cost it shifts is from compute to memory bandwidth, and managing that memory well is where the wins compound: vLLM's PagedAttention reported up to 24× throughput over Hugging Face Transformers and TGI [Source 141], while FP8 quantization of the cache "significantly reduce[s] its memory footprint" to "store more tokens in memory" [Source 25].
Evidence (5–7 numbered insights)
1. The cache exists to delete redundant work, not to add a feature. Without it, generating token N requires recomputing K and V projections for all N tokens every step — quadratic waste. The cache makes each decode step append-only: compute K/V for the current token, append, attend.
"Let's say we don't store K and V projection for current token. It would mean that we need to compute all K and V projections for the current and all tokens before we can compute attention for current token. Again, pure waste." — [Source 86]
2. PagedAttention turns the cache from a contiguous block into OS-style pages, unlocking the throughput. Pre-allocating HBM for max sequence length strands memory; paging allocates per actual decode length and shares pages across requests, which is what produces the headline multiplier.
"Instead of allocating GPU high-bandwidth memory (HBM) for the maximum output token lengths of the models, the paged attention of vLLM allocates GPU HBM dynamically for its actual decoding lengths." — Source 2
3. Decode is memory-bandwidth-bound, which is exactly why cache layout dominates latency. A GPU has ~2 orders of magnitude more compute than a CPU but only ~1 order more bandwidth — so the cache-read-heavy decode phase is throttled by memory, not FLOPs.
4. KV cache capacity is the concurrency budget — it is a number you can read off the engine. vLLM prints the cache size in tokens and derives how many simultaneous requests fit; if that number is below your traffic, you add GPUs.
"INFO 07-23 13:56:04 [kv_cache_utils.py:775] GPU KV cache size: 643,232 tokens" — Source 6
5. Disaggregating prefill from decode, with the KV cache as the hand-off, yields large tail-latency and TTFT gains. Prefill (compute-bound) runs on high-memory GPUs; decode (memory-bound) scales separately; both share the same cache for similar requests.
"implementing LLM-D improved P90 latency... at a improvement of uh, three times... there was also an increase by 57 times in the first token response time." — Source 9
6. Prefix caching reuses the KV of shared prompt spans across requests, deleting repeated prefill. vLLM exposes this as a first-class switch (--enable-prefix-caching) keyed by a prompt hash, so a common system prompt is computed once.
"Warmup so that the shared prompt's KV cache is computed." — [Source 158]
7. Quantizing the cache to FP8 buys more tokens per byte and longer context for free-ish. kv_cache_dtype="fp8_e4m3" is supported on both CUDA 11.8+ and ROCm, and with FlashAttention-3 the attention math itself runs in FP8.
"Quantizing the KV (Key-Value) cache to FP8 format can significantly reduce its memory footprint. This optimization enables you to store more tokens in memory, leading to improved throughput and support for longer context windows." — [Source 25]
How It Works
Prefill fills the paged cache once; each decode step then computes only the new token's query, attends it against the entire cached K/V, emits one token, and appends that token's K/V back — a tight append-and-read loop bounded by HBM bandwidth, not by re-running the prompt [Source 142], [Source 25].
What This Means in Practice
The KV cache is the inference-side analog of memoization, and the same instinct a staff engineer brings to a high-traffic e-commerce stack applies: cache the expensive shared prefix, then pay only for the delta. Treat a stable system prompt or product-catalog preamble the way you'd treat a CDN edge cache or a React Server Component boundary — compute once, reuse across requests — by enabling prefix caching so repeated prefill disappears [Source 158], [Source 52]. Size capacity from the printed GPU KV cache size and Maximum concurrency lines rather than guessing, the same way you'd budget INP against a measured p75 rather than a vibe Source 6. When TTFT is the metric users feel — the inference equivalent of LCP — separate the compute-bound prefill from the memory-bound decode and let each scale on its own, mirroring how useTransition and Next.js streaming decouple the heavy first paint from interactive updates Source 9, [Source 138]. And when VRAM is the wall, FP8 cache quantization is the cheapest lever before you reach for more GPUs [Source 25].
Counter-Evidence / Limits
The cache is not free throughput — it relocates the bottleneck to memory bandwidth, so a workload that is already compute-bound (large-batch prefill) sees little decode-side benefit, which is precisely why chunked prefill exists to co-schedule the two [Source 138]. Capacity is finite and contested: tensor parallelism duplicates the cache tp_size / H times, and decode context parallel (-dcp) only claws that back at the cost of added communication overhead [Source 150]. Quantizing the cache to FP8 is not unconditionally safe — the highest-quality scales require dataset calibration via llm-compressor, and the default scales of 1.0 are a correctness compromise [Source 25]. Disaggregated designs that move the cache between prefill and decode nodes depend on high-bandwidth RDMA interconnect to avoid the transfer becoming the new bottleneck [Source 154]. The sources are unanimous that the cache is essential and equally clear that every optimization on top of it is a memory-vs-communication-vs-accuracy trade, not a pure win.
Today's CEMENT brick
Map — Spin up vllm serve on any model you have access to and read the two startup log lines: GPU KV cache size: N tokens and Maximum concurrency for M tokens per request: Xx Source 6. In 30 minutes, divide your real max context (max_model_len) into N to derive your true concurrency ceiling, then toggle --enable-prefix-caching and re-read the numbers to see how shared-prefix reuse changes the budget [Source 52].
Sources
- vLLM inference PagedAttention overview
- Parallelism and Scaling — GPU KV cache size logging
- What is vLLM? Efficient AI Inference for LLMs
- LLM‑D Explained: Building Next‑Gen AI with LLMs, RAG & Kubernetes
- Quantized KV Cache FP8
- Optimization and Tuning — Chunked Prefill
- Context Parallel Deployment — Decode Context Parallel
- Building Windsurf with Varun Mohan memory-bound GPUs
- tiny-vllm — Prefill, decode and KV cache
- tiny-vllm/README.md — Why KV cache exists
- Flexkv Connector — prefix-cache warmup example
- SGLang distributed inference with Mooncake RDMA KV transfer
- vllm serve — CacheConfig prefix caching flags