KV Cache Optimization: Cut LLM Latency and VRAM Use

KV Cache Optimization: Cut LLM Latency and VRAM Use

Photo by Teemu on Unsplash

Quick Answer: The KV cache — not model weights — is what blows up your VRAM at long context. A Llama-class 8B needs 16 GB of cache at 128k context in FP16, on top of the weights. The highest-impact fixes, in order: quantize the cache to FP8 (halves it, near-zero quality loss), enable prefix caching for repeated system prompts (10-50x faster time-to-first-token on cache hits), and serve with PagedAttention (vLLM/SGLang) to eliminate the 60-80% memory waste of static allocation. Q4 cache quantization saves another half but visibly hurts long-context retrieval.

On This Page

• What the KV Cache Is and Why It Eats VRAM
• The Size Formula, Worked Out
• The Optimization Toolbox
• KV Cache Quantization: FP8 vs Q4
• Framework Support Matrix
• Practical Configs: llama.cpp and vLLM
• Latency vs Quality Tradeoffs
• Frequently Asked Questions

What the KV Cache Is and Why It Eats VRAM

Every transformer layer computes key and value vectors for each token in the sequence. Without caching, generating token N would require recomputing attention over all N-1 previous tokens from scratch — quadratic waste. So inference engines store those K and V tensors once and reuse them: that's the KV cache.

The catch: the cache grows linearly with context length and batch size, and it lives in VRAM next to the weights. Weights are a fixed cost; the cache is a variable cost that dominates as contexts stretch to 32k, 128k, and beyond. When people say "the model fits but the context doesn't," this is what they mean.

There's a second cost people miss: time-to-first-token (TTFT). Prefill has to build the cache for the whole prompt before the first output token appears. A 50k-token prompt on a single consumer GPU can mean 10–30 seconds of prefill. Cache reuse techniques attack this directly.

The Size Formula, Worked Out

The formula for cache size per sequence:

KV bytes = 2 (K and V) × n_layers × n_kv_heads × head_dim × context_len × bytes_per_element

Note n_kv_heads, not attention heads — grouped-query attention (GQA) is why modern models are viable at all. Llama-class 8B: 32 layers, 8 KV heads, head_dim 128 → 128 KB per token at FP16. A 70B: 80 layers, 8 KV heads, 128 dim → 320 KB per token.

Read that 8B/128k row again: the cache (16 GB) is twice the size of the Q4_K_M weights (4.9 GB). At long context, cache optimization matters more than weight quantization — a point we expand on in our GGUF quantization guide. And these numbers are per sequence: serving 16 concurrent users at 32k each multiplies the FP16 figure by 16.

The Optimization Toolbox

Two of these deserve emphasis:

PagedAttention (vLLM's signature idea) allocates the cache in fixed-size blocks — like OS virtual memory pages — instead of reserving max-context-length per request up front. Before it, engines wasted the majority of cache memory on padding; after it, memory utilization routinely exceeds 90%, which converts directly into larger batch sizes and 2–4x throughput.

Prefix caching stores the cache for a shared prompt prefix (system prompt, few-shot examples, RAG boilerplate) and reuses it across requests. SGLang's RadixAttention generalizes this into a radix tree over all live prefixes, which is why it dominates agentic workloads where hundreds of calls share 90% of their prompt.

"In agent-heavy deployments we see prefix cache hit rates above 80%. That single feature cuts our P50 time-to-first-token from 4.2 seconds to 180 milliseconds." — infrastructure engineering talk, AI Infra Summit, Q2 2026

Photo by Google DeepMind on Unsplash

KV Cache Quantization: FP8 vs Q4

Not all cache bits are equal. Keys are more sensitive than values — quantization error in K distorts which tokens get attended to, while error in V only distorts what gets retrieved.

• FP8 (E4M3 or E5M2): the 2026 default for serving. Halves cache size, hardware-accelerated on Ada/Hopper/Blackwell GPUs, and long-context benchmark deltas are within noise (<0.5% on RULER-style evals).
• Q4 cache: in llama.cpp, q4_0 for both K and V quarters the FP16 footprint. Short-context chat is fine, but needle-in-a-haystack and multi-hop retrieval past 32k degrade measurably. The common compromise: K at q8_0, V at q4_0 — keys keep precision where it matters.
• INT8/INT4 with per-channel scales (vLLM/TensorRT-LLM): similar story, with calibration reducing the Q4 penalty.

Framework Support Matrix

Rule of thumb: vLLM or SGLang for serving many users; llama.cpp for one user on consumer hardware. SGLang wins agentic/multi-turn workloads on RadixAttention; TensorRT-LLM wins raw single-node throughput on NVIDIA data-center silicon when you can afford the engine-build workflow.

Practical Configs: llama.cpp and vLLM

llama.cpp — 32B model on a 24 GB GPU, 32k context that wouldn't otherwise fit:

llama-server -m qwen-32b-q4_k_m.gguf \
  -c 32768 \
  --cache-type-k q8_0 \
  --cache-type-v q4_0 \
  -fa on \
  -ngl 99

Flash attention (-fa on) is required for quantized V cache and reduces compute-buffer memory on top. This config drops the 32k cache from ~5 GB to ~1.9 GB for a Qwen-class 32B — the difference between fitting and spilling.

vLLM — 8B serving with FP8 cache and prefix reuse:

vllm serve meta-llama/Llama-4-Scout-Instruct \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.92

--gpu-memory-utilization matters more than people think: vLLM gives every byte not used by weights to the paged cache, and cache capacity is what determines concurrent batch size and throughput.

Latency vs Quality Tradeoffs

Ordering the techniques by risk-adjusted return:

1. Free wins (do always): PagedAttention, chunked prefill, prefix caching, flash attention. Zero quality cost.
2. Cheap wins (do by default): FP8 or q8_0 cache. Quality deltas are below eval noise on 2026 models.
3. Situational: Q4 V-cache — fine for chat and RAG under 16k, avoid for 64k+ document analysis or agent pipelines that depend on precise long-range recall.
4. Last resorts: aggressive sliding windows or cache eviction (H2O-style token dropping) — these change model behavior, not just performance, so eval before shipping.

The meta-lesson: profile where your memory actually goes before quantizing weights harder. At 4k context the cache is a rounding error; at 128k it's the whole story.

Related Reads

• SGLang RadixAttention: Why Prefix Caching Matters for RAG

Key Takeaways

• At 128k context, an 8B Llama-class model's KV cache (16 GB FP16) dwarfs its Q4_K_M weights (4.9 GB)—optimize cache before weights for long-context workloads.
• FP8 KV cache quantization halves VRAM use with near-zero quality loss; use fp8_e4m3 in vLLM or q8_0 in llama.cpp for keys (values tolerate q4_0 with minimal degradation).
• PagedAttention (vLLM/SGLang) eliminates 60–80% of static cache waste, enabling 2–4x larger batches—mandatory for multi-user serving at scale.
• Prefix caching (vLLM’s --enable-prefix-caching, SGLang’s RadixAttention) slashes TTFT by 10–50x for repeated prompts (e.g., system prompts, RAG templates).
• For llama.cpp on consumer GPUs: combine -fa on (flash attention) with --cache-type-k q8_0 --cache-type-v q4_0 to fit 32k context in 24 GB VRAM for 32B models.
• Profile memory first: cache dominates at 32k+ context, but at 4k it’s negligible—prioritize weight quantization for short-context use cases.