Inside the KV Cache: The Life of a Gigabyte

In this post, I’ll follow the memory on a GPU running vLLM [1], gigabyte by gigabyte, from the moment vllm serve starts to the KV cache pool that ends up serving your requests. In particular, I’ll show how vLLM works out the size of that pool, and why the flags everyone tunes by feel (gpu_memory_utilization, max_model_len and max_num_seqs) all act on that one number.

It starts with what the KV cache is and then layers in detail, one term at a time, so you can build an accurate mental model of where the memory goes before we do any serious arithmetic.

This post is structured into five parts:

1. Background: what the KV cache is, how vLLM hands it out in blocks, and why the pool’s size is the server’s capacity
2. Startup: how vLLM sizes the pool, step by step, and what each flag changes along the way
3. KV bytes per token: the number that turns gigabytes into tokens
4. Scaling up: from one RTX 4090 to two H100s, and a calculator for your own setup
5. Failure modes: five common OOMs, each traced back to the step that causes it

Background: the KV cache

We’ll use the following command as our running example, on a single RTX 4090:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --max-num-batched-tokens 8192

This configuration is:

• Llama 3.1 8B [2] with 16-bit weights (2 bytes per parameter)
• one RTX 4090, which CUDA reports as 23.5 GiB (tensor parallel size 1)
• gpu_memory_utilization at 0.90 (the default is 0.92 in v0.30; 0.90 leaves a little more headroom)
• max_num_batched_tokens at 8,192
• no max_model_len, so vLLM uses the model’s own maximum (we’ll see why that matters)
• a standard dense transformer with grouped-query attention

From here, we’ll gradually build up to a 70B model on two H100s. The hardware and the numbers get bigger, but the method never changes.

Before this command can serve a single request, vLLM has to decide how much of the card goes to the KV cache. So let’s start with what the KV cache is.

What the KV cache is

A transformer [3] generates text one token at a time. To produce each new token, every attention layer compares that token against all the tokens before it, and for that it needs each earlier token’s key and value vectors, at every layer.

Those vectors don’t change once they’re computed, so recomputing them on every step would redo the whole prompt for every token generated. Instead, the engine keeps them in GPU memory. That store is the KV cache, and it works in two phases:

• Prefill: a forward pass over the whole prompt writes K and V for every prompt token.
• Decode: each step generates one token and appends that token’s K and V.

So the cache grows by a fixed number of bytes per token, and the memory a server needs at any moment is the total number of tokens in flight across all of its requests: every prompt plus everything generated so far. How many bytes one token costs is part 3; for now, it’s enough that the cost is per token.

Blocks and the pool

The obvious way to store this would be one contiguous buffer per request, sized for the longest sequence it might reach. That wastes most of the buffer on requests that finish early. PagedAttention [4] borrows the fix from operating systems: split the memory into fixed-size blocks (16 tokens each by default) and give each request blocks only as its tokens arrive.

How does this work in vLLM?

1. At startup, vLLM allocates one large pool of blocks and puts all of them on a free_block_queue (in vllm/v1/core/block_pool.py). The size of that pool is the subject of this whole post.
2. Each engine step, the scheduler calls the KV cache manager’s allocate_slots for every request it wants to run. A request with 17 new tokens needs ceil(17 / 16) = 2 blocks, taken from the free queue.
3. When a request finishes, its blocks go back on the queue.
4. If a running request needs a new block and none is free, the scheduler preempts another running request: it frees that request’s blocks, moves it back to the waiting queue, and later recomputes its tokens. (With prefix caching, on by default, freed blocks keep their contents until they’re reused, so some of that work can be recovered. Under memory pressure, usually little of it is.)

Here is a toy example with a pool of just 6 blocks:

# block_pool.pyself.free_block_queue = FreeKVCacheBlockQueue(self.blocks)# sched/scheduler.pypreempted_req = self.running.pop()

Two things follow from this, and the rest of the post leans on both:

• Nothing is reserved per request. A request only holds blocks for tokens it actually has. Setting a large max_model_len doesn’t set any memory aside.
• The pool is the server’s capacity. Its size in tokens caps how many tokens can be in flight across all requests at once. When it runs out, requests wait or get preempted, and preempted work is paid for twice.

So the question that matters is: how big is the pool? vLLM doesn’t let you set it directly. It works it out at startup. Next, let’s watch it do that.

Startup: how vLLM sizes the pool

The pool gets whatever memory is left once everything else is on the card. vLLM can’t know most of those other terms in advance, so it measures them one at a time while it starts up, and sizes the pool last.

How does this work in vLLM? The engine core runs these steps on each GPU worker, in this order (see _initialize_kv_caches in vllm/v1/engine/core.py):

1. Load the model (load_model): build the architecture and load the weights. The CUDA context, about 0.6 GiB, is already there from when the worker initialised its GPU. That’s also when vLLM checks that the card has at least gpu_memory_utilization × total memory free, and refuses to start if it doesn’t.
2. Profile a forward pass (determine_available_memory → profile_run): run a dummy batch of max_num_batched_tokens tokens through the model and record the peak memory. That peak is mostly activations. They’re freed afterwards, but the space is kept back, because real batches will need it.
3. Estimate the CUDA graphs (profile_cudagraph_memory): unless you pass --enforce-eager, measure what capturing the CUDA graphs will cost, and hold that back too.
4. Size the pool: the budget (total memory × gpu_memory_utilization) minus everything measured so far. vLLM logs this as “Available KV cache memory”.
5. Check max_model_len (get_kv_cache_configs): make sure the pool can hold at least one request of max_model_len tokens, and stop with an error if it can’t. Otherwise, divide the pool into blocks and log its size in tokens.
6. Allocate and capture (initialize_from_config, then compile_or_warm_up_model): allocate the blocks, capture the CUDA graphs into the space held for them, and start serving.

Here is the running example going through those steps:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192INFO Model loading took 14.96 GiB memory and … secondsINFO Estimated CUDA graph memory: 0.60 GiB totalINFO Available KV cache memory: 2.83 GiBERROR ValueError: To serve at least one request with the model's max seq len (131072), (16.0 GiB KV cache is needed, which is larger than the available KV cache memory (2.83 GiB). Based on the available memory, the estimated maximum model length is 23184. Try increasing `gpu_memory_utilization` …# rerun with --max-model-len 8192INFO GPU KV cache size: 23,200 tokens, Maximum concurrency for 8,192 tokens per request: 2.83xINFO Graph capturing finished in … secs, took 0.60 GiB

Our running example fails at step 5. Llama 3.1 advertises a 131,072-token context, and without a max_model_len vLLM uses that. One request that long needs 16 GiB of KV cache, and the pool is 2.83 GiB. That is the whole content of the error everyone pastes into GitHub issues. The error even tells you what would fit: about 23,000 tokens. We’ll come back to the fix in part 4.

What each flag actually changes

With the steps in place, we can say precisely what each flag does [5], and it’s often not what people assume:

The useful consequence: the only big levers on the pool are the weights and the utilization fraction. max_model_len decides whether vLLM agrees to start, and max_num_seqs barely touches memory.

The terms of the subtraction

Putting the steps together, these are the terms that come out of the budget before the pool gets anything:

For the running example, the finished subtraction looks like this:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.9 --max-model-len 8192 --max-num-batched-tokens 8192INFO Available KV cache memory: 2.83 GiBINFO GPU KV cache size: 23,200 tokens, Maximum concurrency for 8,192 tokens per request: 2.83x

We’ve now seen every line of that box except the middle one. It’s the line that turns gigabytes into tokens, and the one people most often get wrong, so let’s look at it next.

KV bytes per token

The pool comes out of the subtraction in gigabytes, but requests are measured in tokens. The conversion rate between the two is how many bytes of KV cache one token costs:

KV per token = 2 (key and value)
             × num_layers
             × num_kv_heads
             × head_dim
             × bytes per value (2 for BF16)

vLLM does the same sum per block, with one more factor of block_size (16). The term that needs explaining is num_kv_heads.

In the original transformer, every attention head had its own keys and values. Grouped-query attention [6] lets a group of query heads share one set: the model still has many query heads, but only a few KV heads, and only those are cached. The query vectors themselves are recomputed on every step and never stored. Here is one token of the running example:

"hidden_size": 4096,"num_attention_heads": 32,"num_hidden_layers": 32,"num_key_value_heads": 8,# head_dim = hidden_size / num_attention_heads = 128

Here is the same sum for a few common models:

Look at the 128k column. On Llama 3.1 70B, one full-length conversation needs 40 GiB of KV cache, half an H100 on top of the weights. Long context isn’t something you switch on; it’s capacity you have to pay for.

With the cost of a token in hand, we have everything we need. Let’s finish the running example, then scale it up.

Scaling up: from one 4090 to two H100s

The running example, in full

Here is the whole subtraction for our 4090, including the conversion to tokens:

Card: RTX 4090, 23.5 GiB reported ·   weights BF16   ·   util 0.90

budget          23.5 × 0.90            =  21.15 GiB
− weights       8.03e9 × 2 bytes       =  14.96 GiB
− activations   (8192 batched tokens)  =   2.16 GiB
− CUDA graphs                          =   0.60 GiB
− CUDA context                         =   0.60 GiB
                                          ─────────
KV pool                                =   2.83 GiB

tokens = 2.83 GiB ÷ 128 KiB            =  23,200 tokens

As we saw in Figure 2, this fails the max_model_len check out of the box. Set --max-model-len 8192 and it starts, with room for two full-length requests at a time. That works, but it isn’t much of a server. Here is what each fix buys, ranked by what it costs you:

Quantising the weights did more than make room: it multiplied the pool by 3.3. None of the other terms changed, so every gigabyte saved on weights went straight into the pool. Keep an eye on that pattern; it shows up again at every scale.

$ vllm serve meta-llama/Llama-3.1-8B-Instruct --quantization fp8 --gpu-memory-utilization 0.9 --max-model-len 8192 --max-num-batched-tokens 8192INFO Model loading took 8.46 GiB memory and … secondsINFO Available KV cache memory: 9.33 GiBINFO GPU KV cache size: 76,448 tokens, Maximum concurrency for 8,192 tokens per request: 9.33x

70B at FP8 on one H100

Now let’s swap in a model nearly nine times the size, on a card a little over three times as big, and run the same subtraction:

Card: H100 80GB, 79.2 GiB reported ·  weights FP8  ·  util 0.90

budget          79.2 × 0.90            =  71.28 GiB
− weights       68.45e9 × 1 byte
                + 2.10e9 × 2 bytes     =  67.66 GiB
− activations                          =   3.92 GiB
− CUDA graphs + context                =   1.20 GiB
                                          ─────────
KV pool                                =  −1.50 GiB   →  won't start

The first line of the weights is the 68.45B parameters in the linear layers, at FP8. The second is the 2.10B in the embedding table and output head, which stay 16-bit. The weights alone fit on the card, which is why so many “can I run 70B on one H100?” answers say yes. But with the overheads they come to 72.8 GiB against a 71.3 GiB budget, and vLLM stops with “No available memory for the cache blocks”. Even at the 0.92 default, the pool is 0.08 GiB: 269 tokens.

Push utilization to 0.96 and the pool reaches 3.25 GiB, about 10,650 tokens. That’s one 8k request at a time with no headroom left, so in practice a 70B at FP8 doesn’t serve on one H100. There are two real options:

• INT4 weights: about 37 GiB (4-bit linear layers plus their group scales, 16-bit embeddings), which leaves a 29 GiB pool: about 95,000 tokens, or 11 concurrent requests at 8k. That’s a real server, with a quality question you have to answer by measuring.
• A second GPU, which is next.

The same model at tensor_parallel_size 2

How does this work in vLLM? With --tensor-parallel-size 2, vLLM starts one worker process per GPU, and each one runs the startup steps from Figure 2 on its own card. Three things change per GPU:

• The weights are split, so each GPU loads half.
• The attention heads are split too, so each GPU caches only its share of every token’s KV: half the bytes per token (more on this in Figure 7).
• NCCL buffers appear. The GPUs combine their partial results twice in every layer, and the communication buffers take about 0.9 GiB per card, out of the same budget.

Here is the subtraction for one of the two GPUs:

2 × H100 80GB  ·  weights FP8  ·  util 0.90  ·  tp=2

budget / GPU                           =  71.28 GiB
− weights / GPU   67.66 ÷ 2            =  33.83 GiB
− activations (sharded)                =   2.16 GiB
− CUDA graphs + context + NCCL        =   2.20 GiB
                                          ─────────
KV pool / GPU                          =  33.09 GiB

KV per token per GPU  320 KiB ÷ 2      =     160 KiB
tokens = 33.09 GiB ÷ 160 KiB           =  216,900 tokens

Two GPUs didn’t double the capacity. Compared with the most one card can give (10,650 tokens at 0.96), they multiplied it by twenty. The reason is the pattern from the 4090: the weights are a fixed cost on each card, and everything freed above that cost becomes pool. Halving the weights per GPU freed 34 GiB on each card, and halving the KV bytes per token made each of those gigabytes hold twice as many tokens.

It’s also why “does it fit on one GPU?” is the wrong question to ask when sizing hardware. The better one is: how many tokens of cache do I get per dollar?

# one H100, pushed to 0.96$ vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization fp8 --gpu-memory-utilization 0.96 --max-model-len 8192 --max-num-batched-tokens 8192INFO GPU KV cache size: 10,640 tokens, Maximum concurrency for 8,192 tokens per request: 1.30x# two H100s at 0.90$ vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization fp8 --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --max-model-len 8192 --max-num-batched-tokens 8192INFO GPU KV cache size: 216,848 tokens, Maximum concurrency for 8,192 tokens per request: 26.47x

Now, where did the halving of KV bytes per token come from? Tensor parallelism splits the attention heads across the GPUs, and the KV heads go with them. That works as long as there are heads to split:

def get_num_kv_heads(self, parallel_config, arch_config=None) -> int:    # If tensor parallelism is used, we divide the number of KV heads by    # the tensor parallel size. We will replicate the KV heads in the    # case where the number of KV heads is smaller than the tensor    # parallel size so each GPU has at least one KV head.    return max(1, total_num_kv_heads // parallel_config.tensor_parallel_size)

Size it yourself

Here is the same arithmetic as a calculator. Pick a model, a GPU and your flags, and it runs the subtraction, tells you whether vLLM will start, and prints the command:

vllm serve Llama-3.1-8B \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9 \
  --max-num-batched-tokens 8192

The calculator also has its own page, /tools/kv-calculator, for bookmarking or linking.

Estimates, not a simulator [9]. Activation and graph overheads are modelled from typical observed values and vary by version, attention backend and batch shape. Treat the result as the right order of magnitude and the right direction for each flag, then confirm against what vLLM logs at startup.

Failure modes, and which step each one breaks

With the full picture in place, each of the common failures maps to one step of Figure 2 or one term of the subtraction. Here are the five I see most often.

1. “KV cache is needed, which is larger than the available KV cache memory”

This is step 5: the pool is smaller than one request of max_model_len. It’s exactly what happened to our running example. Fix it in this order, cheapest first:

1. Lower max_model_len to the p99 context you actually serve, not the model’s advertised maximum.
2. Switch to --kv-cache-dtype fp8.
3. Quantise the weights.
4. Add a GPU.

Raising gpu_memory_utilization belongs last, not first, because it borrows from the headroom that failure mode 3 needs. If you only need the server to start, --max-model-len auto makes vLLM pick the largest length that fits. That tells you your capacity, though, not what your traffic needs.

2. OOM during startup, before the server is ready

This one happens in step 2, the profiling forward pass, so it isn’t about the pool at all: the pool doesn’t exist yet. The activation peak is too big. Halve --max-num-batched-tokens, then add --enforce-eager if you still need room. If you raised the batched-token budget to speed up prefill, this is the cost of that choice.

3. Starts fine, runs for twenty minutes, then OOMs

This usually means gpu_memory_utilization is at 0.95 or above. The pool was sized correctly, but there’s almost no memory left above the budget for allocator fragmentation and short-lived buffers, so eventually a particular batch shape asks for memory that no free region can satisfy. Drop to 0.88–0.90. If the model only fits at 0.95, it doesn’t really fit on this GPU, and the fix is one of the levers from part 4.

4. Works at tp=1, OOMs at tp=2

A new cost appears the moment you shard: NCCL communication buffers, roughly 0.5–1 GiB per GPU, taken from the same budget as the weights, so the pool shrinks by that much. A config sized to the last gigabyte on one GPU can fail on two for that reason alone.

There’s also a quieter limit: KV heads only split while tensor_parallel_size is at most the number of KV heads. Past that, every GPU holds a full copy of at least one head, so KV bytes per token per GPU stop shrinking and only the weight savings still grow the pool. With 8 KV heads, tp of 1, 2, 4 and 8 split the cache cleanly, but tp=16 buys much less than you’d expect (Figure 7).

5. No OOM, but throughput falls off a cliff under load

This isn’t a memory error but a memory symptom, and it’s Figure 1 happening at scale. The pool is full, so the scheduler preempts running requests, frees their blocks and later recomputes them, which means paying for the same prefill twice. Watch the preemption count in vLLM’s metrics: if it’s above zero under normal traffic, the pool is too small for your load. The fix is the same arithmetic, not a bigger max_num_seqs; raising max_num_seqs here usually makes it worse.

Putting it all together: a sizing checklist

1. Compute KV/token = 2 × layers × kv_heads × head_dim × bytes. Use the KV heads.
2. Compute the pool: the budget, minus the weights, minus about 3–5 GiB of overhead.
3. Divide. That’s your token capacity, and it’s what your deployment is really made of.
4. Set max_model_len from the p99 context you really serve, not the model’s maximum. This is the biggest lever most people never touch.
5. Leave gpu_memory_utilization at or just below the default (0.92), and treat any need to raise it as a sign the hardware is undersized.
6. Only then tune max_num_seqs, and tune it for latency, which is what it’s for.

Epilogue

We began with what the KV cache is and why vLLM hands it out in blocks, followed the running example through startup as vLLM sized its pool, worked out what one token costs, and scaled up to two H100s, where a second GPU bought twenty times the cache instead of two. Finally, we traced the common OOMs back to the step that causes each one.

This post skipped some things that change the arithmetic. E.g.:

• Mixture-of-experts [10]: only a couple of experts are active per token, but all of them stay resident, so the weight term uses total parameters while the compute uses active ones.
• Multi-head latent attention [11], which caches a compressed latent instead of full K and V and changes the per-token number by roughly an order of magnitude.
• Sliding-window attention [12], which caps KV per layer instead of letting it grow with context.
• LoRA adapters [13], which add their own resident weights per adapter.
• Quantised-KV quality, which deserves measurement rather than a rule of thumb.

The nice thing is that each of these changes one term of the subtraction, not its shape. Defaults move between vLLM versions and the overhead terms drift with the attention backend, but the method doesn’t. If a number here disagrees with what your server logs at startup, trust the server, then check which term you mis-estimated.

References

1. vLLM, github.com/vllm-project/vllm
2. “The Llama 3 Herd of Models”, arxiv.org/abs/2407.21783
3. “Attention Is All You Need”, arxiv.org/abs/1706.03762
4. “Efficient Memory Management for Large Language Model Serving with PagedAttention”, arxiv.org/abs/2309.06180
5. vLLM docs: Engine Arguments, docs.vllm.ai/en/latest/configuration/engine_args.html
6. “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”, arxiv.org/abs/2305.13245
7. vLLM docs: Quantized KV Cache, docs.vllm.ai/en/latest/features/quantization/quantized_kvcache.html
8. “FP8 Formats for Deep Learning”, arxiv.org/abs/2209.05433
9. vLLM docs: Optimization and Tuning, docs.vllm.ai/en/latest/configuration/optimization.html
10. “Mixtral of Experts”, arxiv.org/abs/2401.04088
11. “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model”, arxiv.org/abs/2405.04434
12. “Mistral 7B”, arxiv.org/abs/2310.06825
13. “LoRA: Low-Rank Adaptation of Large Language Models”, arxiv.org/abs/2106.09685