Writing
- Inside the KV Cache: The Life of a Gigabyte
From gpu_memory_utilization and max_model_len to weights, activations, CUDA graphs and the KV pool: where every gigabyte on the card actually goes.
Get notified when I publish a new post.
Or follow the RSS feed.