The attention cache (KV cache) stores the keys and values already computed for previous tokens, so they do not have to be recomputed for each new token. It speeds up generation, but occupies memory that grows with the context length and the number of simultaneous requests.
Example
Doubling the context length roughly doubles the memory used by the KV cache.