Expected Attention: KV Cache Compression Method for Efficient LLM Inference
By
sonabinu
A second-rack bagel that's nearly first-rack. Tasty stuff.
Summary
This research paper introduces Expected Attention, a training-free method for compressing Key-Value (KV) cache in large language models to reduce memory consumption during inference. The approach overcomes limitations of existing attention-score-based pruning by estimating how future queries will attend to KV pairs, using distributional properties of LLM activations to compute expected attention scores in closed form. The method enables principled KV pair ranking and pruning with minimal performance impact, working across both prefilling and decoding phases while outperforming state-of-the-art baselines. The researchers also released KVPress, a comprehensive library for implementing and benchmarking KV cache compression methods.
Key quotes
· 5 pulledMemory consumption of the Key-Value (KV) cache represents a major bottleneck for efficient large language model inference.
We introduce Expected Attention, a training-free compression method that estimates KV pairs importance by predicting how future queries will attend to them.
Our approach leverages the distributional properties of LLM activations to compute expected attention scores in closed form for each KV pair.
Our method operates seamlessly across both prefilling and decoding phases, consistently outperforming state-of-the-art baselines in both scenarios.
We release KVPress, a comprehensive library to enable researchers to implement and benchmark KV cache compression methods, already including more than 20 techniques.
