Attention Matching: Fast KV Cache Compaction for Language Models
By
cbracketdash
3mo ago· 2 min readenInsight
75/100
Toasty
Bagelometer↗
Crusty in the right places. Worth the chew.
Score75TypeanalysisSentimentpositive
Summary
This article presents a new approach called Attention Matching for fast key-value (KV) cache compaction in language models. Traditional methods for handling long contexts through summarization are lossy and harm performance, while existing latent space compaction methods like Cartridges require slow, expensive optimization. The proposed Attention Matching technique constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. This formulation decomposes into simple subproblems with efficient closed-form solutions, achieving up to 50x compaction in seconds with minimal quality loss.
Key quotes
· 5 pulledScaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache.
Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization.
This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level.
We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions.
Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.
Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly los
