Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
arXiv:2607.08057v1 Announce Type: cross Abstract: Despite the rapid advancements of large language models (LLMs), LLM serving systems remain memory-intensive and costly. The key-value (KV) cache…
Read the full articleYou might also wanna read
LMCache: A KV Cache Management Layer for Scalable LLM Inference
Learn how LMCache reduces TTFT and improves throughput for LLM inference with tiered KV cache offloading, non-prefix reuse, PD disaggregatio
Expected Attention: KV Cache Compression Method for Efficient LLM Inference
Memory consumption of the Key-Value (KV) cache represents a major bottleneck for efficient large language model inference. While attention-s
arxiv.org·9mo ago
Attention Matching: Fast KV Cache Compaction for Language Models
Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts
arxiv.org·4mo ago
Research Directions for Overcoming Memory and Interconnect Challenges in Large Language Model Inference Hardware
Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundam
arxiv.org·5mo ago
Multi-Stream LLMs: A Parallel Architecture to Overcome Single-Stream Bottlenecks in Language Models
The continued improvements in language model capability have unlocked their widespread use as drivers of autonomous agents, for example in c
arxiv.org·1mo agoDepthWeave-KV: Unlocking Long Context Efficiency
DepthWeave-KV tackles long-context LLM memory bottlenecks with token-adaptive cache compression, achieving 8.3x reduction and high throughpu
Comments
Sign in to join the conversation.
No comments yet. Be the first.