PEEK: Revolutionizing Online LLM Serving with Smart Scheduling
PEEK introduces a novel scheduling framework that significantly boosts cache hit rates and throughput for online LLM serving. By leveraging a radix tree and dual-walk mechanism, it delivers…
Read the full articleYou might also wanna read
LMCache: A KV Cache Management Layer for Scalable LLM Inference
Learn how LMCache reduces TTFT and improves throughput for LLM inference with tiered KV cache offloading, non-prefix reuse, PD disaggregatio
DepthWeave-KV: Unlocking Long Context Efficiency
DepthWeave-KV tackles long-context LLM memory bottlenecks with token-adaptive cache compression, achieving 8.3x reduction and high throughpu
MCP Caching Strategies: Prompt Caching, Server-Side Caching, Semantic Caching, and Gateway Patterns
A comprehensive guide to caching in MCP systems — covering Anthropic prompt caching (90% cost reduction), FastMCP ResponseCachingMiddleware,
How New Open-Weight LLMs Are Reducing Long-Context Costs: KV Sharing, Attention Budgeting, and Compressed Attention
From Gemma 4 to DeepSeek V4, How New Open-Weight LLMs Are Reducing Long-Context Costs
magazine.sebastianraschka.com·1mo ago
Optimizing LLM Inference: A C++ Backend for VRAM-Aware Sequence Packing
A comprehensive guide to optimizing LLM inference by eliminating padding overhead with hardware-aware sequence packing.
towardsdatascience.com·1mo ago
Competitive analysis of online paging algorithms may be based on a flawed model assumption
In any real system a newly computed datum begins its existence in the processor rather than in external memory, and thus does not inevitably
arxiv.org·17d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.