Jenga: A Memory Allocation Framework for Heterogeneous LLM Inference
Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory efficiently is…
Read the full articleYou might also wanna read
Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices
arXiv:2607.08786v1 Announce Type: new Abstract: With the growing deployment of large language models (LLMs), LLM inference cost has become a
Building high-performance expert-parallel dispatch and combine kernels for MoE LLM inference
How expert-parallel dispatch and combine kernels work, built up from scratch: the high-throughput shape and the low-latency one.
fergusfinn.com·1mo agoInside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.
Guide to Calculating GPU Memory for Self-Hosted LLM Inference
Calculate GPU memory requirements and max concurrent requests for self-hosted LLM inference. Support for Llama, Qwen, DeepSeek, Mistral and
Product Hunt·11mo ago
RTP-LLM: Alibaba's High-Performance Inference Engine for Large Language Model Deployment
Large Language Models (LLMs) have revolutionized AI applications, but deploying them at scale presents significant challenges. We present RT
arxiv.org·1mo agoTowards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
arXiv:2607.08057v1 Announce Type: cross Abstract: Despite the rapid advancements of large language models (LLMs), LLM serving systems remain
Comments
Sign in to join the conversation.
No comments yet. Be the first.