LLM Inference Costs Are Driven by Memory Bandwidth, Not Compute FLOPs
Most discussions around large language model infrastructure costs focus on FLOPs and GPU compute, but the real bottleneck during inference is memory bandwidth — specifically the cost of moving the…
Read the full articleYou might also wanna read
Research Directions for Overcoming Memory and Interconnect Challenges in Large Language Model Inference Hardware
Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundam
arxiv.org·5mo ago
Roofline Model for Estimating Speculative Decoding Speedup in LLM Inference
A roofline model for estimating the optimal speculative-decoding draft length and the speedup it yields across models, hardware, and batch s
modal.com·23d agoAccelerating Large-Scale LLM Inference on AMD Instinct MI350X/MI355X with Eagle3 and AMD Quark
Large language model (LLM) inference is increasingly constrained by autoregressive decoding. Even when prefill is highly optimized, the deco
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
arXiv:2607.08057v1 Announce Type: cross Abstract: Despite the rapid advancements of large language models (LLMs), LLM serving systems remain
Speculative Speculative Decoding: Parallelizing LLM Inference for Faster Performance
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by
arxiv.org·4mo ago
Hardware-Aware Dynamic Speculative Decoding: Optimizing LLM Inference Through Adaptive Draft Token Selection
See how DSD overcomes the limitations of standard SD by controlling the optimal K based on hardware constraints.
cohere.com·4d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.