How ByteDance Scales Offline Inference with multi-modal LLMs to 200 TB Data
Read the full article2y ago
You might also wanna read
Inside vLLM: Anatomy of a High-Throughput LLM Inference System
From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale.
aleksagordic.com·10mo ago
RTP-LLM: Alibaba's High-Performance Inference Engine for Large Language Model Deployment
Large Language Models (LLMs) have revolutionized AI applications, but deploying them at scale presents significant challenges. We present RT
arxiv.org·1mo ago
Modular: Modular Platform 25.5: Introducing Large Scale Batch Inference
Modular Platform 25.5: Introducing Large Scale Batch Inference
modular.com·11mo ago
Research Directions for Overcoming Memory and Interconnect Challenges in Large Language Model Inference Hardware
Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundam
arxiv.org·5mo ago
LMCache: A KV Cache Management Layer for Scalable LLM Inference
Learn how LMCache reduces TTFT and improves throughput for LLM inference with tiered KV cache offloading, non-prefix reuse, PD disaggregatio
pyshine.com·15d ago
Tokasaurus: An LLM Inference Engine for High-Throughput Workloads
Article URL: Comments URL: Points: 14 # Comments: 0
scalingintelligence.stanford.edu·1y ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.