Why CPU-Based Autoscaling Fails for GPU Inference — and How KEDA Fixes It
By
Sneha Gullapalli
Summary
This article explains why CPU utilization is a poor autoscaling signal for GPU inference workloads, using a real-world incident where an HPA configured with targetCPUUtilizationPercentage failed to scale during a traffic spike, causing latency to spike. It argues that CPU is a decorrelated proxy on GPU pods — it can sit idle while the GPU is pinned and requests pile up. The article recommends scaling on signals actually on the critical path: queue depth (backlog) and GPU utilization, using KEDA (Kubernetes Event-Driven Autoscaling) for faster, more reliable scaling of AI workloads.
Source
bskyWhy CPU-Based Autoscaling Fails for GPU Inference — and How KEDA Fixes Ithackernoon.comKey quotes
· 3 pulledOn a GPU inference pod, CPU is a decorrelated proxy: it can sit at 30% while the accelerator is pinned and requests pile up behind it.
Scale on the signal actually on the critical path—backlog and GPU utilization—with KEDA.
Replica count stayed flat at two for eleven minutes while p99 latency went vertical and requests stacked up.
You might also wanna read
Scaling Karpathy's Autoresearch: Parallel GPU Processing Enables New AI Experimentation Strategies
The article describes an experiment where researchers scaled Andrej Karpathy's autoresearch system by giving it access to 16 GPUs on a Kuber
blog.skypilot.co·3mo agoDynamic batching: a practical how-to guide
How Moondream's Photon Engine Eliminates the GPU Bubble for Faster AI Inference
Moondream's engineering team explains the "GPU bubble" phenomenon — where GPUs sit idle waiting for CPU instructions during AI model inferen
moondream.ai·7d ago
Systalyze's Utilyze Tool Reveals True GPU Compute Utilization in AI Workloads
Systalyze introduces Utilyze, a GPU compute utilization monitoring tool that reveals actual compute throughput versus traditional metrics li
systalyze.com·2mo ago
AutoKernel: Autonomous AI System for GPU Kernel Optimization in PyTorch Models
AutoKernel is an autonomous AI system that automatically optimizes GPU kernels for PyTorch models. Inspired by autonomous AI research agents
github.com·3mo ago
How Memory Snapshots Reduce GPU Cold Starts for AI Inference Workloads
Cerebrium presents a technical deep-dive on solving GPU cold starts in serverless AI inference using memory snapshots. The article explains
cerebrium.ai·6d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.