Alibaba Cloud's Aegaeon System Reduces Nvidia GPU Requirements by 82% for AI Inference

Summary

Alibaba Cloud has developed a new GPU pooling system called Aegaeon that significantly reduces the number of Nvidia GPUs needed for large language model inference. During a multi-month beta test in Alibaba's Model Studio marketplace, the system reduced GPU requirements by 82%, allowing 213 H20 GPUs to handle workloads that previously required 1,192 GPUs. The technology, detailed in a peer-reviewed paper presented at the 2025 ACM Symposium on Operating Systems, uses token-level scheduling to enable one GPU to serve multiple LLMs simultaneously, potentially helping cloud providers extract more capacity from existing silicon, particularly in constrained markets like China.