APEX4: Platform-Dependent W4A4 LLM Inference via Intra-SM Compute Rebalancing
By
[Submitted on 7 Jun 2026]
A respectable bake. You'd come back tomorrow for another.
Summary
This paper presents APEX4, a system for efficient W4A4 (4-bit weights, 4-bit activations) LLM inference that addresses the bottleneck of group dequantization overhead on CUDA Cores. Through systematic benchmarks across Ampere and Ada GPUs, the authors identify the Tensor Cores to CUDA Cores throughput ratio (ρ) as the key hardware indicator determining W4A4 viability. They find that W4A4-g128 kernels yield 2.0-2.5× speedup on RTX 3090 (ρ=16) but degrade to 0.43-0.47× on A100 (ρ=64), showing performance is platform-dependent. APEX4 co-designs pure INT4 GEMM kernels with ρ-aware granularity adaptation, achieving perplexity within 0.63 of FP16 on LLaMA-2-70B and delivering up to 1.78× end-to-end speedup on RTX 3090 and 2.09× on A40 when deployed in vLLM.
Key quotes
· 5 pulledWe present the first systematic study of how intra-SM compute balance governs this bottleneck.
Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio (ρ) as the primary hardware indicator
the W4A4-g128 kernel yields 2.0--2.5× speedup on RTX~3090 (ρ=16) yet degrades to 0.43--0.47× on A100 (ρ=64) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible
APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0%--4.4% in zero-shot accuracy
Deployed as a drop-in replacement in unmodified vLLM, it delivers up to 1.66× end-to-end speedup on L40S (ρ=8), and 1.78× on RTX~3090 (ρ=16), 2.09× on A40 (ρ=16), while recovering A100 (ρ=64) to 1.20--1.40× via the mixed-granularity mode
You might also wanna read
Technical Analysis of LLM Inference Engines: Exploring Nano-vLLM Architecture and Scheduling
This article provides an in-depth technical exploration of LLM inference engines, focusing on Nano-vLLM as a case study. It explains the cri
neutree.ai·4mo ago
Optimizing LLM Inference by Combining NVIDIA DGX Spark and Apple Mac Studio Architectures
The article explores combining NVIDIA DGX Spark AI supercomputers with Apple Mac Studio systems to optimize large language model (LLM) infer
Unsloth and NVIDIA Partner to Accelerate LLM Fine-Tuning by 20%
Unsloth has partnered with NVIDIA to optimize fine-tuning of large language models, achieving 20% faster training speeds. The collaboration
Unsloth - Train and Run Models Locally·1mo ago
Rotary GPU: Enabling Large Mixture-of-Experts Models on Consumer Laptop GPUs with Limited Memory
This paper presents Rotary GPU, an exploratory approach to running large Mixture-of-Experts (MoE) language models on consumer-grade hardware
arxiv.org·10d ago
Guide to Optimizing GEMM Operations with AMD RDNA 4 Architecture GPUs
The article discusses the use of WMMA intrinsics with AMD RDNA 4 architecture GPUs, highlighting the improvements in performance for General
gpuopen.com·10mo ago
Guide to Calculating GPU Memory for Self-Hosted LLM Inference
The article provides a guide on calculating GPU memory requirements and managing concurrent requests for self-hosted large language model (L
Product Hunt·10mo ago