Reverse-engineering NVIDIA's cuda-checkpoint: Achieving 4x faster CUDA process restoration
This article reverse-engineers NVIDIA's closed-source cuda-checkpoint utility, which allows freezing a running CUDA process, serializing its GPU state to host memory, and restoring it later. The authors investigate why checkpoint transfers fail to saturate PCIe bandwidth, and use their understanding of the driver's internals to develop techniques that restore CUDA processes up to 4x faster. The work builds on earlier findings where they sped up SGLang server startup by up to 70x using this checkpoint/restore mechanism.
Key quotes
There's a little known feature in the closed-source NVIDIA driver that lets you freeze a running CUDA process, serialize its GPU state into host memory, and later restore it to the GPU exactly as it was.
One very frustrating aspect, that dogs anyone trying to use it to checkpoint complex GPU processes, is that the checkpoint transfers come nowhere close to saturating PCIe bandwidth.
understanding that lets us restore CUDA processes up to 4x faster.
From the article
You might also wanna read
Converting FP8 Quantized CLIP Checkpoints to TensorRT Engines for Production Inference
This article provides a technical walkthrough for converting FP8-quantized checkpoints (specifically a CLIP model) into NVIDIA TensorRT engi
developer.nvidia.com·22d ago
Modular: How to Beat Unsloth's CUDA Kernel Using Mojo—With Zero GPU Experience
Building a GPU backend for Emacs: A technical deep dive into hardware-accelerated text rendering
The author documents their personal journey of building a GPU-accelerated rendering backend for Emacs, driven by curiosity about why the edi
en.andros.dev·13d ago
Modular: How did CUDA succeed? (Democratizing AI Compute, Part 3)
Dynamic GPU Capacity Controller Reallocates Idle Production GPUs to Research During Off-Peak Hours
The article describes how the authors built a capacity controller that dynamically reallocates GPUs between production inference workloads a
runwayml.com·6d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.