Systemd cgroup driver bug: containerd CPU quota calculation mismatch causes container creation failures
By
dropbox_miner
Slow-proofed and worth the wait. Worth its weight in flour.
Summary
The article describes a technical bug in container orchestration systems where using the systemd cgroup driver with a CPU limit of 4096m causes intermittent pod creation failures. The issue stems from containerd non-deterministically calculating either 409600 or 410000 microseconds for parent cgroups, while runc consistently calculates 410000 for child cgroups. When these calculations mismatch, the Linux kernel rejects child cgroup creation with an "invalid argument" error. The article provides technical details about the root cause and implications for container management systems.
Key quotes
· 4 pulledWhen using the systemd cgroup driver with a CPU limit of 4096m, pod creation fails intermittently because containerd non-deterministically calculates either 409600 or 410000 microseconds for the parent cgroup, while runc consistently calculates 410000 for child cgroups.
When they mismatch, the Linux kernel rejects the child cgroup creation with 'invalid argument'.
Investigation reveals non-deterministic behavior in containerd when converting 4096m to microseconds.
Containerd (when creating pod sandbox) - INCONSISTENT: Sometimes calculates: 4096m → 409600 microseconds
You might also wanna read
Bootc and OSTree: Modern Approaches to Linux System Deployment and Configuration Management
The article explores using Bootc and OSTree as modern solutions for Linux system deployment, focusing on achieving reproducible, consistent
a-cup-of.coffee·3mo ago
systemd v259 Announces Deprecation of System V Service Scripts, Removal Planned for v260
systemd v259 announces the deprecation and upcoming removal of System V service script support in v260, requiring users to migrate to native
github.com·5mo ago
Technical Analysis and Critique of D-Bus Implementation in Linux Systems
The article is a technical critique of D-Bus, a message bus system used in Linux desktop environments. The author acknowledges D-Bus's usefu
blog.vaxry.net·5mo ago
Linux Input Stack Architecture: A Comprehensive Technical Overview
This comprehensive technical article provides an in-depth architectural overview of the Linux input stack, explaining how input devices and
venam.net·6mo ago
Production Container Deployment with Podman Quadlets and Systemd Integration
This article provides a practical guide for deploying production-grade container applications using Podman Quadlets and systemd integration,
Performance Analysis of Zram Compression Algorithms and System Impact
This article provides a technical analysis of Zram, a Linux kernel module for compressed virtual memory. It examines the performance of diff
notes.xeome.dev·7mo ago