Understanding How Bazel Build Caching and glibc Version Mismatches Cause Production Failures
By
goranmoomin
Hand-rolled, kettle-boiled, baked to perfection. Worth every minute at the bakery.
Summary
This technical article examines how Bazel's build caching and remote execution features can interact dangerously with different glibc (GNU C Library) versions across development environments, leading to production failures. The piece explains how seemingly successful builds can produce artifacts that crash in production due to version mismatches, specifically addressing the 'GLIBC_2.28 not found' error scenario. It explores action non-determinism in build systems and provides insights into preventing these subtle but critical deployment issues.
Key quotes
· 4 pulledImagine this scenario: your team uses Bazel for fast, distributed C++ builds. A developer builds a change on their workstation, all tests pass, and the change is merged.
The CI system picks it up, gets a cache hit from the developer's build, and produces a release artifact. Everything looks green.
But when you deploy to production, the service crashes with a mysterious error: version 'GLIBC_2.28' not found.
The answer lies in the subtle but dangerous interaction between Bazel's caching, remote execution, and differing glibc versions across your fleet.
You might also wanna read
Microsoft's Evolution of .NET Build System: From Distributed to Unified Build Architecture
The article details Microsoft's evolution of the .NET build system from a distributed model to a unified approach called 'Unified Build.' It
devblogs.microsoft.com·6mo ago
Guide to Migrating from Legacy Macros to Symbolic Macros in Bazel
This technical article explains the migration from legacy macros to symbolic macros in Bazel, a build system. Symbolic macros were introduce
tweag.io·6mo ago
Why Average LLM Use Is Likely Destroying Value in Software Development
The author argues that, contrary to prevailing hype, the average use of Large Language Models (LLMs) is likely destroying value rather than
unessays.substack.com·16h ago
How AI Accelerated Prototyping: From Idea to Tangible in Record Time
The author reflects on how AI has transformed their prototyping workflow. Previously, the biggest bottleneck was the time needed to scaffold
darylcecile.net·16h ago
GitLab 19.0 launches with Secrets Manager, agentic workflows, and self-hosted AI models
GitLab 19.0 has been released, positioning itself as an intelligent orchestration platform for DevSecOps. The release includes expanded secr
bit.ly·1d ago
Centralizing Error Handling in Rust with Custom AppError Enums
This article discusses the importance of centralizing error handling in Rust applications using a custom AppError enum combined with map_err