All Topics

Technology

Art

Debugging Envoy Load Balancer Latency with eBPF Zero-Code Instrumentation

sergiocipriano

5mo ago· 7 min readen

100/100

Golden Brown

Bagelometer↗

A five-star bake. Worth schmearing, sharing, saving.

Score100Typehow-toSentimentneutral

Summary

The article describes a technical solution for debugging an Envoy Network Load Balancer using eBPF (Extended Berkeley Packet Filter) for zero-code instrumentation. The author faced HTTP 499 errors caused by latency in their cloud infrastructure, and Envoy's built-in options were insufficient for identifying bottlenecks. The solution involves using eBPF to instrument the TCP proxy without modifying code, allowing for detailed latency analysis and troubleshooting of network performance issues in a Load Balancer as a Service (LBaaS) environment.

Key quotes

· 5 pulled

We were seeing a small number of HTTP 499 errors caused by latency somewhere in our cloud, but it wasn't clear what the bottleneck was.

My team is responsible for the LBaaS product (Load Balancer as a Service) and, of course, we are the first suspects when this kind of problem appear.

Before going for the current solution, I read a lot of Envoy's documentation.

The options Envoy provides just weren't enough.

As a result, each team had to set up additional instrumentation to catch latency spikes and figure out what was going on.

Snippet from the RSS feed

I recently had to debug an Envoy Network Load Balancer, and the options Envoy provides just weren't enough. We were seeing a small number of HTTP 499 errors caused by latency somewhere in our cloud, but it wasn't clear what the bottleneck was. As a result

You might also wanna read

Why average CPU utilization is a misleading metric for cloud-native applications

The article discusses the pitfalls of relying on average CPU utilization metrics in cloud-native environments, particularly in Kubernetes. I

theocharis.dev·9d ago

Agumbe: AI-Powered Workspace Platform for Kubernetes Application Development

Agumbe is a platform that provides AI-powered workspaces for building and running applications on Kubernetes. It helps teams go from idea to

Product Hunt·2mo ago

Towlion: Self-Hosted Micro-PaaS for GitHub-Based Application Deployment

Towlion is a self-hosted micro-PaaS (Platform as a Service) that enables developers to deploy full web applications directly from GitHub to

towlion.github.io·2mo ago

aws-doctor: Open-Source CLI Tool for AWS Security, Cost, and Best Practices Auditing

aws-doctor is an open-source command-line tool written in Golang that performs comprehensive health checks on AWS accounts. It audits securi

github.com·4mo ago

Netflix's Simian Army: Testing Cloud Reliability Through Intentional Failures

Netflix discusses their cloud infrastructure reliability strategy called the "Simian Army" - a suite of tools designed to test and improve s

netflixtechblog.com·4mo ago

Wozz: Kubernetes Cost Optimization Tool for Preventing Resource Waste

Wozz is a Kubernetes cost optimization tool that helps engineering teams reduce cloud spending through two main approaches: a PR Cost Linter

github.com·5mo ago