eBPF Makes Production Debugging Feel Like Cheating - A Practical Intro for App Engineers

The last time a service of mine went slow in production and the dashboards showed nothing, I did what every application engineer does: added a log line, waited for a deploy, added another log line, waited for another deploy. Two hours to find out a DNS lookup was taking 5 seconds because a resolver had gone stale. The information was sitting in the kernel the entire time. I just did not know how to ask for it.

eBPF is how you ask for it. And the thing nobody tells you until you have used it: you do not need to redeploy, you do not need to change your code, and you do not need to be a kernel engineer. You attach a probe to a running process, the kernel runs your tiny program on every relevant event, and you get answers in seconds. The first time it feels like cheating.

This post is for the application engineer who has never written a line of C for the kernel and does not intend to. We will start with how everyone debugs today, show exactly where it falls apart, and then use eBPF to close each gap.

The Naive Approach and Where It Breaks

Here is the debugging toolkit most of us actually use in production:

• Add log lines, redeploy, read logs.
• top and htop to see if CPU or memory is high.
• Application-level metrics (request duration, error rate) from a Prometheus client.
• If desperate, strace -p <pid> on one process for a few seconds.

This works until it does not, and it stops working at exactly the moments you care about most.

Logs require a deploy and only show what you already thought to log. The bug you are chasing is by definition the thing you did not anticipate. Every log-and-redeploy cycle is 5-15 minutes, and each one only answers the question you guessed at.

Application metrics stop at the application boundary. Your histogram says the request took 800ms. It does not say that 750ms of that was a connect() syscall blocked on a slow TCP handshake to a downstream service. The slowness happened below your code, so your instrumentation cannot see it.

strace is a trap in production. It uses ptrace, which stops the target process on every syscall to hand control to strace. On a busy service that can slow the process by 10-100x. I have seen an engineer strace a production process and trip the very latency alert they were investigating. You cannot leave it running, and you cannot point it at all your processes at once.

The common failure across all three: they either require a code change, or they cannot see below your application, or they are too expensive to run on real traffic. What you want is to observe the kernel boundary - syscalls, network events, scheduler decisions - on live production processes, with near-zero overhead, without touching your code.

That is the exact shape of the problem eBPF was built for.

What eBPF Actually Is (The 90-Second Version)

eBPF lets you load a small program into the running kernel and attach it to an event: a syscall entry, a function return, a network packet, a tracepoint. When that event fires, the kernel runs your program. Your program can read arguments, measure time, increment counters, and stash data in shared maps that user space reads.

Three properties make this safe enough to run in production:

  your probe (C or a bpftrace one-liner)
          |
     compiled to eBPF bytecode
          |
     [ verifier ]  <- rejects loops, bad memory access, unbounded work
          |
     JIT-compiled to native code
          |
   attached to a kernel event, runs on every hit
          |
     writes to a map  ---> user-space tool reads and prints

1. The verifier statically proves your program terminates and touches only memory it is allowed to. A program that could loop forever or read arbitrary kernel memory is rejected at load time. This is why you can run it on a production box without fear of a kernel panic.
2. It is JIT-compiled to native instructions, so a probe on a hot syscall costs nanoseconds, not the microseconds ptrace costs.
3. It is attached, not injected. Your application does not know it is being observed and does not need to be restarted.

You will almost never write raw eBPF C as an application engineer. You will use tools built on top of it. The two worth knowing are bpftrace for ad-hoc one-liners on a single host, and Pixie for always-on, cluster-wide observability in Kubernetes.

bpftrace: A One-Liner Replaces the Deploy Cycle

bpftrace is awk for the kernel. You write a short script that says “on this event, do this,” and it handles compilation, loading, and printing. Install it (apt install bpftrace on Debian/Ubuntu, or run the packaged container) and you have a debugger that sees everything.

Start with the question the metrics could not answer: which syscalls are slow, and for whom?

Tracing syscall latency

This measures the wall-clock time of every read syscall system-wide and prints a histogram when you hit Ctrl-C:

// read-latency.bt
tracepoint:syscalls:sys_enter_read
{
    @start[tid] = nsecs;
}

tracepoint:syscalls:sys_exit_read
/@start[tid]/
{
    @us = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
}

Run it with bpftrace read-latency.bt. The output is a log-scale histogram in microseconds:

@us:
[0, 1)            4021 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1, 2)            1897 |@@@@@@@@@@@@@@@@@@@                      |
[2, 4)             612 |@@@@@@                                  |
...
[8K, 16K)           37 |                                        |
[16K, 32K)          14 |                                        |

Most reads finish in under a microsecond. But there is a tail out at 16-32 milliseconds. That tail is what your P99 latency alert is made of, and it never showed up in an application metric because your code just called read() and waited. Now you can see it, and you can narrow by process, by file descriptor, by argument.

The key idea: tid (thread ID) as a map key lets you match an entry event to its exit event and compute a duration. That single pattern - stash a timestamp on entry, subtract on exit - is most of practical tracing.

Which processes are making a syscall

Suppose something is hammering openat and you do not know what. Count by process name:

bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

@[systemd-journal]: 44
@[nginx]: 1201
@[my-api-server]: 88134

my-api-server is opening files 88,000 times in the window. That is a config file being re-read on every request, or a missing cache. You found it in one line and zero deploys.

Catching Slow Network Connections Before They Time Out

The most valuable eBPF trick for application engineers: seeing network latency at the syscall level, before it becomes a timeout your users report.

Your application sees a timeout as an exception 30 seconds later. The kernel saw the connect() syscall block from the first millisecond. bpftrace ships with pre-built tools for exactly this. tcpconnect shows every outbound TCP connection as it happens; tcplife shows the full lifetime and duration of each connection.

Here is a focused script that flags any connect() that takes longer than 100ms:

// slow-connect.bt
tracepoint:syscalls:sys_enter_connect
{
    @start[tid] = nsecs;
}

tracepoint:syscalls:sys_exit_connect
/@start[tid]/
{
    $ms = (nsecs - @start[tid]) / 1000000;
    if ($ms > 100) {
        printf("%-16s pid=%-7d connect took %d ms\n", comm, pid, $ms);
    }
    delete(@start[tid]);
}

my-api-server    pid=20344   connect took 3021 ms
my-api-server    pid=20344   connect took 3009 ms
payment-worker   pid=88120   connect took 214 ms

Two 3-second connects from the API server. That is a downstream host whose TCP handshake is slow, almost always DNS returning a bad address or a SYN being silently dropped by a firewall. You are watching it happen live, and you know the exact process and timing before a single request has actually timed out. Compare that to the old path: wait for the timeout, read the stack trace, guess.

For DNS specifically, gethostlatency (another bundled tool) traces the resolver library calls and prints how long each name lookup took. A stale resolver adding 5 seconds to lookups is invisible to your app metrics and glaringly obvious here.

Profiling CPU Hot Paths Without a Redeploy

The other place application metrics go dark: CPU. Your service is pinned at 100% CPU and you have no profiler wired in. The classic answer is “add a profiler, redeploy, reproduce the load.” eBPF lets you profile a running process, right now.

The technique is sampling: interrupt the CPU 99 times a second, record the stack trace of whatever is running, and aggregate. Do this for 30 seconds and the stacks that show up most often are your hot paths.

# Sample all CPUs at 99 Hz for 30 seconds, collapse to stack counts
profile -F 99 -a -d 30 > stacks.txt

# Or scoped to one process:
profile -F 99 -p 20344 -d 30 > stacks.txt

profile is part of the bcc tools and is itself an eBPF program. The output is a set of folded stacks with counts, which you feed straight into a flame graph:

flamegraph.pl stacks.txt > profile.svg

The flame graph shows exactly which functions the CPU is spending time in, all the way down through your language runtime into kernel functions. No agent, no code change, no redeploy, negligible overhead because it is sampling at 99 Hz rather than instrumenting every call. On a service that was mysteriously CPU-bound, I have found the culprit - a JSON serializer being called in a loop it did not need to be in - in under two minutes this way.

The overhead comparison is the whole argument:

Pixie: eBPF Without Writing Any Probes

bpftrace is perfect for “SSH into the box and ask a question.” It does not scale to “always watch every service in a 200-pod cluster.” That is where Pixie fits.

Pixie is a CNCF project that installs as a DaemonSet and uses eBPF to automatically capture, with no instrumentation, application traffic across your whole cluster: HTTP, gRPC, DNS, MySQL, Postgres, Redis, Kafka, and more. Because it works at the syscall level, it sees full request and response bodies without you adding a single line of tracing code or a sidecar.

Kubernetes node
+-------------------------------------------+
|  pod A    pod B    pod C                  |
|    \        |        /                    |
|     syscalls (read/write/connect/...)     |
|              |                            |
|       [ Pixie eBPF probes ]  <- DaemonSet |
|              |                            |
|     in-memory columnar store on node      |
+-------------------------------------------+
              |
        query with PxL / Grafana

The design that makes it practical: data is stored in memory on the node where it was collected and queried on demand, rather than shipped to a central store and retained forever. That keeps overhead and cost bounded, at the price of a short retention window. You get automatic service maps, per-endpoint latency, and full-body request tracing that you can turn on the moment an incident starts.

A PxL query to get the slowest HTTP requests in the last minute looks like this:

import px

df = px.DataFrame('http_events', start_time='-1m')
df.latency_ms = df.latency_duration_ns / 1e6
df = df[df.latency_ms > 200]
px.display(df[['time_', 'pod', 'req_path', 'resp_status', 'latency_ms']])

No SDK in your services produced that data. eBPF read it off the wire at the syscall boundary. The trade-off versus bpftrace is clear: Pixie is always-on and cluster-wide but is a platform you operate; bpftrace is a scalpel you pull out for a specific question on a specific host.

The Rough Edges

eBPF is not magic, and pretending otherwise leads to disappointment.

Kernel version matters. The good stuff (BTF, CO-RE, most modern tracepoints) wants Linux 5.4+ and ideally 5.8+. On an old CentOS 7 box with a 3.10 kernel, half of this does not work. Check uname -r before you get excited.

You need privileges. Loading eBPF programs requires CAP_BPF/CAP_SYS_ADMIN or root. In a locked-down managed Kubernetes environment you may not be allowed to run bpftrace in a pod, and getting Pixie approved is a security-team conversation, not a helm install.

Tracing dynamic language internals is harder. Syscalls and C-level functions are easy. Getting a clean Python or JVM stack out of a sampling profiler needs the runtime built with frame pointers or an interpreter-aware unwinder, or your flame graph is a wall of [unknown].

It shows you what, not always why. eBPF will tell you a connect() took 3 seconds. It will not tell you the firewall rule that dropped the SYN. It closes the gap between “the app is slow” and “the kernel is blocked on X,” which is usually 80% of the investigation, but the last mile is still yours.

What to Actually Do

If you take one thing from this: install bpftrace on a non-critical host this week and run the read-latency script against something. The concept clicks the moment you see a real histogram from a real process you did not instrument.

Then, in order of value for an application engineer:

• Keep the slow-connect.bt and syscall-count one-liners in a gist. The next time a service is slow and the dashboards are blank, they are the first thing to reach for, and they cost you nothing to run.
• Learn the bundled bcc tools by name - tcplife, tcpconnect, gethostlatency, profile, execsnoop, opensnoop. Ninety percent of production questions have a prebuilt tool; you rarely need to write a probe.
• If you run Kubernetes and can get security sign-off, deploy Pixie in staging and see the automatic service map. It changes what “we have observability” means, because it is not limited to what you remembered to instrument.

What eBPF does not replace: your application metrics and structured logs are still how you know a problem exists and how you alert on it. eBPF is how you find the cause once you know something is wrong. It is the layer between “the P99 alert fired” and “here is the exact syscall that is slow,” and for years that layer was either invisible or too expensive to look at on real traffic. It is neither anymore. That is the whole reason it feels like cheating: the information was always there, and now you can just ask.