You’ve been there: a service is acting up, users are complaining about latency or intermittent failures, but journalctl -u your-service shows nothing but happy INFO messages. No errors, no warnings, just green lights. And you’re tearing your hair out because you know something is wrong. Your application’s internal logging isn’t catching it, and that usually means the problem isn’t within the application’s logic, but in its interaction with the OS itself. This bit me hard once when a supposedly ‘idle’ service was quietly forking hundreds of short-lived processes, chewing up CPU just enough to cause grief for everything else. Journald knew nothing.

Journald: Your trusted friend, until it isn’t

Look, I’m not knocking journald. It’s a fantastic piece of engineering, centralizing logs, structuring them, giving us great filtering and persistence. For 90% of your operational tasks, journalctl is your go-to. You can trace by unit, by PID, by time range, even structured fields if your application plays nice. But journald sees what your application chooses to emit. It’s application-centric. It doesn’t inherently see every execve, every openat, every socket call your process makes. And sometimes, those low-level syscalls are where the silent killers hide.

Getting dirty with eBPF: What’s really going on down there?

This is where eBPF steps in. Forget user-space strace for a second – that’s synchronous, injects overhead, and can completely change the timing of what you’re trying to observe. eBPF gives you programmable hooks right inside the kernel. You can attach tiny programs to almost any kernel event: syscall entries/exits, network events, scheduler events, disk I/O, you name it. It’s incredibly powerful, and crucially, it’s designed for low overhead, making it safe for production tracing when used thoughtfully. You’re getting direct insight into how the kernel processes your application’s requests, without modifying a single line of your application code.

execsnoop

• Your application is performing unexpected file I/O or opening too many descriptors, leading to latency or file handle exhaustion. opensnoop or filesnoop can trace these.
• A service is silently spawning child processes that chew up resources or fail quickly, never hitting your application’s logs. execsnoop nails these.
• Intermittent network connection issues that aren’t showing up in netstat or application errors. Tools like tcpconnect or tcplife can show individual TCP session lifecycles.
• Mysterious CPU spikes in a seemingly idle process, indicating hidden kernel activity or poor scheduling. profile or runqlat can pinpoint scheduler contention.
• When you need to understand why a specific syscall returned an error or took a long time, not just that it happened.

eBPF isn’t a silver bullet, and there’s a learning curve, especially if you dive into writing bpftrace scripts yourself. But understanding that you have this level of visibility into your kernel’s operations can save you days of frustration when application logs hit a dead end. When journald is giving you the silent treatment about a performance issue, remember there’s a whole world of syscalls happening underneath, and eBPF is your best friend for peeking into that abyss. It’s often the difference between a quick fix and an endless guessing game that leaves you watching the clock.