You spend all morning trying to track down why an application is throwing silent I/O errors. You assume it is a permissions issue or maybe a bad mount, so you pivot to strace, but that is a mistake. Running strace on a busy production process is a recipe for a frozen production app, and usually, the noise drowns out the very syscalls you need to see. Most of the time, the application is failing way before it even touches the kernel interface.

Why your logs are lying to you

The trap most of us fall into is staring at journald, hoping for a clean error log. If a library is swallowing exceptions or if the runtime is silently failing during a system call wrapper, you will never see it in the system logs. You need to verify what the kernel actually sees, not what the application claims it sent. That is where eBPF shines, specifically by hooking into the syscall table without stopping the world.

bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s (%d): %s\n", comm, pid, str(args->filename)); }'

• Use bpftrace to attach to raw syscall entry points instead of relying on slow, user-space debugging tools
• Focus on the filename arguments to see if the process is hitting paths you didn’t expect
• Keep the filter narrow to prevent overloading the BPF buffer on a high-throughput box

Stop the performance tax

If you are worried about the overhead of tracing, keep in mind that bpftrace runs inside the kernel via a JIT-compiled bytecode. It is significantly safer for production than traditional debuggers. If you see a syscall entry without a matching return code, you have found exactly where your code is hanging or hitting an invalid memory segment. Put this tool in your back pocket for the next time you see a service heartbeat die with no trace of an error message.