You probably think your eBPF-based observability stack is free. It is not. I have seen more than one production outage caused by a BCC or bpftrace script that was supposedly low-overhead but actually caused lock contention on busy scheduler runqueues. When your monitoring tool hooks into common syscalls or scheduler events, it acts as a gatekeeper that the kernel must pass through. If your script logic is even slightly inefficient, you end up instrumenting your own performance death spiral.

The hidden cost of dynamic instrumentation

The kernel is not a black box, but it is a delicate ecosystem. When you attach a kprobe to a hot function, the kernel stops to execute your probe code. If that code is doing heavy calculations or maps lookups at high frequency, you are stealing CPU cycles from the threads actually doing the work. You might think the impact is negligible, but under high load, that microsecond-level penalty multiplies across thousands of calls. Always check how many events per second your filter is actually triggering before you leave a probe running on a mission-critical service.

bpftrace -e 'kprobe:vfs_read { @[kstack] = count(); }' -c "./my-high-load-app"

• Always use tracepoints over kprobes whenever the kernel provides them.
• Set an explicit event filter in your eBPF program to ignore background noise.
• Monitor your own probe execution time using bpf_ktime_get_ns.
• Never run wide-scope stack traces on production boxes during peak traffic.

The real trap is when you use aggregate-everything tools. If your script is recording a call graph for every single process in the system, you are essentially asking for a self-inflicted denial of service. Start with targeted probes. If you need to debug a slow process, anchor your script to the specific PID. I prefer writing my own small bpftrace snippets that target a single function rather than loading a massive pre-packaged collection script that collects telemetry I do not actually need. Next time you reach for a tool to find a bottleneck, ask yourself if the overhead of the monitor is already showing up in your latency metrics.