We have all been there. You attach an eBPF probe to a frequently called kernel function to hunt down a weird latency spike, and suddenly your system logs are drowning in millions of events. Most people just try to filter this in user space, but that is a mistake. By then, the damage is done and you have already wasted massive CPU cycles on context switches just to discard data.

Stop burning CPU on data you will just discard

When you are tracing hot functions like vfs_read or tcp_transmit_skb, your probe fires thousands of times per second. If you aren’t doing the filtering inside the kernel using BPF helpers, you are effectively performing a self-inflicted denial of service attack on your own monitoring pipeline. I once saw a junior engineer bring down a production web server by attaching a naive kprobe to a scheduler function. It was a good lesson, but one you should avoid learning the hard way.

bpftrace -e 'kprobe:vfs_read /pid == 1234/ { printf("Read size: %d\n", arg2); }'

• Use predicate filtering directly in your eBPF program to ignore processes you don’t care about.
• Apply bpf_map objects to aggregate metrics in the kernel instead of streaming every individual event to user space.
• Check your tracepoints before resorting to kprobes, as they are generally more stable and often have built-in filtering.

If you find yourself needing to filter by more than a simple PID, look into using per-CPU buffers to keep your tracing overhead predictable. It is better to write a few extra lines of C or clean bpftrace logic than to deal with the performance impact of a saturated ring buffer. Next time you feel tempted to log everything and sort it out later, remember that the kernel doesn’t care about your post-processing requirements.