You spend all morning setting up a perfect log aggregation pipeline, only to realize that when the load spikes, your logs stop reporting entirely. It is not that the application crashed. It is that your logging daemon reached its rate limit and just dropped the rest of the stream on the floor. I have seen this happen during kernel panics and high-volume I/O events, and finding that gap in your history feels like trying to reconstruct a crime scene from a shredded document.

Why journald is silently lying to you

Most people assume the systemd journal is infinite, but it has internal rate limits. If a service goes rogue and starts screaming errors at the console, journald throttles it to protect your CPU. You end up with a log that says suppressed X messages, and that is where your visibility dies. If you need to see exactly what happened in those milliseconds before a system freeze, you cannot rely on high-level logs alone.

bpftrace -e 'tracepoint:syscalls:sys_enter_write { if (pid == 1234) { printf("%s\n", str(args->buf)); } }'

• Check your rate limits in journald.conf with RateLimitIntervalSec and RateLimitBurst
• Use bpftrace to hook syscalls directly so you bypass the buffer entirely
• Watch for dropped packet counters in your secondary logging infrastructure

The trap here is thinking you have a complete audit trail just because the logs are flowing under normal conditions. When you really need the data, the buffers are usually the first thing to fail. If you have an event that happens too fast for user-space logging to catch, move down to the kernel level with eBPF. It might feel like overkill until you have to explain to your manager why the server rebooted without a single log entry to explain why.