Stop journald from silently dropping logs when your buffer overflows

Technical Briefing | 7/26/2026

You spend all morning chasing a race condition that only fires once every three days. When you finally hit it in production, you head straight for journalctl to piece together the events, but the logs simply aren’t there. It is not that they were rotated; it is that the system rate-limited you and threw the overflow into the ether. Most default configurations set a permissive rate limit, but under heavy load, they will drop lines without telling you.

The trap of default ratelimiting

The issue is that journald has a built-in throttle designed to prevent runaway processes from eating your disk space. It defaults to 1000 messages every 30 seconds. That sounds like a lot until your application decides to spew stack traces during a socket timeout. If you are not monitoring the journal’s own health, you will lose the very data you need during an incident. I have seen developers stare at code for hours while the answers were literally blocked by a daemon trying to save its own skin.

journalctl --verify --quiet

• Check /etc/systemd/journald.conf for RateLimitIntervalSec and RateLimitBurst settings
• Set RateLimitIntervalSec to 5s if you need high-resolution logging during bursts
• Monitor the journald service status for dropped message warnings
• Use bpftrace to hook into printk if you suspect the kernel is dropping logs before they even reach user space

If you really need to capture high-frequency events without kernel interference, eBPF is your only honest option. You can attach a kprobe to the underlying write syscalls or specific kernel functions to bypass the logging daemon’s bottlenecks. It is more work than just piping to a file, but when you are trying to debug a micro-latency issue or a buffer overrun, you stop relying on the standard stack and start asking the kernel exactly what it saw.

The next time your logs go dark during a crash, do not assume your app stopped writing. Check if you have hit the wall that the system built for you. A bit of tuning in your configuration file now is better than a blind spot when the pager goes off.