You check your logs after a production incident, only to find a gaping hole where the system’s final gasps should be. I’ve been there. Most engineers assume journald is durable, but when the disk hits 100 percent utilization or your I/O scheduler is buried, journald defaults to dropping events to stay afloat. It’s a silent failure mode that kills your incident response window before you even open the ticket.

Why journald turns a blind eye

By default, journald attempts to rate-limit or flush messages based on system pressure. If you are logging every single request or a flood of debugging info during a crash, the buffers fill up faster than the disk can write them. You end up with empty log segments or, worse, logs that claim they saved the data when they actually discarded it to protect the system clock.

systemd-analyze dump | grep RateLimit

• RateLimitIntervalSec defines how long the system waits to check for overflow
• RateLimitBurst sets the total allowed events before dropping occurs
• Setting these to zero in journald.conf forces the system to stop discarding logs at the cost of potential process blocking

If you really need to see what is happening in the kernel without hitting the disk at all, skip the standard log pipes. Using an eBPF tracepoint on the printk path or attaching a bcc script to track process execution can show you exactly what died when the disk stopped accepting writes. This doesn’t help if your memory is corrupted, but it stops you from chasing ghosts when the storage stack is what actually failed.

If you are managing high-frequency services, bump those rate limits before you have an outage, not during one. It is a small change in /etc/systemd/journald.conf, but having that final stack trace in your buffer when the system panics is often the difference between solving a bug in an hour and staring at a blank screen for a week.