Most of us treat journald as a black box that just works until our I/O wait spikes or we get that dreaded out of space error on a partition we thought was mostly empty. I have seen more production servers choked by excessive logging than by actual application traffic. If you haven’t looked at your storage pressure settings in /etc/systemd/journald.conf, you are essentially leaving a ticking time bomb in your logs directory.

Why default persistence kills your write endurance

The defaults in systemd are tuned for desktop users who occasionally need to grep through errors from last week. In a high-traffic server environment, letting journald auto-scale its storage leads to gigabytes of binary logs that your filesystem doesn’t really want to keep. You need to enforce hard caps on space and time, and stop trusting the system to decide what is important for you.

journalctl --vacuum-size=1G --vacuum-time=7d

• Set SystemMaxUse to something sane like 500M or 1G
• Change Storage to volatile if you want to push logs to a remote collector anyway
• Set MaxRetentionSec to prevent log bloat beyond your legal compliance requirements

When logs hide the real truth

But sometimes you cannot just prune logs. When an application is throwing errors that journald is busy logging while your disk queue is saturated, you are looking at the symptoms, not the cause. Instead of cranking up log levels and filling the disk faster, pull an eBPF trace to see what the process is actually doing with its file descriptors before the log line even hits the buffer. Next time you hit a wall, skip the log rotation and run execsnoop to see if the process is spawning child shells it shouldn’t be.

If you are worried about the performance hit of monitoring, remember that the kernel handles eBPF programs far more efficiently than it handles a busy syslog daemon. Keep your logs lean, keep your tracing targeted, and stop letting the system daemon decide how your hardware spends its energy.