We have all been there. You get a latency spike, you check your disk I/O, and you see journald chewing through your IOPS like it is hungry for cycles. The default journald configuration is fine for a laptop, but on a busy production server, it can become your biggest bottleneck, silently wearing down your storage while you scramble to figure out why your application logs are lagging behind real-time events.

Why journald logs get out of control

The issue is not just the volume of logs, it is the rate of synchronization. Every time your application writes to standard out, journald intercepts that stream, processes it, and ensures it hits the disk. When your app gets chatty, the synchronous write operations keep the journal engine busy, creating a contention point that you really do not need if you are already streaming logs to a central collector like Vector or Fluentd.

journalctl --disk-usage && systemctl restart systemd-journald

• Cap the MaxRetentionSec to something sane like 7 days to prevent runaway disk growth.
• Set SystemMaxUse to a hard number rather than letting it take up a percentage of your partition.
• Toggle RateLimitIntervalSec to drop logs instead of thrashing your disk during a recursive loop event.

Beyond the journal configuration

If you really need to see what is happening under the hood, stop guessing and pull out bpftrace. Instead of relying on journald to tell you about blocked I/O, trace the write calls yourself. You will often find that it is not the log volume, but the frequency of syscalls that triggers the journal overhead. Watching the kernel context switch for every log line will make you rethink whether that verbose debug logging in production is actually a good idea.

Next time your disk latency looks like a mountain range, check if you have a rogue script dumping gigabytes into the journal. Usually, a quick swap to volatile storage for the logs and a smarter rate limit policy will save you more headaches than a dozen log rotation tweaks ever will.