We have all been there. You run journalctl to investigate a service crash that happened three hours ago, only to find the message that the logs were cleared on reboot or simply rotated out of existence. It is infuriating. By default, most distros treat the journal as a transient, memory-only affair, which is great for keeping your root partition clean but terrible for actually debugging intermittent production weirdness.

Making your logs survive the reboot

The fix is easy, but it is one of those things you have to do manually on every new server you spin up. If you do not explicitly create the persistence directory, systemd will happily ignore your settings. And when you finally need those audit trails, you will be staring at an empty prompt. You need to ensure the directory exists and has the right ownership before telling systemd to actually keep those files.

mkdir -p /var/log/journal
chown root:systemd-journal /var/log/journal
systemctl restart systemd-journald

• Check that /etc/systemd/journald.conf has Storage set to persistent
• Verify your disk space limits, because systemd does not always prune aggressively
• Set SystemMaxUse to something reasonable so you do not fill your disk

When logs are not enough

Sometimes the kernel or a binary daemon just refuses to log what is actually happening. That is when I stop fighting with journald config files and drop an eBPF probe into the mix. If you are desperate to catch a silent file descriptor leak or a process that is dropping packets, do not wait for a log line that might never be written. Use an eBPF tool like opensnoop to see exactly which files are being opened by a specific PID in real time. It cuts through the noise and shows you the syscalls as they hit the kernel, which is the only place the truth actually lives.

Keep your logs persistent enough to catch the start of a trend, but do not hoard them until you run out of inodes. If you find yourself having to log into a server just to grep through gigabytes of text, you have already lost the battle. Automate the ingest to your centralized stack and let your monitoring tools do the heavy lifting while you focus on the real bottlenecks.