Why journalctl logs disappear after reboot and how to make it permanent ?
By default, Journald logs gets stored in /run/log/journal directory and disappear once the system gets rebooted.
I. Default Journald configuration
[root@localhost ~]# date Wed Mar 13 10:50:29 PDT 2019 [root@localhost ~]# uptime 10:51:17 up 1:23, 1 user, load average: 0.00, 0.01, 0.05 ### We can see the logs were started at ### 09:27 PDT i.e. just after system boot ### All old logs are removed. [root@localhost ~]# journalctl | more -- Logs begin at Wed 2019-03-13 09:27:20 PDT, end at Wed 2019-03-13 10:50:01 PDT . -- Mar 13 09:27:20 localhost.localdomain systemd-journal[86]: Runtime journal is us ing 8.0M (max allowed 89.5M, trying to leave 134.2M free of 887.0M available → cu rrent limit 89.5M). Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpuset Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpu Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpuacct Mar 13 09:27:20 localhost.localdomain kernel: Linux version 3.10.0-957.el7.x86_6 4 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8. 5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018
II. Making Journald configuration to store logs permanently
To store Journald logs permanently, we need to change its configuration.
Here are the steps.
# PRE-CHECKS
### Get before activity logs. [root@localhost ~]# uptime 10:51:17 up 1:23, 1 user, load average: 0.00, 0.01, 0.05 [root@localhost ~]# date Wed Mar 13 10:53:05 PDT 2019
# CONFIGURATION
a. Create log directory if not exists
[root@localhost ~]# mkdir /var/log/journal
b. Create prefix which enables the log directory location
/var/log/journal which stores the journal data.
systemd-tmpfiles command sets up correct permissions on the /var/log/journal directory
[root@localhost ~]# systemd-tmpfiles --create --prefix /var/log/journal
c. Set maximum system disk space of 100MB.
SystemMaxUse variable must be defined otherwise 10% of overall filesystem
where /var/log/journal directory stored is the limit set up for journal.
[root@localhost ~]# echo "SystemMaxUse=100M" >> /etc/systemd/journald.conf [root@localhost ~]# grep -i SystemMaxUse /etc/systemd/journald.conf #SystemMaxUse= SystemMaxUse=100M
d. Restart the journald.service
[root@localhost ~]# systemctl restart systemd-journald.service [root@localhost ~]#
# POST-CHECKS
### Checkout the newly created data file. [root@localhost ~]# file /var/log/journal/974de63103a847d3a6879c9a69b0c1b8/system.journal /var/log/journal/974de63103a847d3a6879c9a69b0c1b8/system.journal: data [root@localhost ~]# ### Restart the system and verify if old logs are there. ### After modifying file /etc/systemd/journald.conf file, it is advisable to reboot [root@localhost ~]# init 6 ### Now check logs again after system boot ### The logs still exists from the same old date till it reaches 100M size. [root@localhost ~]# uptime 11:10:26 up 11 min, 4 users, load average: 0.04, 0.26, 0.29 [root@localhost ~]# date Wed Mar 13 11:10:28 PDT 2019 [root@localhost ~]# journalctl | more -- Logs begin at Wed 2019-03-13 09:27:20 PDT, end at Wed 2019-03-13 11:10:18 PDT . -- Mar 13 09:27:20 localhost.localdomain systemd-journal[86]: Runtime journal is us ing 8.0M (max allowed 89.5M, trying to leave 134.2M free of 887.0M available → cu rrent limit 89.5M). Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpuset
From Systemd v219 and RHEL 7.2 onwards, /var/log/journal mount point requires ACL support.
Quick way to check ACLs:
[root@localhost ~]# cat /boot/config-3.10.0-957.el7.x86_64 | grep -i acl CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_XFS_POSIX_ACL=y CONFIG_BTRFS_FS_POSIX_ACL=y CONFIG_FS_POSIX_ACL=y CONFIG_GENERIC_ACL=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_NFS_V3_ACL=y CONFIG_NFSD_V2_ACL=y CONFIG_NFSD_V3_ACL=y CONFIG_NFS_ACL_SUPPORT=m CONFIG_CEPH_FS_POSIX_ACL=y CONFIG_CIFS_ACL=y [root@localhost ~]#