Why journalctl logs disappear after reboot and how to make it permanent ?

By default, Journald logs gets stored in /run/log/journal directory and disappear once the system gets rebooted.

I. Default Journald configuration

[root@localhost ~]# date
Wed Mar 13 10:50:29 PDT 2019

[root@localhost ~]# uptime
 10:51:17 up  1:23,  1 user,  load average: 0.00, 0.01, 0.05

### We can see the logs were started at
### 09:27 PDT i.e. just after system boot
### All old logs are removed.
[root@localhost ~]# journalctl | more
-- Logs begin at Wed 2019-03-13 09:27:20 PDT, end at Wed 2019-03-13 10:50:01 PDT
. --
Mar 13 09:27:20 localhost.localdomain systemd-journal[86]: Runtime journal is us
ing 8.0M (max allowed 89.5M, trying to leave 134.2M free of 887.0M available → cu
rrent limit 89.5M).
Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpuset
Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpu
Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpuacct
Mar 13 09:27:20 localhost.localdomain kernel: Linux version 3.10.0-957.el7.x86_6
4 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.
5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018

 


II. Making Journald configuration to store logs permanently

To store Journald logs permanently, we need to change its configuration.

Here are the steps.

# PRE-CHECKS

### Get before activity logs.
[root@localhost ~]# uptime
 10:51:17 up  1:23,  1 user,  load average: 0.00, 0.01, 0.05
[root@localhost ~]# date
Wed Mar 13 10:53:05 PDT 2019

 

# CONFIGURATION


a. Create log directory if not exists

[root@localhost ~]# mkdir /var/log/journal

 

b. Create prefix which enables the log directory location
/var/log/journal which stores the journal data.
systemd-tmpfiles command sets up correct permissions on the /var/log/journal directory

[root@localhost ~]# systemd-tmpfiles --create --prefix /var/log/journal

 

c. Set maximum system disk space of 100MB.
SystemMaxUse variable must be defined otherwise 10% of overall filesystem
where /var/log/journal directory stored is the limit set up for journal.

[root@localhost ~]# echo "SystemMaxUse=100M" >> /etc/systemd/journald.conf 

[root@localhost ~]# grep -i SystemMaxUse /etc/systemd/journald.conf 
#SystemMaxUse=
SystemMaxUse=100M

 

d. Restart the journald.service

[root@localhost ~]# systemctl restart systemd-journald.service 
[root@localhost ~]# 

 

# POST-CHECKS

### Checkout the newly created data file.
[root@localhost ~]# file /var/log/journal/974de63103a847d3a6879c9a69b0c1b8/system.journal 
/var/log/journal/974de63103a847d3a6879c9a69b0c1b8/system.journal: data
[root@localhost ~]# 

### Restart the system and verify if old logs are there.
### After modifying file /etc/systemd/journald.conf file, it is advisable to reboot
[root@localhost ~]# init 6

### Now check logs again after system boot
### The logs still exists from the same old date till it reaches 100M size.
[root@localhost ~]# uptime
 11:10:26 up 11 min,  4 users,  load average: 0.04, 0.26, 0.29
[root@localhost ~]# date
Wed Mar 13 11:10:28 PDT 2019

[root@localhost ~]# journalctl | more
-- Logs begin at Wed 2019-03-13 09:27:20 PDT, end at Wed 2019-03-13 11:10:18 PDT
. --
Mar 13 09:27:20 localhost.localdomain systemd-journal[86]: Runtime journal is us
ing 8.0M (max allowed 89.5M, trying to leave 134.2M free of 887.0M available → cu
rrent limit 89.5M).
Mar 13 09:27:20 localhost.localdomain kernel: Initializing cgroup subsys cpuset

 

From Systemd v219 and RHEL 7.2 onwards, /var/log/journal mount point requires ACL support.

Quick way to check ACLs:

[root@localhost ~]# cat /boot/config-3.10.0-957.el7.x86_64  | grep -i acl
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_FS_POSIX_ACL=y
CONFIG_GENERIC_ACL=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_CEPH_FS_POSIX_ACL=y
CONFIG_CIFS_ACL=y
[root@localhost ~]#

 

Leave a Reply

Your email address will not be published.