RHEL 7: How to use journalctl to examine system logs and its various options ?
Today we will look how to use journalctl command to examine system logs in RHEL 7 or 8.
And various options which are used frequently.
1. Journalctl command displays all logs in paginated view.
[root@ngelinux01 ~]# journalctl -- Logs begin at Fri 2021-06-11 07:25:25 BST, end at Fri 2021-06-11 07:27:06 BST. -- Jun 11 07:25:25 host-0-0.linuxzoo.net systemd-journal[86]: Runtime journal is using 4.0M (max allowed 24.4M, trying to leave 36.6M free of 240. Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Initializing cgroup subsys cpuset Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Initializing cgroup subsys cpu Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Initializing cgroup subsys cpuacct Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Linux version 3.10.0-514.10.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 201 Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=/dev/mapper/centos_lvm-root ro Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: e820: BIOS-provided physical RAM map: Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
2. To see new logs first, we can reverse the output.
[root@ngelinux01 ~]# journalctl -- Logs begin at Fri 2021-06-11 07:25:25 BST, end at Fri 2021-06-11 07:27:06 BST. -- Jun 11 07:25:25 host-0-0.linuxzoo.net systemd-journal[86]: Runtime journal is using 4.0M (max allowed 24.4M, trying to leave 36.6M free of 240. Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Initializing cgroup subsys cpuset Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Initializing cgroup subsys cpu Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Initializing cgroup subsys cpuacct Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Linux version 3.10.0-514.10.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 201 Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-514.10.2.el7.x86_64 root=/dev/mapper/centos_lvm-root ro Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: e820: BIOS-provided physical RAM map: Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
3. To display latest 8 lines of log only.
[root@ngelinux01 ~]# journalctl -r -n 8 -- Logs begin at Fri 2021-06-11 07:25:25 BST, end at Fri 2021-06-11 07:27:06 BST. -- Jun 11 07:27:06 ngelinux01.linuxzoo.net systemd[1]: Started Session 1 of user root. Jun 11 07:27:06 ngelinux01.linuxzoo.net systemd[1]: Starting user-0.slice. Jun 11 07:27:06 ngelinux01.linuxzoo.net systemd[1]: Created slice user-0.slice. Jun 11 07:27:06 ngelinux01.linuxzoo.net sshd[1661]: Accepted password for root from 10.200.0.1 port 55248 ssh2 Jun 11 07:26:42 ngelinux01.linuxzoo.net realmd[1600]: stopping service Jun 11 07:26:42 ngelinux01.linuxzoo.net realmd[1600]: quitting realmd service after timeout Jun 11 07:26:40 ngelinux01.linuxzoo.net sshd[1661]: dispatch_protocol_error: type 98 seq 5 [preauth] Jun 11 07:26:11 ngelinux01.linuxzoo.net fprintd[1598]: ** Message: No devices in use, exit [root@ngelinux01 ~]#
4. To get specific log of warning type.
root@ngelinux01 ~]# journalctl -p warning -- Logs begin at Fri 2021-06-11 07:25:25 BST, end at Fri 2021-06-11 07:48:06 BST. -- Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: ACPI: RSDP 00000000000f69c0 00014 (v00 BOCHS ) Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: ACPI: RSDT 000000001ffe18dc 00030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: ACPI: FACP 000000001ffe17b8 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: ACPI: DSDT 000000001ffe0040 01778 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001) Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: ACPI: FACS 000000001ffe0000 00040 Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: ACPI: APIC 000000001ffe182c 00078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: ACPI: HPET 000000001ffe18a4 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001) Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: kexec: crashkernel=auto resulted in zero bytes of reserved memory. Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: Zone ranges: Jun 11 07:25:25 host-0-0.linuxzoo.net kernel: DMA [mem 0x00001000-0x00ffffff]
5. To see all options of journalctl command.
[root@ngelinux01 ~]# journalctl --help
journalctl [OPTIONS...] [MATCHES...]
Query the journal.
Flags:
--system Show the system journal
--user Show the user journal for the current user
-M --machine=CONTAINER Operate on local container
-S --since=DATE Show entries not older than the specified date
-U --until=DATE Show entries not newer than the specified date
-c --cursor=CURSOR Show entries starting at the specified cursor
--after-cursor=CURSOR Show entries after the specified cursor
6. Journal Logs saves only current boot messages.
To save the messages from past boots also, we can make following changes.
#### First step is to create below directory # mkdir -p /var/log/journal #### Make "storage=persistent" from "storage=auto" like below. [root@ngelinux01 ~]# more /etc/systemd/journald.conf # This file is part of systemd. # # systemd is free software; you can redistribute it and/or modify it # under the terms of the GNU Lesser General Public License as published by # the Free Software Foundation; either version 2.1 of the License, or # (at your option) any later version. # # Entries in this file show the compile time defaults. # You can change settings by editing this file. # Defaults can be restored by simply deleting this file. # # See journald.conf(5) for details. [Journal] Storage=persistent #Compress=yes ##### Now restart the journalctl service [root@ngelinux01 ~]# systemctl restart systemd-journald.service [root@ngelinux01 ~]#
7. View logs between specific time frames.
[root@ngelinux01 ~]# journalctl --since "2021-01-10" --until "2021-01-10 01:00" -- Logs begin at Fri 2021-06-11 07:25:25 BST, end at Fri 2021-06-11 08:20:02 BST. -- [root@ngelinux01 ~]# [root@ngelinux01 ~]# journalctl --since "2021-06-11 07:45" --until "2021-06-11 08:00:00" -- Logs begin at Fri 2021-06-11 07:25:25 BST, end at Fri 2021-06-11 08:20:02 BST. -- Jun 11 07:46:40 ngelinux01.linuxzoo.net sshd[1892]: Bad packet length 2790760703. [preauth] Jun 11 07:46:40 ngelinux01.linuxzoo.net sshd[1892]: Disconnecting: Packet corrupt [preauth] Jun 11 07:46:58 ngelinux01.linuxzoo.net sshd[1894]: dispatch_protocol_error: type 98 seq 5 [preauth] Jun 11 07:47:33 ngelinux01.linuxzoo.net sshd[1894]: error: buffer_get_ret: trying to get more bytes 4 than in buffer 0 [preauth] Jun 11 07:47:33 ngelinux01.linuxzoo.net sshd[1894]: error: buffer_get_string_ret: cannot extract length [preauth] Jun 11 07:47:33 ngelinux01.linuxzoo.net sshd[1894]: fatal: buffer_get_string: buffer error [preauth] Jun 11 07:47:39 ngelinux01.linuxzoo.net sshd[1904]: dispatch_protocol_error: type 98 seq 5 [preauth] Jun 11 07:47:45 ngelinux01.linuxzoo.net sshd[1904]: Accepted password for root from 10.200.0.1 port 56718 ssh2 Jun 11 07:47:45 ngelinux01.linuxzoo.net systemd[1]: Started Session 2 of user root. Jun 11 07:47:45 ngelinux01.linuxzoo.net sshd[1904]: pam_unix(sshd:session): session opened for user root by (uid=0) Jun 11 07:47:45 ngelinux01.linuxzoo.net systemd-logind[604]: New session 2 of user root. Jun 11 07:47:45 ngelinux01.linuxzoo.net systemd[1]: Starting Session 2 of user root. Jun 11 07:47:46 ngelinux01.linuxzoo.net dbus[619]: [system] Activating service name='org.freedesktop.problems' (using servicehelper) Jun 11 07:47:46 ngelinux01.linuxzoo.net dbus-daemon[619]: dbus[619]: [system] Activating service name='org.freedesktop.problems' (using service Jun 11 07:47:46 ngelinux01.linuxzoo.net dbus[619]: [system] Successfully activated service 'org.freedesktop.problems' Jun 11 07:47:46 ngelinux01.linuxzoo.net dbus-daemon[619]: dbus[619]: [system] Successfully activated service 'org.freedesktop.problems' Jun 11 07:48:06 ngelinux01.linuxzoo.net sshd[1661]: pam_unix(sshd:session): session closed for user root Jun 11 07:48:06 ngelinux01.linuxzoo.net systemd-logind[604]: Removed session 1. lines 1-19/19 (END)
8. To see logs of a specific unit of specific time.
[root@ngelinux01 ~]# journalctl -u cups.service --since "2021-06-11 12:30" -- No entries --
9. To check logs of a specific PID.
[root@ngelinux01 ~]# journalctl _PID=1 -- Logs begin at Fri 2021-06-11 07:25:25 BST, end at Fri 2021-06-11 08:20:02 BST. -- Jun 11 07:25:25 host-0-0.linuxzoo.net systemd[1]: Started dracut cmdline hook. Jun 11 07:25:25 host-0-0.linuxzoo.net systemd[1]: Starting dracut pre-udev hook... Jun 11 07:25:25 host-0-0.linuxzoo.net systemd[1]: Started dracut pre-udev hook. Jun 11 07:25:25 host-0-0.linuxzoo.net systemd[1]: Starting udev Kernel Device Manager... Jun 11 07:25:25 host-0-0.linuxzoo.net systemd[1]: Started udev Kernel Device Manager. Similarly we have various options, however the options mentioned above are most widely used and help us to serve the day to day purpose.