Kdump also known as Kernel dump i.e. the contents of our system memory when our kernel was crashed.
Kdump is configured and collected when we want to debug why our kernel got panic or crashed.
Today we will look how to setup and use kdump in Linux.
Enabling Crash Dump
1. Default Kdump Configuraton in your Linux System
By default, kdump is not configured on the linux system and when you try to start its service, it will show kdump is not operational.
[root@nglinux ~]# service kdump status Kdump is not operational [root@nglinux ~]# ### If the kdump service is not available then please install kexec-tools package using command: ### yum install kexec-tools
2. Enable kdump by putting crashkernel paramter in grub.conf file.
We will put crashkernel=128M in grub.conf file to allot 128MB of memory for storing the kernel crash dump memory data.
[root@nglinux ~]# cat /boot/grub/grub.conf default=0 timeout=30 splashimage=(hd0,0)/boot/grub/splash.xpm.gz title NG Linux 1.0 (Stable) (2.6.32-696.el6.i686) root (hd0,0) kernel /boot/vmlinuz-2.6.32-696.el6.i686 root=UUID=7ec00b6f-397d-422d-8130-b22caf5b278f rhgb quiet crashkernel=128M initrd /boot/initramfs-2.6.32-696.el6.i686.img
3. Reboot your system to take this into effect
In the Next step, we will reboot our system.
4. Finally, start the kdump system service.
[root@nglinux ~]# service kdump status Kdump is not operational [root@nglinux ~]# service kdump restart Stopping kdump: [ OK ] No kdump initial ramdisk found. [WARNING] Rebuilding /boot/initrd-2.6.32-696.el6.i686kdump.img Starting kdump: [ OK ] [root@nglinux ~]# service kdump status Kdump is operational [root@nglinux ~]#
5. Now we will generate a sample crash dump by using sysrq(System Request).
### Enable sysrq [root@nglinux ~]# echo 1 > /proc/sys/kernel/sysrq [root@nglinux ~]# ### Trigger kernel panic to generate crash dump [root@nglinux ~]# echo c > /proc/sysrq-trigger packet_write_wait: Connection to 172.21.49.107 port 22: Broken pipe
6. Below crash dump will be generated on the path set:
[root@nglinux ~]# ls -l /var/crash/ total 632 drwxr-xr-x. 2 root root 4096 Jul 10 20:51 127.0.0.1-2018-07-10-20:51:46 -rw-------. 1 root root 581632 Jul 9 23:59 core.0.top.3835 ### See the vmcore file generated under the directory. [root@nglinux ~]# ls -l /var/crash/127.0.0.1-2018-07-10-20\:51\:46/ total 24116 -rw-------. 1 root root 24660073 Jul 10 20:51 vmcore -rw-r--r--. 1 root root 27910 Jul 10 20:51 vmcore-dmesg.txt
Analyzing crash dump
To analyze the crash dump, we need to install the crash and kernel-debuginfo packages.
1. Install crash and debuginfo packages
[root@nglinux yum.repos.d]# yum install crash kernel-debuginfo Failed to set locale, defaulting to C Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Install Process Loading mirror speeds from cached hostfile * base: mirror.riverfrontnetworks.com * epel: ewr.edge.kernel.org * extras: distro.ibiblio.org * updates: www.gtlib.gatech.edu base-debuginfo | 2.5 kB 00:00 base-debuginfo/primary_db | 1.8 MB 00:15 Resolving Dependencies --> Running transaction check ---> Package crash.i686 0:7.1.0-8.el6 will be installed ---> Package kernel-debuginfo.i686 0:2.6.32-754.el6 will be installed --> Processing Dependency: kernel-debuginfo-common-i686 = 2.6.32-754.el6 for package: kernel-debuginfo-2.6.32-754.el6.i686 --> Running transaction check ---> Package kernel-debuginfo-common-i686.i686 0:2.6.32-754.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================== Installing: crash i686 7.1.0-8.el6 base 2.4 M kernel-debuginfo i686 2.6.32-754.el6 base-debuginfo 275 M Installing for dependencies: kernel-debuginfo-common-i686 i686 2.6.32-754.el6 base-debuginfo 45 M Transaction Summary ==================================================================================================================================================== Install 3 Package(s) Total download size: 322 M Installed size: 1.4 G Is this ok [y/N]: y Downloading Packages: Running Transaction Installing : kernel-debuginfo-common-i686-2.6.32-754.el6.i686 1/3 Installing : kernel-debuginfo-2.6.32-754.el6.i686 2/3 Installing : crash-7.1.0-8.el6.i686 3/3 Verifying : crash-7.1.0-8.el6.i686 1/3 Verifying : kernel-debuginfo-2.6.32-754.el6.i686 2/3 Verifying : kernel-debuginfo-common-i686-2.6.32-754.el6.i686 3/3 Installed: crash.i686 0:7.1.0-8.el6 kernel-debuginfo.i686 0:2.6.32-754.el6 Dependency Installed: kernel-debuginfo-common-i686.i686 0:2.6.32-754.el6 Complete! [root@nglinux yum.repos.d]#
2. Debug the generated crash dump using crash utility and vmlinux kernel debuginfo file.
[root@nglinux 127.0.0.1-2018-07-11-20:26:50]# crash ./vmcore /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux
crash 7.1.0-8.el6
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux
DUMPFILE: ./vmcore [PARTIAL DUMP]
CPUS: 1
DATE: Wed Jul 11 20:26:46 2018
UPTIME: 00:03:13
LOAD AVERAGE: 0.21, 0.23, 0.10
TASKS: 234
NODENAME: nglinux
RELEASE: 2.6.32-754.el6.i686
VERSION: #1 SMP Tue Jun 19 21:51:20 UTC 2018
MACHINE: i686 (2693 Mhz)
MEMORY: 1 GB
PANIC: "SysRq : Trigger a crash"
PID: 2417
COMMAND: "bash"
TASK: f4fb2000 [THREAD_INFO: f288a000]
CPU: 0
STATE: TASK_RUNNING (SYSRQ)
crash>
Kdump options
Here we will see different options we have in /etc/kdump.conf to customize the crash dump generated.
For example:- We can save our generated crash dump to a remote location, or to a different partition.
1. Saving the generated crash dump to some other location.
### We can see below the different options available where we can save the crash dump.
### raw is another raw disk partition(i.e. unformatted)
### net keyword is used to transfer kdump on a NFS share.
### path determines the local system path.
[root@nglinux ~]# cat /etc/kdump.conf | more
172 #raw /dev/sda5
173 #ext4 /dev/sda3
174 #ext4 LABEL=/boot
175 #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
176 #net my.server.com:/export/tmp
177 #net user@my.server.com
178 path /var/crash
2. Running a customized script after kdump is collected.
[root@nglinux ~]# cat /etc/kdump.conf | grep -i post # kdump_post # is fed to the kdump_post executable, which can be # - works just like the kdump_post directive, but instead # conjunction with a kdump_post binary or script that #kdump_post /var/crash/scripts/kdump-post.sh
3. Core collector options
# core_collector # - This allows you to specify the command to copy the # vmcore. You could use the dump filtering program # makedumpfile, the default one, to retrieve your core, # which on some arches can drastically reduce core file # size. See /usr/sbin/makedumpfile --help for a list of # options. Note that the -i and -g options are not # needed here, as the initrd will automatically be # populated with a config file appropriate for the # running kernel. # For ssh dump, scp should be used instead of cp. [root@nglinux ~]# cat /etc/kdump.conf | grep -i core_collector core_collector makedumpfile -c --message-level 1 -d 31
In above core_collector line, makedumpfile is the available core_collector program, “-c” enables the core file compression, “-d” will omit the mentioned pages depicted by the value 31 here.
It means all free pages(16)+Zero pages(1)+Cache pages(2)+Cache private(4)+User pages(8) are omitted. we can change the value to 16 to omit only free pages, or 17 to omit free+zero pages.
4. Default action after kdump is collected
[root@nglinux ~]# cat /etc/kdump.conf | grep -i default # to configured dump target fails, the default action will be preformed. # Default action may be configured with the "default" directive below. # If unset, will default to /var/crash. # makedumpfile, the default one, to retrieve your core, # to continue to save dump. By default kdump waits # the default value is /root/.ssh/kdump_id_rsa. When # identity_file.pub which by default is # default # fails. If no default action is specified, "reboot" # is assumed default. # reboot: If the default action is reboot simply reboot # halt: If the default action is halt, then simply # shell: If the default action is shell, then drop to # - By default, kdump initrd only will be rebuilt when #default shell