How to setup kdump in Linux to collect crash dump ?

Kdump also known as Kernel dump i.e. the contents of our system memory when our kernel was crashed.

Kdump is configured and collected when we want to debug why our kernel got panic or crashed.

Today we will look how to setup and use kdump in Linux.

Enabling Crash Dump

1. Default Kdump Configuraton in your Linux System
By default, kdump is not configured on the linux system and when you try to start its service, it will show kdump is not operational.

[root@nglinux ~]# service kdump status
Kdump is not operational
[root@nglinux ~]# 

### If the kdump service is not available then please install kexec-tools package using command:
### yum install kexec-tools

2. Enable kdump by putting crashkernel paramter in grub.conf file.
We will put crashkernel=128M in grub.conf file to allot 128MB of memory for storing the kernel crash dump memory data.

[root@nglinux ~]# cat /boot/grub/grub.conf 
default=0
timeout=30
splashimage=(hd0,0)/boot/grub/splash.xpm.gz

title NG Linux 1.0 (Stable) (2.6.32-696.el6.i686)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-696.el6.i686 root=UUID=7ec00b6f-397d-422d-8130-b22caf5b278f rhgb quiet crashkernel=128M
initrd /boot/initramfs-2.6.32-696.el6.i686.img

3. Reboot your system to take this into effect
In the Next step, we will reboot our system.

4. Finally, start the kdump system service.

[root@nglinux ~]# service kdump status
Kdump is not operational

[root@nglinux ~]# service kdump restart
Stopping kdump:                                            [  OK  ]
No kdump initial ramdisk found.                            [WARNING]
Rebuilding /boot/initrd-2.6.32-696.el6.i686kdump.img
Starting kdump:                                            [  OK  ]

[root@nglinux ~]# service kdump status
Kdump is operational
[root@nglinux ~]# 

5. Now we will generate a sample crash dump by using sysrq(System Request).

### Enable sysrq
[root@nglinux ~]# echo 1 > /proc/sys/kernel/sysrq
[root@nglinux ~]# 

### Trigger kernel panic to generate crash dump
[root@nglinux ~]# echo c > /proc/sysrq-trigger      packet_write_wait: Connection to 172.21.49.107 port 22: Broken pipe

6. Below crash dump will be generated on the path set:

[root@nglinux ~]# ls -l /var/crash/
total 632
drwxr-xr-x. 2 root root   4096 Jul 10 20:51 127.0.0.1-2018-07-10-20:51:46
-rw-------. 1 root root 581632 Jul  9 23:59 core.0.top.3835

### See the vmcore file generated under the directory.
[root@nglinux ~]# ls -l /var/crash/127.0.0.1-2018-07-10-20\:51\:46/
total 24116
-rw-------. 1 root root 24660073 Jul 10 20:51 vmcore
-rw-r--r--. 1 root root    27910 Jul 10 20:51 vmcore-dmesg.txt

Analyzing crash dump

To analyze the crash dump, we need to install the crash and kernel-debuginfo packages.
1. Install crash and debuginfo packages

[root@nglinux yum.repos.d]# yum install crash kernel-debuginfo
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirror.riverfrontnetworks.com
 * epel: ewr.edge.kernel.org
 * extras: distro.ibiblio.org
 * updates: www.gtlib.gatech.edu
base-debuginfo                                                                                                               | 2.5 kB     00:00     
base-debuginfo/primary_db                                                                                                    | 1.8 MB     00:15     
Resolving Dependencies
--> Running transaction check
---> Package crash.i686 0:7.1.0-8.el6 will be installed
---> Package kernel-debuginfo.i686 0:2.6.32-754.el6 will be installed
--> Processing Dependency: kernel-debuginfo-common-i686 = 2.6.32-754.el6 for package: kernel-debuginfo-2.6.32-754.el6.i686
--> Running transaction check
---> Package kernel-debuginfo-common-i686.i686 0:2.6.32-754.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================================================================
 Package                                         Arch                    Version                            Repository                         Size
====================================================================================================================================================
Installing:
 crash                                           i686                    7.1.0-8.el6                        base                              2.4 M
 kernel-debuginfo                                i686                    2.6.32-754.el6                     base-debuginfo                    275 M
Installing for dependencies:
 kernel-debuginfo-common-i686                    i686                    2.6.32-754.el6                     base-debuginfo                     45 M

Transaction Summary
====================================================================================================================================================
Install       3 Package(s)

Total download size: 322 M
Installed size: 1.4 G
Is this ok [y/N]: y
Downloading Packages:

Running Transaction
  Installing : kernel-debuginfo-common-i686-2.6.32-754.el6.i686                                                                                 1/3 
  Installing : kernel-debuginfo-2.6.32-754.el6.i686                                                                                             2/3 
  Installing : crash-7.1.0-8.el6.i686                                                                                                           3/3 
  Verifying  : crash-7.1.0-8.el6.i686                                                                                                           1/3 
  Verifying  : kernel-debuginfo-2.6.32-754.el6.i686                                                                                             2/3 
  Verifying  : kernel-debuginfo-common-i686-2.6.32-754.el6.i686                                                                                 3/3 

Installed:
  crash.i686 0:7.1.0-8.el6                                          kernel-debuginfo.i686 0:2.6.32-754.el6                                         

Dependency Installed:
  kernel-debuginfo-common-i686.i686 0:2.6.32-754.el6                                                                                                

Complete!
[root@nglinux yum.repos.d]# 

2. Debug the generated crash dump using crash utility and vmlinux kernel debuginfo file.

[root@nglinux 127.0.0.1-2018-07-11-20:26:50]# crash ./vmcore /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux 
crash 7.1.0-8.el6
Copyright (C) 2002-2014  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...

      KERNEL: /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux
    DUMPFILE: ./vmcore  [PARTIAL DUMP]
        CPUS: 1
        DATE: Wed Jul 11 20:26:46 2018
      UPTIME: 00:03:13
LOAD AVERAGE: 0.21, 0.23, 0.10
       TASKS: 234
    NODENAME: nglinux
     RELEASE: 2.6.32-754.el6.i686
     VERSION: #1 SMP Tue Jun 19 21:51:20 UTC 2018
     MACHINE: i686  (2693 Mhz)
      MEMORY: 1 GB
       PANIC: "SysRq : Trigger a crash"
         PID: 2417
     COMMAND: "bash"
        TASK: f4fb2000  [THREAD_INFO: f288a000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)

crash> 

Kdump options

Here we will see different options we have in /etc/kdump.conf to customize the crash dump generated.
For example:- We can save our generated crash dump to a remote location, or to a different partition.

1. Saving the generated crash dump to some other location.

### We can see below the different options available where we can save the crash dump.
### raw is another raw disk partition(i.e. unformatted)
### net keyword is used to transfer kdump on a NFS share.
### path determines the local system path.

[root@nglinux ~]# cat /etc/kdump.conf | more

    172 #raw /dev/sda5
    173 #ext4 /dev/sda3
    174 #ext4 LABEL=/boot
    175 #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
    176 #net my.server.com:/export/tmp
    177 #net user@my.server.com
    178 path /var/crash

2. Running a customized script after kdump is collected.

[root@nglinux ~]# cat /etc/kdump.conf | grep -i post
# kdump_post 
# 			  is fed to the kdump_post executable, which can be
#			- works just like the kdump_post directive, but instead
# 			  conjunction with a kdump_post binary or script that
#kdump_post /var/crash/scripts/kdump-post.sh

3. Core collector options

# core_collector  
#                       - This allows you to specify the command to copy the
#                         vmcore.  You could use the dump filtering program
#                         makedumpfile, the default one, to retrieve your core,
#                         which on some arches can drastically reduce core file
#                         size. See /usr/sbin/makedumpfile --help for a list of
#                         options. Note that the -i and -g options are not
#                         needed here, as the initrd will automatically be
#                         populated with a config file appropriate for the
#                         running kernel.
#                         For ssh dump, scp should be used instead of cp.

[root@nglinux ~]# cat /etc/kdump.conf | grep -i core_collector
core_collector makedumpfile -c --message-level 1 -d 31

In above core_collector line, makedumpfile is the available core_collector program, “-c” enables the core file compression, “-d” will omit the mentioned pages depicted by the value 31 here.
It means all free pages(16)+Zero pages(1)+Cache pages(2)+Cache private(4)+User pages(8) are omitted. we can change the value to 16 to omit only free pages, or 17 to omit free+zero pages.

4. Default action after kdump is collected

[root@nglinux ~]# cat /etc/kdump.conf | grep -i default
# to configured dump target fails, the default action will be preformed.
# Default action may be configured with the "default" directive below.
#		 	  If unset, will default to /var/crash.
#			  makedumpfile, the default one, to retrieve your core,
#			  to continue to save dump. By default kdump waits
#			  the default value is /root/.ssh/kdump_id_rsa. When
#			  identity_file.pub which by default is
# default 
#			  fails. If no default action is specified, "reboot"
#			  is assumed default.
#			  reboot: If the default action is reboot simply reboot
#			  halt:   If the default action is halt, then simply
#			  shell:  If the default action is shell, then drop to
#			- By default, kdump initrd only will be rebuilt when
#default shell

Tags: corecrashdebugdebuginfodumpkernellinuxoperationalpanicsysrqtriggerunixvmcorevmlinux