RHEL or Centos Linux: Kdump generated manually however not able to find it on reboot ?

Today in this article we will see an interesting issue where you crash a server manually however the crash dump is not available on server post reboot.

In my scenario i have configured the kdump on a separate path “/var/crash”. However i am not able to find it even after crash.

Lets see the issue details and its solution.

I. Issue Details
Kdump configured and operational.
However when i crashed server, i can’t see dump files in /var/crash and /var became 100% full.

[root@ngelinux01 ~]# service kdump status
Kdump is operational

[root@ngelinux01 ~]# ls -l /var/crash/
[root@ngelinux01 ~]# 

 

II. Investigation
Now we will investigate the issue how come /var became 100% with no data on /var/crash.

a. Check what is consuming space on /var.
[root@ngelinux01 ~]# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-varlv
                      7.9G  7.9G     0 100% /var
[root@ngelinux01 ~]#

### As a precaution, just checking if we have enough space in VG.
[root@ngelinux01 ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  appvg    1   4   0 wz--n- 410.10g 390.10g
  rootvg   1  13   0 wz--n- 136.19g  54.19g

[root@ngelinux01 ~]# cd /var
[root@ngelinux01 var]# du -sh *
4.0K    account
0       adm
1.3G    cache
12K     cpq
16K     crash
48K     db
12K     empty
4.0K    games
4.0K    gopher
276M    lib
4.0K    local
16K     lock
253M    log
89M     logII
20K     lost+found
0       mail
4.0K    nis
230M    nmon
162M    opt
4.0K    preserve
116K    run
161M    spool
101M    tmp
4.0K    yp

[root@ngelinux01 var]# du -sh /var
2.6G    /var

Interesting the total used space is around 2.6GB however no free space.
Interesting, lets see any deleted/open files using the space.

### First check any deleted files
[root@ngelinux01 var]# lsof | grep -i deleted
JOBMAN     2953    root   17r      REG             253,11 14249984       8484 /usr/tws/TWS/SymphOld1636379233 (deleted)
JOBMAN     6550    root   17r      REG             253,11 13806592       8485 /usr/tws/TWS/SymphOld1636292817 (deleted)
nge_gathe  6607    root    1u      REG              253,7        0         46 /tmp/sf01.ib4 (deleted)
nge_gathe  6975    root    1u      REG              253,7        0         40 /tmp/sf3a.958 (deleted)
check_clo  7455    root    0r      REG              253,0        1      83165 /usr/local/cloudamize/bin/system_key (deleted)
check_clo  7472    root    0r      REG              253,0        1      83167 /usr/local/cloudamize/bin/system_key (deleted)
check_clo  7473    root    0r      REG              253,0        1      83166 /usr/local/cloudamize/bin/system_key (deleted)
qualys-cl  7532    root    8u      REG              253,0    12288     155844 /usr/local/qualys/cloud-agent/patchmanagement/PatchStatus.db (deleted)
qualys-cl  7532    root    9u      REG              253,0    32768     155845 /usr/local/qualys/cloud-agent/patchmanagement/PatchManagement.db (deleted)
sleep     23588    root    0r      REG              253,0        1      83166 /usr/local/cloudamize/bin/system_key (deleted)
JOBMAN    30545    root   17r      REG             253,11 14249984       8484 /usr/tws/TWS/SymphOld1636379233 (deleted)
nge_gathe 30599    root    1u      REG              253,7        0         34 /tmp/sf25.cuq (deleted)

### Any open files using space
[root@ngelinux01 var]# lsof /var
COMMAND     PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
rpm         676    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm         676    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm         676    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm         676    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm         676    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm         676    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm        1141    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm        1141    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm        1141    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13662    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13662    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13662    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13662    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13662    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       13833    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       13833    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13833    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13833    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13833    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13833    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       13880    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       13880    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13880    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13880    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13880    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13880    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       13960    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       13960    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       13960    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       13960    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       13960    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       13960    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15040    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       15040    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       15040    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       15040    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       15040    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       15040    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15246    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       15246    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       15246    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       15246    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       15246    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       15246    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15666    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       15666    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       15666    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       15666    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       15666    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       15666    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       15704    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       16887    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       16887    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       16887    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       16887    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       19623    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       19623    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       19623    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       19652    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       19652    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       19652    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       19652    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       19652    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       19652    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       19991    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       19991    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003rpm       22038    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22038    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22038    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22038    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22038    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22038    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22063    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22063    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22063    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22063    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22063    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22063    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22441    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22441    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22441    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22441    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22441    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22441    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22680    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22680    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22680    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22680    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22680    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22680    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22737    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22737    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22737    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22737    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22737    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22737    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       22972    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       22972    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       22972    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       22972    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       22972    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       22972    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       23761    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       23761    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       23761    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       23761    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       23761    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       23761    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       24215    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       24215    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       24215    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       24215    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       24215    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       24215    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       24322    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       24322    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       24322    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       24322    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       24322    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       24322    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       25122    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       27399    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       27399    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       27399    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       27399    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
bash      27840    root  cwd    DIR  253,6     4096     2 /var
rpm       28485    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       28485    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       28933    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       28933    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       28933    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       29026    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       29026    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       29026    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       29026    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       29026    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       29026    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       29035    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       29035    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       29035    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       29035    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       29035    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       29035    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
lsof      29954    root  cwd    DIR  253,6     4096     2 /var
lsof      29955    root  cwd    DIR  253,6     4096     2 /var
rpm       29960    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       29960    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       29960    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       29960    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       29960    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       29960    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
yum       29976    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
yum       29976    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
yum       29976    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
yum       29976    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
yum       29976    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
yum       29976    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       30208    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       30208    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       30208    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       30208    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       30208    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       30208    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       30648    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       30848    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       30848    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       30848    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       30848    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       30848    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       30848    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       31019    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       31019    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       31019    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       31019    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       31019    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       31019    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name
rpm       31036    root  mem    REG  253,6   770048  8223 /var/lib/rpm/__db.004
rpm       31036    root  mem    REG  253,6  1318912  8203 /var/lib/rpm/__db.003
rpm       32675    root  mem    REG  253,6   253952  8202 /var/lib/rpm/__db.002
rpm       32675    root  mem    REG  253,6    73728  8198 /var/lib/rpm/__db.001
rpm       32675    root    3rR  REG  253,6 58920960  8199 /var/lib/rpm/Packages
rpm       32675    root    4r   REG  253,6    24576  8200 /var/lib/rpm/Name

[root@ngelinux01 var]# du -sh /var/lib/rpm/Packages
57M     /var/lib/rpm/Packages

[root@ngelinux01 var]# ps -ef | grep -i var
root      6619     1  0 Nov05 ?        00:00:13 /sbin/rsyslogd -i /var/run/syslogd.pid -c 5
ntp       7156     1  0 Nov05 ?        00:00:02 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
root     31389 27840  0 11:34 pts/2    00:00:00 grep -i var

 

b. No files found using space, try to restart Kdump service.

[root@ngelinux01 var]# service kdump restart
Stopping kdump:                                            [  OK  ]
Starting kdump:                                            [  OK  ]
[root@ngelinux01 var]#


### No improvement after service restart.
[root@ngelinux01 var]# df -h /var
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-varlv
                      7.9G  7.9G     0 100% /var
[root@ngelinux01 var]#

### /var is 100% used, and /var/crash is empty.
[root@ngelinux01 var]# df -h /var/crash
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/rootvg-crashlv
                       35G  177M   33G   1% /var/crash
[root@ngelinux01 var]#

 

III. Solution

If you have observed, /var became full after kdump collection and /var/crash is empty as it is a separate partition.
So it seems interesting, when we crashed the system using echo c > /proc/sysrq-trigger, it just umounted other partitions on system except the main hierarchy and hence kdump was collected, however not visible.

Lets see the core files.

### First umount /var/crash
[root@ngelinux01 var]# umount /var/crash

### Now list the files
[root@ngelinux01 var]# ls -ltr /var/crash
total 12
drwsrwsrwt. 2 root root 4096 May 24 20:21 127.0.0.1-2021-05-24-20:14:31
drwsrwsrwt. 2 root root 4096 Aug 30 14:10 127.0.0.1-2021-08-30-14:10:01
drwsrwsrwt. 2 root root 4096 Nov 5 15:37 127.0.0.1-2021-11-05-15:37:46
[root@ngelinux01 var]#

So its interesting that we can see the files now.

Hence we came to conclusion that system umounts all non-dependent filesystems when we crash the system and collects the dump on main partition path.

Leave a Reply

Your email address will not be published.