In this post, we will look how to use crash utility to debug the generated core file(via kdump).
To understand this post, you should know what is kdump and how to install & configure it.
Please click here to see above details: http://ngelinux.com/how-to-setup-kdump-in-linux-to-collect-crash-dump/
To run crash utility, we will run crash command followed by generated vmcore file with the debuginfo kernel file.
1. Running Crash utility
[root@nglinux 127.0.0.1-2018-07-11-20:26:50]# crash ./vmcore /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux
crash 7.1.0-8.el6
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...
KERNEL: /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux
DUMPFILE: ./vmcore [PARTIAL DUMP]
CPUS: 1
DATE: Wed Jul 11 20:26:46 2018
UPTIME: 00:03:13
LOAD AVERAGE: 0.21, 0.23, 0.10
TASKS: 234
NODENAME: nglinux
RELEASE: 2.6.32-754.el6.i686
VERSION: #1 SMP Tue Jun 19 21:51:20 UTC 2018
MACHINE: i686 (2693 Mhz)
MEMORY: 1 GB
PANIC: "SysRq : Trigger a crash"
PID: 2417
COMMAND: "bash"
TASK: f4fb2000 [THREAD_INFO: f288a000]
CPU: 0
STATE: TASK_RUNNING (SYSRQ)
crash>
crash>
crash>
crash> help
* files mach repeat timer
alias foreach mod runq tree
ascii fuser mount search union
bt gdb net set vm
btop help p sig vtop
dev ipcs ps struct waitq
dis irq pte swap whatis
eval kmem ptob sym wr
exit list ptov sys q
extend log rd task
crash version: 7.1.0-8.el6 gdb version: 7.6
For help on any command above, enter "help ".
For help on input options, enter "help input".
For help on output options, enter "help output".
crash>
As we can see above, we can use “help” command to see all available options.
2. See all processes
Now we are at the debugging crash prompt. We can use different command to analyze the system state during the crash situation.
To see all processes, we can use ps command.
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
0 0 0 c0abbc40 RU 0.0 0 0 [swapper]
1 0 0 f703eaa0 IN 0.1 2896 1436 init
2 0 0 f703e550 IN 0.0 0 0 [kthreadd]
3. To see the available/used virtual memory
crash> vm PID: 2417 TASK: f4fb2000 CPU: 0 COMMAND: "bash" MM PGD RSS TOTAL_VM f5102580 f28a8000 1688k 5244k VMA START END FLAGS FILE f29cb8ec 2f2000 2f3000 40075 f29cb4a0 47a000 490000 8000075 /lib/libtinfo.so.5.7 f29cb504 490000 493000 8100073 /lib/libtinfo.so.5.7 f29cb374 622000 62e000 8000075 /lib/libnss_files-2.12.so f2a0743c 62e000 62f000 8100071 /lib/libnss_files-2.12.so f2a073d8 62f000 630000 8100073 /lib/libnss_files-2.12.so f29cb824 898000 8b6000 8000875 /lib/ld-2.12.so f29cb888 8b6000 8b7000 8100871 /lib/ld-2.12.so f29cb43c 8b7000 8b8000 8100873 /lib/ld-2.12.so
4. See all open files
crash> files PID: 2417 TASK: f4fb2000 CPU: 0 COMMAND: "bash" ROOT: / CWD: /var/crash FD FILE DENTRY INODE TYPE PATH 0 f4f80480 c1b56d74 c1af71e8 CHR /dev/pts/0 1 f287c500 eee27324 f6fdc4dc REG /proc/sysrq-trigger 2 f4f80480 c1b56d74 c1af71e8 CHR /dev/pts/0 10 f4f80480 c1b56d74 c1af71e8 CHR /dev/pts/0 255 f4f80480 c1b56d74 c1af71e8 CHR /dev/pts/0 crash>
5. See kernel backtrace logs.
crash> bt
PID: 2417 TASK: f4fb2000 CPU: 0 COMMAND: "bash"
#0 [f288bda8] crash_kexec at c04b0abc
#1 [f288be00] oops_end at c088c302
#2 [f288be14] no_context at c04425dd
#3 [f288be38] bad_area at c0442856
#4 [f288be4c] __do_page_fault at c0442d20
#5 [f288bec8] do_page_fault at c088dc35
#6 [f288bee0] error_code (via page_fault) at c088b27d
EAX: 00000063 EBX: 00000063 ECX: c06cdec0 EDX: 00000000 EBP: 00000000
DS: 007b ESI: 00000286 ES: 007b EDI: c0af50e0 GS: 00e0
CS: 0060 EIP: c06cdecf ERR: ffffffff EFLAGS: 00010082
#7 [f288bf14] sysrq_handle_crash at c06cdecf
#8 [f288bf20] __handle_sysrq at c06ce0eb
#9 [f288bf44] write_sysrq_trigger at c06ce18a