How to debug crash dump in Linux using crash and debuginfo ?

In this post, we will look how to use crash utility to debug the generated core file(via kdump).

To understand this post, you should know what is kdump and how to install & configure it.
Please click here to see above details: http://ngelinux.com/how-to-setup-kdump-in-linux-to-collect-crash-dump/

To run crash utility, we will run crash command followed by generated vmcore file with the debuginfo kernel file.

1. Running Crash utility

[root@nglinux 127.0.0.1-2018-07-11-20:26:50]# crash ./vmcore /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux 
crash 7.1.0-8.el6
Copyright (C) 2002-2014  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...


      KERNEL: /usr/lib/debug/lib/modules/2.6.32-754.el6.i686/vmlinux
    DUMPFILE: ./vmcore  [PARTIAL DUMP]
        CPUS: 1
        DATE: Wed Jul 11 20:26:46 2018
      UPTIME: 00:03:13
LOAD AVERAGE: 0.21, 0.23, 0.10
       TASKS: 234
    NODENAME: nglinux
     RELEASE: 2.6.32-754.el6.i686
     VERSION: #1 SMP Tue Jun 19 21:51:20 UTC 2018
     MACHINE: i686  (2693 Mhz)
      MEMORY: 1 GB
       PANIC: "SysRq : Trigger a crash"
         PID: 2417
     COMMAND: "bash"
        TASK: f4fb2000  [THREAD_INFO: f288a000]
         CPU: 0
       STATE: TASK_RUNNING (SYSRQ)

crash> 
crash> 
crash> 
crash> help

*              files          mach           repeat         timer          
alias          foreach        mod            runq           tree           
ascii          fuser          mount          search         union          
bt             gdb            net            set            vm             
btop           help           p              sig            vtop           
dev            ipcs           ps             struct         waitq          
dis            irq            pte            swap           whatis         
eval           kmem           ptob           sym            wr             
exit           list           ptov           sys            q              
extend         log            rd             task           

crash version: 7.1.0-8.el6   gdb version: 7.6
For help on any command above, enter "help ".
For help on input options, enter "help input".
For help on output options, enter "help output".

crash> 

As we can see above, we can use “help” command to see all available options.

2. See all processes
Now we are at the debugging crash prompt. We can use different command to analyze the system state during the crash situation.
To see all processes, we can use ps command.

crash> ps
   PID    PPID  CPU   TASK    ST  %MEM     VSZ    RSS  COMM
      0      0   0  c0abbc40  RU   0.0       0      0  [swapper]
      1      0   0  f703eaa0  IN   0.1    2896   1436  init
      2      0   0  f703e550  IN   0.0       0      0  [kthreadd]

3. To see the available/used virtual memory

crash> vm
PID: 2417   TASK: f4fb2000  CPU: 0   COMMAND: "bash"
   MM       PGD      RSS    TOTAL_VM
f5102580  f28a8000  1688k    5244k  
  VMA       START      END    FLAGS  FILE
f29cb8ec    2f2000    2f3000  40075  
f29cb4a0    47a000    490000 8000075  /lib/libtinfo.so.5.7
f29cb504    490000    493000 8100073  /lib/libtinfo.so.5.7
f29cb374    622000    62e000 8000075  /lib/libnss_files-2.12.so
f2a0743c    62e000    62f000 8100071  /lib/libnss_files-2.12.so
f2a073d8    62f000    630000 8100073  /lib/libnss_files-2.12.so
f29cb824    898000    8b6000 8000875  /lib/ld-2.12.so
f29cb888    8b6000    8b7000 8100871  /lib/ld-2.12.so
f29cb43c    8b7000    8b8000 8100873  /lib/ld-2.12.so

4. See all open files

crash> files
PID: 2417   TASK: f4fb2000  CPU: 0   COMMAND: "bash"
ROOT: /    CWD: /var/crash
 FD    FILE     DENTRY    INODE    TYPE  PATH
  0  f4f80480  c1b56d74  c1af71e8  CHR   /dev/pts/0
  1  f287c500  eee27324  f6fdc4dc  REG   /proc/sysrq-trigger
  2  f4f80480  c1b56d74  c1af71e8  CHR   /dev/pts/0
 10  f4f80480  c1b56d74  c1af71e8  CHR   /dev/pts/0
255  f4f80480  c1b56d74  c1af71e8  CHR   /dev/pts/0
crash> 

5. See kernel backtrace logs.

crash> bt
PID: 2417   TASK: f4fb2000  CPU: 0   COMMAND: "bash"
 #0 [f288bda8] crash_kexec at c04b0abc
 #1 [f288be00] oops_end at c088c302
 #2 [f288be14] no_context at c04425dd
 #3 [f288be38] bad_area at c0442856
 #4 [f288be4c] __do_page_fault at c0442d20
 #5 [f288bec8] do_page_fault at c088dc35
 #6 [f288bee0] error_code (via page_fault) at c088b27d
    EAX: 00000063  EBX: 00000063  ECX: c06cdec0  EDX: 00000000  EBP: 00000000 
    DS:  007b      ESI: 00000286  ES:  007b      EDI: c0af50e0  GS:  00e0
    CS:  0060      EIP: c06cdecf  ERR: ffffffff  EFLAGS: 00010082 
 #7 [f288bf14] sysrq_handle_crash at c06cdecf
 #8 [f288bf20] __handle_sysrq at c06ce0eb
 #9 [f288bf44] write_sysrq_trigger at c06ce18a

Tags: analysisbtcorecrashdebugdumplinuxpsunixvmvmcorevmlinux