How to debug core file in Linux using gdb and crash utility ?

By Saket Jain Published February 27, 2018 Linux/Unix

In this post we will look how to debug core file generated either by process fault or by system crash.

To debug a core file, you need to make sure we have gdb or crash utility, and vmlinuz debug symbols installed.

Lets have a look how to debug a core file step by step.

1. Make sure gdb is installed on your system.

### Check if gdb is installed.
[root@ngelinux 2.6.32-696.el6.i686]# rpm -q gdb
gdb-7.2-92.el6.i686

### GDB is installed on my system, however if its not installed on your system then:
### Install it using yum.
[root@ngelinux 2.6.32-696.el6.i686]# yum install gdb
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process

2. Install Crash Utility
We also have crash utility, which provides better output when debugging the file, however sometimes it is not able to debug.

Hence we recommend to use this utility first and in case its failed, go with gdb.

[root@ngelinux ~]# yum install crash
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process
Repository remi is listed more than once in the configuration
Loading mirror speeds from cached hostfile
epel/metalink                                                                | 5.3 kB     00:00     
epel-debuginfo/metalink                                                      | 5.5 kB     00:00     
epel-source/metalink                                                         | 5.3 kB     00:00     
epel-testing/metalink                                                        | 6.0 kB     00:00     
epel-testing-debuginfo/metalink                                              | 6.1 kB     00:00     
epel-testing-source/metalink                                                 | 6.0 kB     00:00     
remi/metalink                                                                | 5.3 kB     00:00     
 * base: del-mirrors.extreme-ix.org
 * epel: mirror.rise.ph
 * epel-debuginfo: mirror.rise.ph
 * epel-source: mirror.rise.ph
 * epel-testing: mirror.rise.ph
 * epel-testing-debuginfo: mirror.rise.ph
 * epel-testing-source: mirror.rise.ph
 * extras: del-mirrors.extreme-ix.org
 * remi: mirror.rise.ph
 * remi-php54: fr.mirror.babylon.network
 * remi-php55: fr.mirror.babylon.network
 * remi-php56: fr.mirror.babylon.network
 * remi-php70: fr.mirror.babylon.network
 * remi-php70-test: fr.mirror.babylon.network
 * remi-php72: fr.mirror.babylon.network
 * remi-php72-test: fr.mirror.babylon.network
 * remi-safe: fr.mirror.babylon.network
 * remi-test: fr.mirror.babylon.network
 * updates: del-mirrors.extreme-ix.org
C6.0-base                                                                    | 3.7 kB     00:00     
C6.2-base                                                                    | 3.7 kB     00:00     
C6.2-extras                                                                  | 2.9 kB     00:00     
C6.2-updates                                                                 | 2.9 kB     00:00     

Resolving Dependencies
--> Running transaction check
---> Package crash.i686 0:7.1.0-6.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package              Arch                Version                    Repository                Size
====================================================================================================
Installing:
 crash                i686                7.1.0-6.el6                C6.8-base                2.4 M

Transaction Summary
====================================================================================================
Install       1 Package(s)

Total download size: 2.4 M
Installed size: 6.6 M
Is this ok [y/N]: y
Downloading Packages:
crash-7.1.0-6.el6.i686.rpm                                                   | 2.4 MB     00:09     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : crash-7.1.0-6.el6.i686                                                           1/1 
  Verifying  : crash-7.1.0-6.el6.i686                                                           1/1 

Installed:
  crash.i686 0:7.1.0-6.el6                                                                          

Complete!
[root@ngelinux ~]#

What is crash utility ?
From its MAN page, we can have its definition as:
Crash is a utility to analyze the state of the Linux system interactively whether it is in running state, or after a kernel crash has occurred and a core dump file has been created either by netdump, LKCD, diskdump, kdump, xendump or kvmdump facilities.

### Lets see what files created by crash utility.
[root@ngelinux ~]# rpm -ql crash
/usr/bin/crash
/usr/share/doc/crash-7.1.0
/usr/share/doc/crash-7.1.0/COPYING3
/usr/share/doc/crash-7.1.0/README
/usr/share/man/man8/crash.8.gz

### Check details in its man page
[root@ngelinux ~]# man crash
CRASH(8)                                                              CRASH(8)
NAME
       crash - Analyze Linux crash dump data or a live system

3. Install Kernel Specific Debuginfo on which core file is generated
Now we have the tool installed on our system to debug the core file.

Its time to install the debuginfo rpm package which contains the symbols to decode the core file and read where the crash has occurred.

[root@ngelinux ~]# yum --enablerepo=base-debuginfo install kernel-debuginfo-2.6.32-696.el6
Loaded plugins: fastestmirror, refresh-packagekit, security
Repository remi is listed more than once in the configuration
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: del-mirrors.extreme-ix.org
 * epel: mirror.rise.ph
 * epel-debuginfo: mirror.rise.ph
 * epel-source: mirror.rise.ph
 * epel-testing: mirror.rise.ph
 * epel-testing-debuginfo: mirror.rise.ph
 * epel-testing-source: mirror.rise.ph
 * extras: del-mirrors.extreme-ix.org
 * remi: mirror.rise.ph
 * remi-php54: fr.mirror.babylon.network
 * remi-php55: fr.mirror.babylon.network
 * remi-php56: fr.mirror.babylon.network
 * remi-php70: fr.mirror.babylon.network
 * remi-php70-test: fr.mirror.babylon.network
 * remi-php72: fr.mirror.babylon.network
 * remi-php72-test: fr.mirror.babylon.network
 * remi-safe: fr.mirror.babylon.network
 * remi-test: fr.mirror.babylon.network
 * updates: del-mirrors.extreme-ix.org
base-debuginfo                                                               | 2.5 kB     00:00     
base-debuginfo/primary_db                                                    | 1.8 MB     00:12     
Resolving Dependencies
--> Running transaction check
---> Package kernel-debuginfo.i686 0:2.6.32-696.el6 will be installed
--> Processing Dependency: kernel-debuginfo-common-i686 = 2.6.32-696.el6 for package: kernel-debuginfo-2.6.32-696.el6.i686
--> Running transaction check
---> Package kernel-debuginfo-common-i686.i686 0:2.6.32-696.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package                             Arch        Version                Repository             Size
====================================================================================================
Installing:
 kernel-debuginfo                    i686        2.6.32-696.el6         base-debuginfo        271 M
Installing for dependencies:
 kernel-debuginfo-common-i686        i686        2.6.32-696.el6         base-debuginfo         44 M

Transaction Summary
====================================================================================================
Install       2 Package(s)

Total download size: 316 M
Installed size: 1.4 G
Is this ok [y/N]: y
Downloading Packages:
(1/2): kernel-debuginfo-2.6.32-696.el6.i686.rpm                              | 271 MB     15:38     
(2/2): kernel-debuginfo-common-i686-2.6.32-696.el6.i686.rpm                  |  44 MB     02:33     
----------------------------------------------------------------------------------------------------
Total                                                               295 kB/s | 316 MB     18:14     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : kernel-debuginfo-common-i686-2.6.32-696.el6.i686                                 1/2 
  Installing : kernel-debuginfo-2.6.32-696.el6.i686                                             2/2 
  Verifying  : kernel-debuginfo-common-i686-2.6.32-696.el6.i686                                 1/2 
  Verifying  : kernel-debuginfo-2.6.32-696.el6.i686                                             2/2 

Installed:
  kernel-debuginfo.i686 0:2.6.32-696.el6                                                            

Dependency Installed:
  kernel-debuginfo-common-i686.i686 0:2.6.32-696.el6                                                

Complete!
[root@ngelinux ~]#

4. Find path of vmlinux file to debug the core file.

[root@ngelinux nglinux]# cd /usr/lib/debug/lib/modules/2.6.32-696.el6.i686/
[root@ngelinux 2.6.32-696.el6.i686]# ls
kernel  vdso  vmlinux

5. Debugging the core file using gdb.

[root@ngelinux 2.6.32-696.el6.i686]# gdb vmlinux /proc/kcore 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/lib/debug/lib/modules/2.6.32-696.el6.i686/vmlinux...done.
BFD: Warning: /proc/kcore is truncated: expected core file size >= 1063251968, found: 4096.
[New ]
Core was generated by `root=UUID=e91618e7-a10b-499a-9652-682de8b5cfa1 rhgb quiet'.
#0  0x00000000 in ?? ()
(gdb)

6. Debugging with crash utility.

### Get current process id.
[root@ngelinux 2.6.32-696.el6.i686]# echo $$
333

### Generate core file.
[root@ngelinux 2.6.32-696.el6.i686]# gcore 333
__kernel_vsyscall () at arch/x86/vdso/vdso32/sysenter.S:49
49		pop %ebp
Saved corefile core.333

[root@ngelinux 2.6.32-696.el6.i686]# ls
core.333  kernel  vdso  vmlinux

### Read core file using crash utility
[root@ngelinux 2.6.32-696.el6.i686]# crash core.333 vmlinux 

crash 7.1.0-6.el6
Copyright (C) 2002-2014  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-linux-gnu"...


crash: read error: kernel virtual address: c088fcc8  type: "cpu_possible_mask"
[root@ngelinux 2.6.32-696.el6.i686]#

We can see above that our debugging session is not initiated in some cases using crash.
In such situations, we can use gdb command as follows.

[root@ngelinux 2.6.32-696.el6.i686]# gdb vmlinux core.333 
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/lib/debug/lib/modules/2.6.32-696.el6.i686/vmlinux...done.

warning: core file may not match specified executable file.
[New Thread 333]
Core was generated by `-bash'.
#0  __kernel_vsyscall () at arch/x86/vdso/vdso32/sysenter.S:49
49		pop %ebp
(gdb)

7. Debug in detail using crash utility.

[root@ngelinux 2.6.32-696.el6.i686]# crash -d8 vmlinux core.333 
crash 7.1.0-6.el6
Copyright (C) 2002-2014  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
vmcore_data: 
                  flags: a0 (KDUMP_LOCAL|KDUMP_ELF32) 
                   ndfd: 3
                    ofp: 339960
            header_size: 2000
   num_pt_load_segments: 18
     pt_load_segment[0]:
            file_offset: 7d0
             phys_start: 0
               phys_end: 3000
              zero_fill: 0
     pt_load_segment[1]:
            file_offset: 37d0
             phys_start: 0
               phys_end: 1000
              zero_fill: 0
     pt_load_segment[2]:
            file_offset: 47d0
             phys_start: 0
               phys_end: 1000
              zero_fill: 0
     pt_load_segment[3]:
            file_offset: 57d0
             phys_start: 0
               phys_end: 2000
              zero_fill: 0
     pt_load_segment[4]:
            file_offset: 77d0
             phys_start: 0
               phys_end: 1000
              zero_fill: 0
     pt_load_segment[5]:

Module debugging in Solaris

Similarly in solaris, we have mdb command in place of gdb i.e. known as Module debugger.

Debug file using module debugger.

### Start module debugger in read mode
# mdb -k
::pgrep tty

### Start MDB in read-write mode to change any paramater.
# mdb -kw

0 0 votes

Article Rating

Tags: crash debuginfo gdb gdb vmlinux kw linux mdb solaris vmlinux vmlinuz yum

2 Comments

Newest

Oldest Most Voted

Inline Feedbacks

View all comments

Anonymous

3 years ago

This article is wrong and misleading.

Mohit

6 years ago

Good article…

Vishu on How to create full size one partition using parted command in Linux ?: “Thanks a lot. This was exactly what I was looking for. Other blogs are very confusing but this worked for…” Jul 30, 23:26
cccc on Print only usernames from /etc/passwd file using grep, awk or cut commands.: “love it” Oct 18, 16:13
Saket Jain on How to configure and install Nagios Server on Linux ?: “Please check your system resolv.conf/DNS settings, it looks its not able to resolve the hostname. The URL is correct.” Jul 18, 13:37
deepanshu on How to configure and install Nagios Server on Linux ?: “[root@localhost nagios]# wget https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.4.5.tar.gz –2023-07-02 19:15:08– https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.4.5.tar.gz Resolving assets.nagios.com (assets.nagios.com)… failed: Name or service not known. wget: unable to resolve host…” Jul 3, 08:13
aasdasdKEKEK on Solved: subscription-manager – Not supported by a valid subscription.: “You Genius. How do we “verify if we have enough subscription available on redhat support to add this new server.”” May 27, 18:26

How to debug core file in Linux using gdb and crash utility ?

Module debugging in Solaris

Like this:

Related

TAGS

Module debugging in Solaris

Share this NG Linux post:

Like this:

Related