In this post we will look how to debug core file generated either by process fault or by system crash.
To debug a core file, you need to make sure we have gdb or crash utility, and vmlinuz debug symbols installed.
Lets have a look how to debug a core file step by step.
1. Make sure gdb is installed on your system.
### Check if gdb is installed. [root@ngelinux 2.6.32-696.el6.i686]# rpm -q gdb gdb-7.2-92.el6.i686 ### GDB is installed on my system, however if its not installed on your system then: ### Install it using yum. [root@ngelinux 2.6.32-696.el6.i686]# yum install gdb Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Install Process
2. Install Crash Utility
We also have crash utility, which provides better output when debugging the file, however sometimes it is not able to debug.
Hence we recommend to use this utility first and in case its failed, go with gdb.
[root@ngelinux ~]# yum install crash Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Install Process Repository remi is listed more than once in the configuration Loading mirror speeds from cached hostfile epel/metalink | 5.3 kB 00:00 epel-debuginfo/metalink | 5.5 kB 00:00 epel-source/metalink | 5.3 kB 00:00 epel-testing/metalink | 6.0 kB 00:00 epel-testing-debuginfo/metalink | 6.1 kB 00:00 epel-testing-source/metalink | 6.0 kB 00:00 remi/metalink | 5.3 kB 00:00 * base: del-mirrors.extreme-ix.org * epel: mirror.rise.ph * epel-debuginfo: mirror.rise.ph * epel-source: mirror.rise.ph * epel-testing: mirror.rise.ph * epel-testing-debuginfo: mirror.rise.ph * epel-testing-source: mirror.rise.ph * extras: del-mirrors.extreme-ix.org * remi: mirror.rise.ph * remi-php54: fr.mirror.babylon.network * remi-php55: fr.mirror.babylon.network * remi-php56: fr.mirror.babylon.network * remi-php70: fr.mirror.babylon.network * remi-php70-test: fr.mirror.babylon.network * remi-php72: fr.mirror.babylon.network * remi-php72-test: fr.mirror.babylon.network * remi-safe: fr.mirror.babylon.network * remi-test: fr.mirror.babylon.network * updates: del-mirrors.extreme-ix.org C6.0-base | 3.7 kB 00:00 C6.2-base | 3.7 kB 00:00 C6.2-extras | 2.9 kB 00:00 C6.2-updates | 2.9 kB 00:00 Resolving Dependencies --> Running transaction check ---> Package crash.i686 0:7.1.0-6.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================== Package Arch Version Repository Size ==================================================================================================== Installing: crash i686 7.1.0-6.el6 C6.8-base 2.4 M Transaction Summary ==================================================================================================== Install 1 Package(s) Total download size: 2.4 M Installed size: 6.6 M Is this ok [y/N]: y Downloading Packages: crash-7.1.0-6.el6.i686.rpm | 2.4 MB 00:09 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : crash-7.1.0-6.el6.i686 1/1 Verifying : crash-7.1.0-6.el6.i686 1/1 Installed: crash.i686 0:7.1.0-6.el6 Complete! [root@ngelinux ~]#
What is crash utility ?
From its MAN page, we can have its definition as:
Crash is a utility to analyze the state of the Linux system interactively whether it is in running state, or after a kernel crash has occurred and a core dump file has been created either by netdump, LKCD, diskdump, kdump, xendump or kvmdump facilities.
From its MAN page, we can have its definition as:
Crash is a utility to analyze the state of the Linux system interactively whether it is in running state, or after a kernel crash has occurred and a core dump file has been created either by netdump, LKCD, diskdump, kdump, xendump or kvmdump facilities.
### Lets see what files created by crash utility.
[root@ngelinux ~]# rpm -ql crash
/usr/bin/crash
/usr/share/doc/crash-7.1.0
/usr/share/doc/crash-7.1.0/COPYING3
/usr/share/doc/crash-7.1.0/README
/usr/share/man/man8/crash.8.gz
### Check details in its man page
[root@ngelinux ~]# man crash
CRASH(8) CRASH(8)
NAME
crash - Analyze Linux crash dump data or a live system
3. Install Kernel Specific Debuginfo on which core file is generated
Now we have the tool installed on our system to debug the core file.
Its time to install the debuginfo rpm package which contains the symbols to decode the core file and read where the crash has occurred.
[root@ngelinux ~]# yum --enablerepo=base-debuginfo install kernel-debuginfo-2.6.32-696.el6 Loaded plugins: fastestmirror, refresh-packagekit, security Repository remi is listed more than once in the configuration Setting up Install Process Loading mirror speeds from cached hostfile * base: del-mirrors.extreme-ix.org * epel: mirror.rise.ph * epel-debuginfo: mirror.rise.ph * epel-source: mirror.rise.ph * epel-testing: mirror.rise.ph * epel-testing-debuginfo: mirror.rise.ph * epel-testing-source: mirror.rise.ph * extras: del-mirrors.extreme-ix.org * remi: mirror.rise.ph * remi-php54: fr.mirror.babylon.network * remi-php55: fr.mirror.babylon.network * remi-php56: fr.mirror.babylon.network * remi-php70: fr.mirror.babylon.network * remi-php70-test: fr.mirror.babylon.network * remi-php72: fr.mirror.babylon.network * remi-php72-test: fr.mirror.babylon.network * remi-safe: fr.mirror.babylon.network * remi-test: fr.mirror.babylon.network * updates: del-mirrors.extreme-ix.org base-debuginfo | 2.5 kB 00:00 base-debuginfo/primary_db | 1.8 MB 00:12 Resolving Dependencies --> Running transaction check ---> Package kernel-debuginfo.i686 0:2.6.32-696.el6 will be installed --> Processing Dependency: kernel-debuginfo-common-i686 = 2.6.32-696.el6 for package: kernel-debuginfo-2.6.32-696.el6.i686 --> Running transaction check ---> Package kernel-debuginfo-common-i686.i686 0:2.6.32-696.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================== Package Arch Version Repository Size ==================================================================================================== Installing: kernel-debuginfo i686 2.6.32-696.el6 base-debuginfo 271 M Installing for dependencies: kernel-debuginfo-common-i686 i686 2.6.32-696.el6 base-debuginfo 44 M Transaction Summary ==================================================================================================== Install 2 Package(s) Total download size: 316 M Installed size: 1.4 G Is this ok [y/N]: y Downloading Packages: (1/2): kernel-debuginfo-2.6.32-696.el6.i686.rpm | 271 MB 15:38 (2/2): kernel-debuginfo-common-i686-2.6.32-696.el6.i686.rpm | 44 MB 02:33 ---------------------------------------------------------------------------------------------------- Total 295 kB/s | 316 MB 18:14 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : kernel-debuginfo-common-i686-2.6.32-696.el6.i686 1/2 Installing : kernel-debuginfo-2.6.32-696.el6.i686 2/2 Verifying : kernel-debuginfo-common-i686-2.6.32-696.el6.i686 1/2 Verifying : kernel-debuginfo-2.6.32-696.el6.i686 2/2 Installed: kernel-debuginfo.i686 0:2.6.32-696.el6 Dependency Installed: kernel-debuginfo-common-i686.i686 0:2.6.32-696.el6 Complete! [root@ngelinux ~]#
4. Find path of vmlinux file to debug the core file.
[root@ngelinux nglinux]# cd /usr/lib/debug/lib/modules/2.6.32-696.el6.i686/ [root@ngelinux 2.6.32-696.el6.i686]# ls kernel vdso vmlinux
5. Debugging the core file using gdb.
[root@ngelinux 2.6.32-696.el6.i686]# gdb vmlinux /proc/kcore GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /usr/lib/debug/lib/modules/2.6.32-696.el6.i686/vmlinux...done. BFD: Warning: /proc/kcore is truncated: expected core file size >= 1063251968, found: 4096. [New] Core was generated by `root=UUID=e91618e7-a10b-499a-9652-682de8b5cfa1 rhgb quiet'. #0 0x00000000 in ?? () (gdb)
6. Debugging with crash utility.
### Get current process id. [root@ngelinux 2.6.32-696.el6.i686]# echo $$ 333 ### Generate core file. [root@ngelinux 2.6.32-696.el6.i686]# gcore 333 __kernel_vsyscall () at arch/x86/vdso/vdso32/sysenter.S:49 49 pop %ebp Saved corefile core.333 [root@ngelinux 2.6.32-696.el6.i686]# ls core.333 kernel vdso vmlinux ### Read core file using crash utility [root@ngelinux 2.6.32-696.el6.i686]# crash core.333 vmlinux crash 7.1.0-6.el6 Copyright (C) 2002-2014 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. GNU gdb (GDB) 7.6 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-pc-linux-gnu"... crash: read error: kernel virtual address: c088fcc8 type: "cpu_possible_mask" [root@ngelinux 2.6.32-696.el6.i686]#
We can see above that our debugging session is not initiated in some cases using crash.
In such situations, we can use gdb command as follows.
In such situations, we can use gdb command as follows.
[root@ngelinux 2.6.32-696.el6.i686]# gdb vmlinux core.333 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /usr/lib/debug/lib/modules/2.6.32-696.el6.i686/vmlinux...done. warning: core file may not match specified executable file. [New Thread 333] Core was generated by `-bash'. #0 __kernel_vsyscall () at arch/x86/vdso/vdso32/sysenter.S:49 49 pop %ebp (gdb)
7. Debug in detail using crash utility.
[root@ngelinux 2.6.32-696.el6.i686]# crash -d8 vmlinux core.333
crash 7.1.0-6.el6
Copyright (C) 2002-2014 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
vmcore_data:
flags: a0 (KDUMP_LOCAL|KDUMP_ELF32)
ndfd: 3
ofp: 339960
header_size: 2000
num_pt_load_segments: 18
pt_load_segment[0]:
file_offset: 7d0
phys_start: 0
phys_end: 3000
zero_fill: 0
pt_load_segment[1]:
file_offset: 37d0
phys_start: 0
phys_end: 1000
zero_fill: 0
pt_load_segment[2]:
file_offset: 47d0
phys_start: 0
phys_end: 1000
zero_fill: 0
pt_load_segment[3]:
file_offset: 57d0
phys_start: 0
phys_end: 2000
zero_fill: 0
pt_load_segment[4]:
file_offset: 77d0
phys_start: 0
phys_end: 1000
zero_fill: 0
pt_load_segment[5]:
Module debugging in Solaris
Similarly in solaris, we have mdb command in place of gdb i.e. known as Module debugger.
Debug file using module debugger.
### Start module debugger in read mode # mdb -k ::pgrep tty ### Start MDB in read-write mode to change any paramater. # mdb -kw