Here we will look at the process how to debug a crash dump in Linux using gdb.
What all tools are required and how to start debugging the core file generated.
1. Install GNU Debugger GDB
[root@nglinux ~]# yum install gdb Failed to set locale, defaulting to C Loaded plugins: fastestmirror, refresh-packagekit, security Setting up Install Process Loading mirror speeds from cached hostfile epel/metalink | 17 kB 00:00 * base: mirror.riverfrontnetworks.com * epel: ewr.edge.kernel.org * extras: mirror.team-cymru.com * updates: mirror.metrocast.net base | 3.7 kB 00:00 base-debuginfo | 2.5 kB 00:00 base-debuginfo/primary_db | 1.8 MB 00:05 extras | 3.3 kB 00:00 updates | 3.4 kB 00:00 updates/primary_db | 14 kB 00:00 Resolving Dependencies --> Running transaction check ---> Package gdb.i686 0:7.2-92.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ==================================================================================================================================================== Package Arch Version Repository Size ==================================================================================================================================================== Installing: gdb i686 7.2-92.el6 base 2.3 M Transaction Summary ==================================================================================================================================================== Install 1 Package(s) Total download size: 2.3 M Installed size: 5.1 M Is this ok [y/N]: y Downloading Packages: gdb-7.2-92.el6.i686.rpm | 2.3 MB 00:02 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : gdb-7.2-92.el6.i686 1/1 Verifying : gdb-7.2-92.el6.i686 1/1 Installed: gdb.i686 0:7.2-92.el6 Complete! [root@nglinux ~]#
2. Check the core file pattern if defined in /etc/sysctl.conf or in /etc/kdmp.conf
[root@nglinux ~]# (sysctl -p; cat /etc/kdump.conf) | grep -i core kernel.core_pattern = /var/crash/core.%e.%p.%u
To generate a sample core file, you can do following as an example:
1. On Terminal 1, run command: # top
2. On Terminal 2, kill the process using: kill -ABRT 3835
1. On Terminal 1, run command: # top
2. On Terminal 2, kill the process using: kill -ABRT 3835
For more details refer to article: http://ngelinux.com/how-to-enable-core-dump-collection-in-linux-for-every-user/
3. Now check out the generated core file.
[root@nglinux crash]# ls -ltr total 952 -rw-------. 1 root root 581632 Jul 9 23:59 core.0.top.3835 -rw-------. 1 root root 581632 Jul 10 00:01 core.0.top.3848 drwxr-xr-x. 2 root root 4096 Jul 10 20:51 127.0.0.1-2018-07-10-20:51:46 drwxr-xr-x. 2 root root 4096 Jul 11 20:27 127.0.0.1-2018-07-11-20:26:50 -rw-------. 1 root root 577536 Jul 18 06:07 core.top.5604.0
4. Debug the generated core file using the binary file which generated this core file.
[root@nglinux crash]# gdb `which top` /var/crash/core.top.5604.0 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: ... Reading symbols from /usr/bin/top...(no debugging symbols found)...done. [New Thread 5604] Reading symbols from /lib/libproc-3.2.8.so...(no debugging symbols found)...done. Loaded symbols for /lib/libproc-3.2.8.so Reading symbols from /lib/libncursesw.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libncursesw.so.5 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib/libtinfo.so.5...(no debugging symbols found)...done. Loaded symbols for /lib/libtinfo.so.5 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Core was generated by `top'. Program terminated with signal 11, Segmentation fault. #0 0x00979424 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install procps-3.2.8-45.el6.i686 (gdb)