HowTo: Debug Crashed Linux Application Core Files Like A Pro

by on June 3, 2010 · 25 comments· LAST UPDATED June 18, 2010

in

Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

Turn On Core File Creation Support

By default most Linux distributions turn off core file creation (at least this is true for RHEL, CentOS, Fedora and Suse Linux). You need to use the ulimit command to configure core files.

See The Current Core File Limits

Type the following command:
# ulimit -c
Sample outputs:

0

The output 0 (zero) means core file is not created.

Change Core File Limits

In this example, set the size limit of core files to 75000 bytes:
# ulimit -c 75000

HowTo: Enable Core File Dumps For Application Crashes And Segmentation Faults

Edit /etc/profile file and find line that read as follows to make persistent configuration:

ulimit -S -c 0 > /dev/null 2>&1

Update it as follows:

ulimit -c unlimited >/dev/null 2>&1

Save and close the file. Edit /etc/sysctl.conf, enter:
# vi /etc/sysctl.conf
Append the following lines:

kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t
fs.suid_dumpable = 2

Save and close the file. Where,

  1. kernel.core_uses_pid = 1 - Appends the coring processes PID to the core file name.
  2. fs.suid_dumpable = 2 - Make sure you get core dumps for setuid programs.
  3. kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t - When the application terminates abnormally, a core file should appear in the /tmp. The kernel.core_pattern sysctl controls exact location of core file. You can define the core file name with the following template whih can contain % specifiers which are substituted by the following values when a core file is created:
    • %% - A single % character
    • %p - PID of dumped process
    • %u - real UID of dumped process
    • %g - real GID of dumped process
    • %s - number of signal causing dump
    • %t - time of dump (seconds since 0:00h, 1 Jan 1970)
    • %h - hostname (same as ’nodename’ returned by uname(2))
    • %e - executable filename

Finally, enable debugging for all apps, enter (Redhat and friends specific):
# echo "DAEMON_COREFILE_LIMIT='unlimited'" >> /etc/sysconfig/init
Reload the settings in /etc/sysctl.conf by running the following command:
# sysctl -p

How Do I Enable Core Dumping For Specific Deamon?

To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/init.d/lighttped and add line as follows:

DAEMON_COREFILE_LIMIT='unlimited'

Please note that DAEMON_COREFILE_LIMIT is Redhat specific, for all other distro add configuration as follows:

ulimit -c unlimited >/dev/null 2>&1
echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern

Save and close the file. Restart / reload lighttpd:
# /etc/init.d/lighttpd restart
# su - lighttpd
$ ulimit -c

Sample outputs:

unlimited

Now, you can send core files to vendor or software writes.

How Do I Read Core Files?

You need use the gdb command as follows:
$ gdb /path/to/application /path/to/corefile
See the gdb command man page for more information.

strace command

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers. See how to use the strace command under Linux to debug the problems.

Recommended readings:

Stay tunned for gdb tutorial which will explains how to use generated core file to track down problem.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 25 comments… read them below or add one }

1 shoaibi June 3, 2010 at 10:12 am

Great article, bravo :)

Reply

2 wonski June 3, 2010 at 11:35 am

pity there’s actually nothing about debugging..

Reply

3 nixCraft June 3, 2010 at 11:44 am

This part explained how to configure core dumps. I’m working on follow up article using gdb; so stay tuned.

Reply

4 neolix June 10, 2010 at 6:01 am

zahn pan tussi great ho topho kabul

Reply

5 storm June 3, 2010 at 12:44 pm

Vivek,

Your site is awesome especially for newbies and intermediates. Very clear and concise.

Thanks

Reply

6 Tapas Mallick June 3, 2010 at 1:36 pm

“strace command” link at the end of the article is not correct and pointing to the current web page only.

Reply

7 nixCraft June 3, 2010 at 5:16 pm

Thanks for the heads up!

@storm, thanks for the kind words!

Reply

8 Hitesh June 3, 2010 at 5:55 pm

Really nice info

Reply

9 nima0102 June 4, 2010 at 4:39 pm

Thanks for this article

Reply

10 Planet_Bob June 8, 2010 at 3:23 am

Nice article, just maybe make the subject line relevant to the article…

“How to enable core dumps” instead of “Debug apps”…esp since there is virtually nothing about debugging in this article.

Reply

11 Anonymous June 9, 2010 at 5:07 am

Hi,

Thanks for nice material, but i want to know how to analyze core dump
thanks again

Reply

12 Jalal Hajigholamali June 9, 2010 at 5:09 am

Hi,

thanks for nice article, but how can we analyze core dump ?
thanks again

Reply

13 lokesh June 10, 2010 at 11:42 am

you send the linux commnad on my Email-ID

Reply

14 benzopilka July 1, 2010 at 1:36 pm

Please, explain more clear wich line add into /etc/sysconfig/daemon-file file, and which line into /etc/init.d/lighttped file; in the part about Specific Deamon?

Reply

15 Vijay Nayani August 23, 2010 at 7:16 am

Hello All,

Anyboby ask me how do i read core files ???
how to use gdb command ?????

Reply

16 BB April 26, 2011 at 10:49 pm

Hello,

Just wondering why setting the following to “2” and not “1”?…

fs.suid_dumpable = 2

Reply

17 Naveen July 5, 2011 at 10:47 am

hi,

i deleted the latest core file by mistake. now the core file is not getting created and it seems some functions like NTP is not working.

How to solve this issue?

Reply

18 sachin July 13, 2011 at 1:31 pm

Anyone knows how to create Core Dump of entire System?

Reply

19 yaro March 4, 2012 at 1:43 am

Thanks for the information. Life becomes easy with people like you.

Reply

20 snapfla March 12, 2012 at 4:11 pm

Misleading title. This is not about debugging, much less, like a pro. It’s so frustrating when articles exist only to lure users that may click on ads.

Reply

21 moi August 28, 2012 at 7:36 pm

Do people really pay that much attention to titles? I came heree for the content, which was how to enable core dumps on linux. To that end, it worked like a charm.

Reply

22 Anthony Albert February 21, 2013 at 1:31 am

I’m returning to *nix programming after a long absence, and was trying to understand the use of DAEMON_COREFILE_LIMIT. What I haven’t seen mentioned is that this environment variable is used by the shell script function ‘daemon()’ which is included in the group of functions in etc/init.d/functions. If you start your daemon by calling daemon(), that variable will get used in a call to ‘ulimit’ by the shell. Otherwise, it will have no effect. I haven’t verified this yet (getting access to our servers is a slow process), but it seems that our custom shell script to start the daemon can do the same thing by calling ulimit directly.

Reply

23 xad February 21, 2013 at 7:25 pm

Thanks!

Reply

24 alex April 26, 2013 at 8:57 am

root@duty-05-ubuntu:/var# echo “DAEMON_COREFILE_LIMIT=’unlimited'” >> /etc/sysconfig/init
bash: /etc/sysconfig/init: No such file or directory
root@duty-05-ubuntu:/var#

Reply

25 freeman June 19, 2013 at 9:03 am

Article does not tell how to actually does what the title says, and sadly google shows this as first page for actual topic in title.

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Previous post:

Next post: