HowTo: Debug Crashed Linux Application Core Files Like A Pro

Posted on in Categories Linux, Troubleshooting last updated June 3, 2010

Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

Turn On Core File Creation Support

By default most Linux distributions turn off core file creation (at least this is true for RHEL, CentOS, Fedora and Suse Linux). You need to use the ulimit command to configure core files.

See The Current Core File Limits

Type the following command:
# ulimit -c
Sample outputs:


The output 0 (zero) means core file is not created.

Change Core File Limits

In this example, set the size limit of core files to 75000 bytes:
# ulimit -c 75000

HowTo: Enable Core File Dumps For Application Crashes And Segmentation Faults

Edit /etc/profile file and find line that read as follows to make persistent configuration:

ulimit -S -c 0 > /dev/null 2>&1

Update it as follows:

ulimit -c unlimited >/dev/null 2>&1

Save and close the file. Edit /etc/sysctl.conf, enter:
# vi /etc/sysctl.conf
Append the following lines:

kernel.core_uses_pid = 1
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t
fs.suid_dumpable = 2

Save and close the file. Where,

  1. kernel.core_uses_pid = 1 – Appends the coring processes PID to the core file name.
  2. fs.suid_dumpable = 2 – Make sure you get core dumps for setuid programs.
  3. kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t – When the application terminates abnormally, a core file should appear in the /tmp. The kernel.core_pattern sysctl controls exact location of core file. You can define the core file name with the following template whih can contain % specifiers which are substituted by the following values when a core file is created:
    • %% – A single % character
    • %p – PID of dumped process
    • %u – real UID of dumped process
    • %g – real GID of dumped process
    • %s – number of signal causing dump
    • %t – time of dump (seconds since 0:00h, 1 Jan 1970)
    • %h – hostname (same as ’nodename’ returned by uname(2))
    • %e – executable filename

Finally, enable debugging for all apps, enter (Redhat and friends specific):
# echo "DAEMON_COREFILE_LIMIT='unlimited'" >> /etc/sysconfig/init
Reload the settings in /etc/sysctl.conf by running the following command:
# sysctl -p

How Do I Enable Core Dumping For Specific Deamon?

To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/init.d/lighttped and add line as follows:


Please note that DAEMON_COREFILE_LIMIT is Redhat specific, for all other distro add configuration as follows:

ulimit -c unlimited >/dev/null 2>&1
echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern

Save and close the file. Restart / reload lighttpd:
# /etc/init.d/lighttpd restart
# su - lighttpd
$ ulimit -c

Sample outputs:


Now, you can send core files to vendor or software writes.

How Do I Read Core Files?

You need use the gdb command as follows:
$ gdb /path/to/application /path/to/corefile
See the gdb command man page for more information.

strace command

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers. See how to use the strace command under Linux to debug the problems.

Recommended readings:

Stay tunned for gdb tutorial which will explains how to use generated core file to track down problem.

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

29 comment

  1. Nice article, just maybe make the subject line relevant to the article…

    “How to enable core dumps” instead of “Debug apps”…esp since there is virtually nothing about debugging in this article.

  2. Please, explain more clear wich line add into /etc/sysconfig/daemon-file file, and which line into /etc/init.d/lighttped file; in the part about Specific Deamon?

  3. I’m returning to *nix programming after a long absence, and was trying to understand the use of DAEMON_COREFILE_LIMIT. What I haven’t seen mentioned is that this environment variable is used by the shell script function ‘daemon()’ which is included in the group of functions in etc/init.d/functions. If you start your daemon by calling daemon(), that variable will get used in a call to ‘ulimit’ by the shell. Otherwise, it will have no effect. I haven’t verified this yet (getting access to our servers is a slow process), but it seems that our custom shell script to start the daemon can do the same thing by calling ulimit directly.

  4. [email protected]:/var# echo “DAEMON_COREFILE_LIMIT=’unlimited'” >> /etc/sysconfig/init
    bash: /etc/sysconfig/init: No such file or directory
    [email protected]:/var#

  5. You have an error in your write up. Please correct.

    You wrote, “To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/init.d/lighttped and add line as follows:”

    It should read instead, “To enable core dumping for specific deamons, add the following line in the /etc/sysconfig/daemon-file file. In this example, edit /etc/sysconfig/lighttped and add line as follows:”

    The init.d script is used for startup/shutdown. The sysconfig file is read by the init script to define the variables. So it should go in /etc/sysconfig/lighttped not /etc/init.d/lighttped

  6. No offence meant here but:
    Besides the fact you only show how to enable coredumps I’m sorry to say that anyone who doesn’t know how to do so will not become a ‘pro’ any time soon; debugging is not science but an art and it’s an art that takes time, patience and practise (this includes programming but also debugging combined). This esp goes if the stack is corrupted! Good luck to the non-programmer there (and I really don’t see how non-programmers are going to become masters at said art anyway. It’s not realistic)! As for suid binaries and coredumps? It’s opening a can of worms security wise even if you know what you’re doing and frankly someone who doesn’t know how to enable coredumping at all shouldn’t be meddling with this esp.
    And finally: storing it in /tmp is not only a bad idea security wise (keep in mind it is a capture of the entire process memory and all!) but also a silly location to store them anyway; /tmp is not meant for debugging purposes (though I know many won’t see the problem). Frankly as a programmer I hate when system configurations capture core dumps (even though I seldom have such issues if e.g. I deliberately add a kill() or signal() or use the ‘kill’ command now I don’t have control of the core location) like that (looking at the idiocy of abrtd in particular though thankfully I am in control of all systems that this matters or otherwise the administrator of the other systems know these things anyway and have no need for such silly software/settings).

Leave a Comment