HowTo: Debug Crashed Linux Application Core Files Like A Pro

Posted on in Categories Linux, Troubleshooting last updated June 18, 2010

Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

Why Does The Segmentation Fault Occur on Linux / UNIX Systems?

Posted on in Categories C Programming, CentOS, Debian Linux, fedora linux, GNU/Open source, Hardware, Linux, Linux distribution, RedHat/Fedora Linux, Sys admin, Troubleshooting, Ubuntu Linux, UNIX last updated October 5, 2008

An error in which a running program attempts to access memory not allocated to it and core dumps with a segmentation violation error. Here are few tips to track down “Segmentation Fault” error under UNIX / Linux.

Debugging Tip: Trace the Process and See What It is Doing with strace

Posted on in Categories Linux, Tips, Troubleshooting last updated April 15, 2008

strace is a useful diagnostic, instructional, and debugging tool. It can save lots of headache. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. This is also useful to submit bug reports to open source developers.

Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value.

Run strace against /bin/foo and capture its output to a text file in output.txt:
$ strace -o output.txt /bin/foo
You can strace the webserver process and see what it’s doing. For example, strace php5 fastcgi process, enter:
$ strace -p 22254 -s 80 -o /tmp/debug.lighttpd.txt
To see only a trace of the open, read system calls, enter :
$ strace -e trace=open,read -p 22254 -s 80 -o debug.webserver.txt
Where,

  • -o filename : Write the trace output to the file filename rather than to screen (stderr).
  • -p PID : Attach to the process with the process ID pid and begin tracing. The trace may be terminated at any time by a keyboard interrupt signal (hit CTRL-C). strace will respond by detaching itself from the traced process(es) leaving it (them) to continue running. Multiple -p options can be used to attach to up to 32 processes in addition to command (which is optional if at least one -p option is given).
  • -s SIZE : Specify the maximum string size to print (the default is 32).

Refer to strace man page for more information:
$ man strace