≡ Menu

segmentation fault

Test and Troubleshoot Chrooted Apache Jail

This is 3rd and the final installment for Apache Chroot Jail for CentOS / RHEL series. Once Apache is configured with mod_chroot, you may need to test and debug problems. This article will provide a few troubleshooting tips.
[click to continue…]

It appears that latest php version 5.1.6-20.el5_2.1 under RHEL / CentOS Linux v5.2 has made some major changes. As a result choort jail setup using previous instructions no longer works.

PHP is crashing with segmentation fault errors. So I had to trace php errors using strace command. After spending couple of hours I found solution for following errors:

Sep 15 03:26:59 lightyproxy kernel: php-cgi[19106]: segfault at 0000003151c1b4b8 rip 0000003151e98477 rsp 00007fff9ecdde20 error 6
Sep 15 03:26:59 lightyproxy kernel: php-cgi[19107]: segfault at 0000003151c1b4b8 rip 0000003151e98477 rsp 00007fff9ecdde20 error 6
Sep 15 03:26:59 lightyproxy kernel: php-cgi[19108]: segfault at 0000003151c1b4b8 rip 0000003151e98477 rsp 00007fff9ecdde20 error 6
Sep 15 03:26:59 lightyproxy kernel: php-cgi[19110]: segfault at 0000003151c1b4b8 rip 0000003151e98477 rsp 00007fff9ecdde20 error 6
WARNING! These examples / workaround is only for RHEL / CentOS 5.2 and not for Debian / Ubuntu / FreeBSD lighttpd chroot instructions.

You need to copy entire /etc/ and /usr/share/zoneinfo/ to jail. If your jail is located at /jail directory enter following commands:
# service lighttpd stop
# D=/path/to/chroot/jail
# mkdir /root/jail.etc
# /bin/cp -avr $D/etc/* /root/jail.etc
# /bin/cp -avr /etc/* $D/etc/

Copy back original customized files such as passwd, group, php.ini :
# cp -avr /root/jail.etc/* $D/etc/
Now copy /usr/share/zoneinfo/:
# cd $D/usr/share
# cp -avr /usr/share/zoneinfo/ .

Copy all latest php-cgi and all extensions to $D
# cd $D/usr/bin
# cp /usr/bin/php-cgi .
# l2chroot php-cgi

Copy php modules (for 64 bit use $D/usr/lib64):
# cd $D/usr/lib/
# cp -avr /usr/lib/php/ .
# cd php/modules
# for l in *.so; do l2chroot $l; done

Start lighttpd:
# service lighttpd start
This should fix all errors. Watch /var/log/messages for php errors:
# tail -f /var/log/messages

According to wikipedia:

A segmentation fault occurs when a program attempts to access a memory location that it is not allowed to access, or attempts to access a memory location in a way that is not allowed (for example, attempting to write to a read-only location, or to overwrite part of the operating system).

Usually signal #11 (SIGSEGV) set, which is defined in the header file signal.h file. The default action for a program upon receiving SIGSEGV is abnormal termination. This action will end the process, but may generate a core file (also known as core dump) to aid debugging, or perform some other platform-dependent action. A core dump is the recorded state of the working memory of a computer program at a specific time, generally when the program has terminated abnormally.

Segmentation fault can also occur under following circumstances:

a) A buggy program / command, which can be only fixed by applying patch.

b) It can also appear when you try to access an array beyond the end of an array under C programming.

c) Inside a chrooted jail this can occur when critical shared libs, config file or /dev/ entry missing.

d) Sometime hardware or faulty memory or driver can also create problem.

e) Maintain suggested environment for all computer equipment (overheating can also generate this problem).

Suggestions to debug Segmentation Fault errors

To debug this kind of error try one or all of the following techniques :

  • Use gdb to track exact source of problem.
  • Make sure correct hardware installed and configured.
  • Always apply all patches and use updated system.
  • Make sure all dependencies installed inside jail.
  • Turn on core dumping for supported services such as Apache.
  • Use strace which is a useful diagnostic, instructional, and debugging tool.
  • Google and find out if there is a solution to problem.
  • Fix your C program for logical errors such as pointer, null pointer, arrays and so on.
  • Analyze core dump file generated by your system using gdb

Further readings:

Please add your suggestions and debugging techniques in the comment below.

Few days back I wrote about strace tool for reporting and finding bug in program. Today I'm going to talk about another interesting tool called valgrind.

Valgrind is a flexible program for debugging and profiling Linux executables. It consists of a core, which provides a synthetic CPU in software, and a series of "tools", each of which is a debugging or profiling tool. The architecture is modular, so that new tools can be created easily and without disturbing the existing structure. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

The Valgrind distribution currently includes five production-quality tools:

  • a memory error detector
  • a thread error detector
  • a cache and branch-prediction profiler
  • a call-graph generating cache profiler
  • a heap profiler

It also includes two experimental tools:

  • a data race detector
  • an instant memory leak detector.

It runs on the following platforms:

  • X86/Linux
  • AMD64/Linux
  • PPC32/Linux
  • PPC64/Linux

How do I use valgrind?

Valgrind is typically run as follows:
$ valgrind command-name arg1 arg2 argN
$ valgrind program args
$ valgrind ./myapp -d /tmp -f 120

You can select tool using the --tool=TOOLName option. For example use memcheck which is a fine-grained memory checker. To generate trace back for command called myapp, enter:
$ valgrind --tool=memcheck -v --log-file=myapp.dbg --num-callers=8 ./myapp -d /tmp -f 120

  • --tool=memcheck : Run the Valgrind tool called memcheck
  • -v : Verbose output
  • --log-file=myapp.dbg : Specifies that Valgrind should send all of its messages to the specified file.
  • --num-callers=8 : By default, Valgrind shows twelve levels of function call names to help you identify program locations. You can change that number with this option. This can help in determining the program’s location in deeply-nested call chains.

The --leak-check option turns on the detailed memory leak detector:
$ valgrind --tool=memcheck -v --log-file=myapp.dbg --num-callers=8 --leak-check=yes ./myapp -d /tmp -f 120

Further readings: