HowTo: Debug Crashed Linux Application Core Files Like A Pro

Posted on in Categories Linux, Troubleshooting last updated June 18, 2010

Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.

Linux HugeTLBfs: Improve MySQL Database Application Performance

Posted on in Categories CentOS, Hardware, High performance computing, Howto, MySQL, RedHat/Fedora Linux last updated May 20, 2009

Applications that perform a lot of memory accesses (several GBs) may obtain performance improvements by using large pages due to reduced Translation Lookaside Buffer (TLB) misses. HugeTLBfs is memory management feature offered in Linux kernel, which is valuable for applications that use a large virtual address space. It is especially useful for database applications such as MySQL, Oracle and others. Other server software(s) that uses the prefork or similar (e.g. Apache web server) model will also benefit.

The CPU’s Translation Lookaside Buffer (TLB) is a small cache used for storing virtual-to-physical mapping information. By using the TLB, a translation can be performed without referencing the in-memory page table entry that maps the virtual address. However, to keep translations as fast as possible, the TLB is usually small. It is not uncommon for large memory applications to exceed the mapping capacity of the TLB. Users can use the huge page support in Linux kernel by either using the mmap system call or standard SYSv shared memory system calls (shmget, shmat).

Linux Increase Process Identifiers Limit with /proc/sys/kernel/pid_max

Posted on in Categories Howto, Linux, Linux Scalability, Networking, Troubleshooting, Tuning last updated April 16, 2014

Yesterday I wrote about increasing local port range with net.ipv4.ip_local_port_range proc file. There is also /proc/sys/kernel/pid_max file, which specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID). The default value for this file, 32768, results in the same range of PIDs as on earlier kernels (<=2.4). The value in this file can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million).

Howto Reboot or halt Linux system in emergency

Posted on in Categories CentOS, Debian Linux, Gentoo Linux, Howto, Linux, Linux distribution, Networking, RedHat/Fedora Linux, Security, Suse Linux, Sys admin, Tips, Troubleshooting, Ubuntu Linux last updated February 20, 2008

Linux kernel includes magic system request keys. It was originally developed for kernel hackers. However, you can use this hack to reboot, shutdown or halt computer safely (remember safe reboot/shutdown == flush filesystem buffers and unmount file system and then reboot so that data loss can be avoided).

This is quite useful when Linux based system is not available after boot or after a X server crashed ( svgalib program crashes) or no display on screen. Sysrq key combo forces the kernel to respond it regardless of whatever else it is doing, unless it is completely locked up (dead).

Using further extension to iptables called ipt_sysrq (new iptables target), which allows you to do the same as the magic sysrq key on a keyboard does, but over the network. So if your network server is not responding you can still reboot it. Please note that Magic SysRq support need to be compiled in your kernel. You need to say “yes” to ‘Magic SysRq key (CONFIG_MAGIC_SYSRQ)’ when configuring the kernel. I’m assuming that you have Magic SysRq key’ support is compiled in your kernel.

Enable sysrq keys

By default it is not enabled on many Linux distributions. Add or modify following line (as soon as new Linux system installed) /etc/sysctl.conf:
# vi /etc/sysctl.conf
Append following config directive:
kernel.sysrq=1
Save and close the file. Reload settings:
# sysctl -p

Save and close the file and reboot system to take effect

How do I use the magic SysRq keys in emergency?

You need to use following key combination in order to reboot/halt/sync file system etc:
ALT+SysRq+COMMAND-KEY

The ‘SysRq’ key is also known as the ‘Print Screen’ key. COMMAND-KEY can be any one of the following (all keys need to hit simultaneously) :

  • ‘b’ : Will immediately reboot the system without syncing or unmounting your disks.
  • ‘o’ : Will shutdown your system off (if configured and supported).
  • ‘s’: Will attempt to sync all mounted filesystems.
  • ‘u’ : Will attempt to remount all mounted filesystems read-only.
  • ‘e’ : Send a SIGTERM to all processes, except for init.
  • ‘h’: Show help, indeed this the one you need to remember.

So whey you need to tell your Linux computer to reboot or when your X server is crashed or you don’t see anything going across the screen then just press:

ALT+SysRQ+s : (Press and hold down ALT, then SysRQ (Print Screen) key and press ‘s’) -Will try to syn all mounted system

ALT+SysRQ+r : (Press and hold down ALT, then SysRQ (Print Screen) key and press ‘r’) -Will reboot the system.

If you wish to shutdown the system instead of reboot then press following key combination:
ALT+SysRQ+o

ipt_sysrq is a new iptables target that allows you to do the same as the magic sysrq key on a keyboard does, but over the network. Sometimes a remote server hangs and only responds to icmp echo request (ping). Every administrator of such machine is very unhappy because (s)he must go there and press the reset button. It takes a long time and it’s inconvenient. So use the Network Magic SysRq and you will be able to do more than just pressing a reset button. You can remotely sync disks, remount them read-only, then do a reboot. And everything comfortably and only in a few seconds. Please see Marek Zelem page to enableIP Tables network magic SysRq function.

For more information read official Documentation for sysrq.c version 1.15 stored in /usr/src/linux/Documentation/sysrq.txt and read man page of sysctl, sysctl.conf.

How to setup Linux as a router for DSL, T1 line etc

Posted on in Categories Howto, Iptables, Linux, Networking, RedHat/Fedora Linux, Tips, Ubuntu Linux last updated November 29, 2007

There are a few ways to set up a Linux machine as route. Here is a relatively straight forward and common method. This method requires that the system use iptables for Network Address Translation (NAT).

This step by step small howto will help you to setup Linux router only in 2 minutes.

Configuration steps

=> First enable packet forwarding
=> Next setup Network Address Translation using IPTABLES MASQUERADE targets
=> Save the changes
=> Verify everything is working

I’m assuming that your setup is as follows:
A) You are using any Linux distro

B) eth0 is internet interface (connected to router for example) and eth1 connected to your internal lan (connected to your HUB/Switch for example).

My Linux   eth0  --> Internet
box       eth1  --> Lan

Step # 1 Turn on ip forwarding in kernel

1) Open linux kernel configuration file (you must be a root user or use su – command to become a root user):
# vi /etc/sysctl.conf

2) Add/modify following line:
net.ipv4.ip_forward = 1

Step # 2 Restart network
# /etc/init.d/network restartOR# service network restart

Step # 3 Setup IP forwarding and Masquerading (to act as router), you need to use NAT option of iptables as follows (add following rules to your iptables shell script) :
# iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
# iptables --append FORWARD --in-interface eth1 -j ACCEPT

Step # 4 You are done! Test it with ping or dig:
# ping your-isp.com
# dig yahoo.com

Step # 5 Point all desktop client to your eth1 IP address as Router/Gateway. Or use DHCP to distribute this information (recommended)

Step # 6 Put code described in step # 3 to script and call it from /etc/rc.local file.