≡ Menu


Linux Commands For Shared Library Management & Debugging Problem

If you are a developer, you will re-use code provided by others. Usually /lib, /lib64, /usr/local/lib, and other directories stores various shared libraries. You can write your own program using these shared libraries. As a sys admin you need to manage and install these shared libraries. Use the following commands for shared libraries management, security, and debugging problems.
[click to continue…]

Linux Configure Netconsole To Log Messages Over UDP Network

Linux can be configured to log dmesg output to another system via network using syslog. It is done using kernel level networking stuff ia UDP port 514. There is module called netconsole which logs kernel printk messages over udp allowing debugging of problem where disk logging fails and serial consoles are impractical. Most modern distro has this netconsole as a built-in module. netconsole initializes immediately after NIC cards. There are two steps to configure netconsole:

  • Syslogd server - Let us assume IP having FQDN - syslogd.nixcraft.in. Please note that the remote host can run either 'netcat -u -l -p <port>' or syslogd.
  • All other systems running netconsole module in kernel

Step # 1: Configure Centralized syslogd

Login to syslogd.nixcraft.in server. Open syslogd configuration file. Different UNIX / Linux variant have different configuration files

Red Hat / CentOS / Fedora Linux Configuration

If you are using Red Hat / CentOS / Fedora Linux open /etc/sysconfig/syslog file and set SYSLOGD_OPTIONS option for udp logging.
# vi /etc/sysconfig/syslog
Configure syslogd option as follows:
SYSLOGD_OPTIONS="-m 0 -r -x"
Save and close the file. Restart syslogd, enter:
# service syslog restart

Debian / Ubuntu Linux Configuration

If you are using Debian / Ubuntu Linux open file /etc/default/syslogd set SYSLOGD option for udp logging.
# vi /etc/default/syslogd
Configure syslogd option as follows:
# /etc/init.d/sysklogd restart

FreeBSD configuration

If you are using FreeBSD open /etc/rc.conf and set syslogd_flags option option for udp logging. Please note that FreeBSD by default accepts network connections. Please refer to syslogd man page for more information.

Firewall configuration

You may need to open UDP port 514 to allow network login. Sample iptables rules to open UDP port 514:
iptables -A INPUT -p udp -s $MYNET --sport 1024:65535 -d $SLSERVER --dport 514 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p udp -s $SLSERVER --sport 514 -d $MYNET --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT

Step # 2: Configure Linux Netconsole

You need to configure netconsole service. Once this service started, you are allowed a remote syslog daemon to record console output from local system. The local port number that the netconsole module will use 6666 (default). You need to set the IP address of the remote syslog server to send messages.

Open /etc/sysconfig/netconsole file under CentOS / RHEL / Fedora Linux, enter:
# vi /etc/sysconfig/netconsole
Set SYSLOGADDR to (IP address of remote syslog server)
Save and close the file. Restart netconsole service, enter:
# /etc/init.d/netconsole restart

A note about Debian / Ubuntu Linux

Red Hat has netconsole init script. However, under Debian / Ubuntu Linux, you need to manually configure netconsole. Type the following command to start netconsole by loading kernel netconsole module, enter:
# modprobe netconsole 6666@,514@

  • 6666 - Local port
  • - Local system IP
  • eth0 - Local system interface
  • 514 - Remote syslogd udp port
  • - Remote syslogd IP
  • 00:19:D1:2A:BA:A8 - Remote syslogd Mac

You can add above modprobe line to /etc/rc.local to load module automatically. Another recommend option is create /etc/modprobe.d/netconsole file and append following text:
# echo 'options netconsole netconsole=6666@,514@ '> /etc/modprobe.d/netconsole

How do I verify netconsole is logging messages over UDP network?

Login to remote syslog udp server (i.e. our sample syslogd system), enter:
# tail -f /var/log/messages
/var/log/messages is default log file under many distributions to log messages. Refer to /etc/syslog.conf for exact location of your file.

How do I use nc / netcat instead of messing with syslogd?

This is called one minute configuration. You can easily get output on without using syslogd. All you have to do is run netcat (nc) command, on
$ nc -l -p 30000 -u
Login to any other box, enter command:
# modprobe netconsole 6666@,30000@
Output should start to appear on from without configuring syslogd or anything else.

Further readings:

valgrind – Linux Tools For Debugging And Profiling Programs ( bug reporting tool )

Few days back I wrote about strace tool for reporting and finding bug in program. Today I'm going to talk about another interesting tool called valgrind.

Valgrind is a flexible program for debugging and profiling Linux executables. It consists of a core, which provides a synthetic CPU in software, and a series of "tools", each of which is a debugging or profiling tool. The architecture is modular, so that new tools can be created easily and without disturbing the existing structure. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

The Valgrind distribution currently includes five production-quality tools:

  • a memory error detector
  • a thread error detector
  • a cache and branch-prediction profiler
  • a call-graph generating cache profiler
  • a heap profiler

It also includes two experimental tools:

  • a data race detector
  • an instant memory leak detector.

It runs on the following platforms:

  • X86/Linux
  • AMD64/Linux
  • PPC32/Linux
  • PPC64/Linux

How do I use valgrind?

Valgrind is typically run as follows:
$ valgrind command-name arg1 arg2 argN
$ valgrind program args
$ valgrind ./myapp -d /tmp -f 120

You can select tool using the --tool=TOOLName option. For example use memcheck which is a fine-grained memory checker. To generate trace back for command called myapp, enter:
$ valgrind --tool=memcheck -v --log-file=myapp.dbg --num-callers=8 ./myapp -d /tmp -f 120

  • --tool=memcheck : Run the Valgrind tool called memcheck
  • -v : Verbose output
  • --log-file=myapp.dbg : Specifies that Valgrind should send all of its messages to the specified file.
  • --num-callers=8 : By default, Valgrind shows twelve levels of function call names to help you identify program locations. You can change that number with this option. This can help in determining the program’s location in deeply-nested call chains.

The --leak-check option turns on the detailed memory leak detector:
$ valgrind --tool=memcheck -v --log-file=myapp.dbg --num-callers=8 --leak-check=yes ./myapp -d /tmp -f 120

Further readings:

Find and Fix Linux Latency Problem with LatencyTOP Software

A TOP-like tool for monitoring system latency and its causes for Linux system.

The Intel Open Source Technology Center is pleased to announce the release of version 0.1 of LatencyTOP, a tool for developers to visualize system latencies. Skipping audio, slower servers, everyone knows the symptoms of latency. But to know what's going on in the system, what's causing the latency, how to fix it... that's a hard question without good answers right now.

LatencyTOP is a Linux tool for software developers (both kernel and userspace), aimed at identifying where in the system latency is happening, and what kind of operation/action is causing the latency to happen so that the code can be changed to avoid the worst latency hiccups.
Linux Latency Problem with LatencyTOP Software
(Fig. 01: LatencyTOP in Action [ Image Credit: Intel Corp. ])

Download LatencyTOP

=> Visit official project site to download LatencyTOP software. Please note that you also need to patch Linux kernel.

Tips for Emacs Power Users

This is great post by Stevey Drunken about mastering Emacs text editor which is quite quite popular among UNIX hackers, computer programmers and power users:

Emacs is the world's best text editor. It's not just the best for editing program source; it's the best for any kind of text-editing. Mastering Emacs will make you more effective at writing and editing email, documentation drafts, blogs, HTML pages, XML files, and virtually everything else that requires any typing.

The tips in this little document are geared towards Emacs power-users. You should be familiar with the basics of launching and editing with Emacs, and you should already know the essentials of copying stuff into your .emacs file, and debugging things (or finding a friendly Emacs Wizard) when something goes wrong.

=> 10 Specific Ways to Improve Your Productivity With Emacs