Find and Fix Linux Latency Problem with LatencyTOP Software

Posted on in Categories GNU/Open source, Howto, Linux, Linux desktop, Monitoring, Troubleshooting, Tuning last updated January 19, 2008

A TOP-like tool for monitoring system latency and its causes for Linux system.

The Intel Open Source Technology Center is pleased to announce the release of version 0.1 of LatencyTOP, a tool for developers to visualize system latencies. Skipping audio, slower servers, everyone knows the symptoms of latency. But to know what’s going on in the system, what’s causing the latency, how to fix it… that’s a hard question without good answers right now.

LatencyTOP is a Linux tool for software developers (both kernel and userspace), aimed at identifying where in the system latency is happening, and what kind of operation/action is causing the latency to happen so that the code can be changed to avoid the worst latency hiccups.
Linux Latency Problem with LatencyTOP Software
(Fig. 01: LatencyTOP in Action [ Image Credit: Intel Corp. ])

Download LatencyTOP

=> Visit official project site to download LatencyTOP software. Please note that you also need to patch Linux kernel.

rssh: Per User Configuration Options For Chroot Jail

Posted on in Categories Debian Linux, File system, FreeBSD, Howto, Linux, Networking, RedHat/Fedora Linux, Security, Suse Linux, Sys admin, Tuning, Ubuntu Linux, UNIX, User Management last updated December 22, 2007

rssh is a restricted shell for providing limited access to a host via ssh. It also allows system wide configuration and per user configuration. From the man page:
The user configuration directive allows for the configuration of options on a per-user basis. THIS KEYWORD OVERRIDES ALL OTHER KEYWORDS FOR THE SPECIFIED USER. That is, if you use a user keyword for user foo, then foo will use only the settings in that user line, and not any of the settings set with the keywords above. The user keyword’s argument consists of a group of fields separated by a colon (:), as shown below. The fields are, in order:

  • username : The username of the user for whom the entry provides options
  • umask : The umask for this user, in octal, just as it would be specified to the shell access bits. Five binary digits, which indicate whether the user is allowed to use rsync, rdist, cvs, sftp, and scp, in that order. One means the command is allowed, zero means it is not.
  • path : The directory to which this user should be chrooted (this is not a command, it is a directory name).

rssh examples of configuring per-user options

Open /etc/rssh.conf file:
# vi /etc/rssh.conf
All user tom to bypass our chroot jail:
user=tom:077:00010
Provide jerry cvs access with no chroot:
user=jerry:011:00100
Provide spike rsync access with no chroot:
user=spike:011:10000
Provide tyke access with chroot jail located at /users
user="tyke:011:00001:/users" # whole user string can be quoted
if your chroot_path contains spaces, it must be quoted. Provide nibbles scp access with chroot directory:
user=nibbles:011:00001:"/usr/local/tv/shows/tom and jerry"

Recommended readings:

=> rssh home page
=> Redhat specific chroot jail script (outdated)
=> Refer man pages: rssh.conf, rssh, ssh, sshd, sftp, scp, rsync, sshd_config

Print / Select a paper size other than A4 when using lp command line utility

Posted on in Categories CentOS, Debian Linux, Hardware, Howto, HP-UX, Linux, OpenBSD, Shell scripting, Suse Linux, Tuning, Ubuntu Linux, UNIX last updated December 21, 2007

Under Linux / UNIX you use lp command to print files from command prompt. lp is quite useful when GUI is not installed on Linux box to print files. The lp command is simply a front end command that calls the lpr command with appropriate options. Its main use is to allow the running of precompiled binary programs and scripts that assume that the lp command is the official printing command.

Changing papa size is very easy under GUI environment. But how do you change paper size under command prompt?
By default lp print to A4 paper size. However sometime you need to print to different paper size from command prompt such as A3 or A5. To print to A3 size, enter:
$ lp -o media=A3 /path/to/file
Where,

  • -o media=size : Sets the page size to size. Most printers support at least the size names “a4”, “letter”, and “legal”.

Other useful examples

Print a double-sided legal document to a printer called “hpljf2”:
$ lp -d hpljf2 -o media=legal -o sides=two-sided-long-edge /path/to/file
Print an image across 4 pages using a printer called “epd2”:
$ lp -d epd2 -o scaling=200 filename
Print a text file with 12 characters per inch, 8 lines per inch, and a 1 inch left margin to a printer called “lpodc2”:
$ lp -d lpodc2 -o cpi=12 -o lpi=8 -o page-left=72 ~/info.txt

To know more about lp option, enter:
$ man lp

Please note that you need to configure print using CUPS configuration file /etc/cups/cupsd.conf or web based tool located at http://localhost:631/
Linux / UNIX CUPS HP printer at http://localhost:631/
(Fig 01: My CUPS Configuration, showing HP PhotoSmart Printer )

How do I find out Linux Disk utilization?

Posted on in Categories CentOS, Debian Linux, File system, Hardware, Howto, Linux, Monitoring, RedHat/Fedora Linux, Storage, Tips, Troubleshooting, Tuning last updated December 17, 2007

I’ve already written about finding out Linux / UNIX cpu utilization using various tools. You can use same iostat command to find out disk utilization and for monitoring system input/output device loading by observing the time the physical disks are active in relation to their average transfer rates.

iostat syntax for disk utilization report

iostat -d -x interval count

  • -d : Display the device utilization report (d == disk)
  • -x : Display extended statistics including disk utilization
  • interval : It is time period in seconds between two samples . iostat 2 will give data at each 2 seconds interval.
  • count : It is the number of times the data is needed . iostat 2 5 will give data at 2 seconds interval 5 times

Display 3 reports of extended statistics at 5 second intervals for disk

Type the following command:
$ iostat -d -x 5 3
Output:

Linux 2.6.18-53.1.4.el5 (moon.nixcraft.in)   12/17/2007

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               1.10    39.82  3.41 13.59   309.50   427.48    43.36     0.17   10.03   1.03   1.75
sdb               0.20    18.32  1.15  6.08   117.36   195.25    43.22     0.51   71.14   1.26   0.91

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   108.40  1.40 64.40    49.60  1382.40    21.76     0.04    0.67   0.44   2.92
sdb               0.00    37.80  0.00 245.20     0.00  2254.40     9.19    28.91  108.49   1.08  26.36

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    97.01  1.00 57.29    39.92  1234.33    21.86     0.03    0.58   0.50   2.89
sdb               0.00    38.32  0.00 288.42     0.00  2623.55     9.10    32.97  122.30   1.15  33.27

Where,

  • rrqm/s : The number of read requests merged per second that were queued to the hard disk
  • wrqm/s : The number of write requests merged per second that were queued to the hard disk
  • r/s : The number of read requests per second
  • w/s : The number of write requests per second
  • rsec/s : The number of sectors read from the hard disk per second
  • wsec/s : The number of sectors written to the hard disk per second
  • avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
  • avgqu-sz : The average queue length of the requests that were issued to the device
  • await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
  • svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
  • %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

How do I interpret the output result for optimization?

First you need to note down following values from the iostat output:

  1. The average service time (svctm)
  2. Percentage of CPU time during which I/O requests were issued (%util)
  3. See if a hard disk reports consistently high reads/writes (r/s and w/s)

If any one of these are high, you need to take one of the following action:

  • Get high speed disk and controller for file system (for example move from SATA I to SAS 15k disk)
  • Tune software or application or kernel or file system for better disk utilization
  • Use RAID array to spread the file system

For example, from about iostat report it appears that /dev/sdb under load. Hope this information will help you diagnose and optimize disk related issues.

Related: How to find out Linux CPU utilization using vmstat, iostat, mpstat and sar commands.

Please note that command and information discussed here almost applies to any other UNIX like variant.

How To: Double Linux disk read performance with readahead parameter

Posted on in Categories CentOS, Debian Linux, File system, Hardware, Howto, Linux, RedHat/Fedora Linux, Storage, Suse Linux, Tuning last updated November 22, 2007

The open source journal has published an interesting hack. It mostly applies to high-end, multiple-disk storage:

Under the right conditions (that is, with certain hardware configurations which I’ll identify later) it is possible to literally double your sequential read performance from disk. If you noticed the terrible performance of the 3Ware 9500S RAID controller and cared enough to investigate. It all has to do with a sneaky little block device parameter known as readahead. Without going into too much gory detail, readahead controls how much in advance the operating system reads when, well, reading, as its name implies. By default, some operating systems (in particular, RHEL5 Server) sets this to 256 (512-byte sectors), or about 128 KB. When dealing with large filesystems spanning many disks, this paltry figure can actually nuke your performance.

=> HowTo: Linux: Double your disk read performance in a single command

Reboot Linux box after a kernel panic

Posted on in Categories CentOS, Debian Linux, Gentoo Linux, Hardware, Howto, Linux, RedHat/Fedora Linux, Tips, Troubleshooting, Tuning last updated November 16, 2007

If you want the server to get rebooted automatically after kernel hit by a pain error message, try adding panic=N to /etc/sysctl.conf file.

It specify kernel behavior on panic. By default, the kernel will not reboot after a panic, but this option will cause a kernel reboot after N seconds. For example following boot parameter will force to reboot Linux after 10 seconds.
Continue reading “Reboot Linux box after a kernel panic”

Linux Filter and block P2P network traffic such as Kazaa / Bittorrent with ipp2p

Posted on in Categories Howto, Linux, Networking, Security, Tips, Tuning last updated November 15, 2007

Well, personally I’m all set to freedom and open internet culture. However, in corporate and in an academic environment you will always find abuse smart users. Large and medium size corporate institutional networks suffer now a days from “smart” users who try to get their latest Movie/soft/Music/TVShow downloaded in their office.

Beside the moral/legal dispute these activities present the network admins with some troubles. To begin with a considerable downgrade in the network performance, and the need to comply with local policy and legal restrictions, and of course the admins needs to have full band with for they own downloads.

ipp2p is a reasonable stable product, I ‘ve use it for 2 years in a large network 4 class C networks in an university environment. Users were use to abuse the network for personal downloads, and after chasing and punishing them for some time we chose to block the traffic once and for all.

Read more at debian-administration: Filtering P2P network traffic with ipp2p.

On a related note we use application layer packet classifier for Linux called L7-filter:

L7-filter is a classifier for Linux’s Netfilter that identifies packets based on application layer data. It can classify packets as Kazaa, HTTP, Jabber, Citrix, Bittorrent, FTP, Gnucleus, eDonkey2000, etc., regardless of port. It complements existing classifiers that match on IP address, port numbers and so on.
Our intent is for l7-filter to be used in conjunction with Linux QoS to do bandwidth arbitration (“packet shaping”) or traffic accounting.

Also if user tunnel packets through SSL or uses encrypt them, none of these technique will work as software classify them as SSL, so your smart user still have a way out 😉 Trust me I’ve seen logs of largest broadband ISP in India and 60-80% traffic is p2p only.

Linux Increase Process Identifiers Limit with /proc/sys/kernel/pid_max

Posted on in Categories Howto, Linux, Linux Scalability, Networking, Troubleshooting, Tuning last updated November 3, 2007

Yesterday I wrote about increasing local port range with net.ipv4.ip_local_port_range proc file. There is also /proc/sys/kernel/pid_max file, which specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID). The default value for this file, 32768, results in the same range of PIDs as on earlier kernels (<=2.4). The value in this file can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million).