How To Avoid Sudden Outburst Of Backup Shell Script or Program Disk I/O on Linux

Posted on in Categories High performance computing, kernel, Linux, Storage, Troubleshooting, Tuning last updated June 3, 2008

A sudden outburst of violent disk I/O activity can bring down your email or web server. Usually, a web, mysql, or mail server serving millions and millions pages (requests) per months are prone to this kind of problem. Backup activity can increase current system load too. To avoid this kind of sudden outburst problem, run your script with scheduling class and priority. Linux comes with various utilities to manage this kind of madness.

How To Setup OpenSolaris Open Storage Server

Posted on in Categories Solaris, Storage last updated May 1, 2008

Sun Microsystems announced that it will provide new software tools and expanded professional support to assist developers in building open-source storage solutions. With the new resources, Sun estimates the average developer will be able to set up an OpenSolaris server is about 10 minutes.

Load Balancer Open Source Software

Posted on in Categories Hardware, High performance computing, Linux, Linux Scalability, Linux Virtualization, Monitoring, Networking, RedHat/Fedora Linux, Storage last updated March 11, 2008

I’ve worked with a various load balancing systems (LBS). They are complex pieces of hardware and software. In this post I will highlight some of the open source load balancing software. But what is load balancing?
It is nothing but a technique used to share (spared) load / services between two or more servers. For example, busy e-commerce or bank website uses load balancer to increase reliability, throughput, uptime, response time and better resource utilization. You can use following softwares as an advanced load balancing solution for web, cache, dns, mail, ftp, auth servers, VoIP services etc.

Linux Virtual Server (LVS)

LVS is ultimate open source Linux load sharing and balancing software. You can easily build a high-performance and highly available server for Linux using this software. From the project page:

Virtual server is a highly scalable and highly available server built on a cluster of real servers. The architecture of server cluster is fully transparent to end users, and the users interact with the cluster system as if it were only a single high-performance virtual server.

=> Project Web Site

Red Hat Cluster Suite

It is a high availability cluster software implementation from Linux leader Red Hat. It provide two type services:

  1. Application / Service Failover – Create n-node server clusters for failover of key applications and services
  2. IP Load Balancing – Load balance incoming IP network requests across a farm of servers

=> Product web page

The High Availability Linux Project

Linux-HA provides sophisticated high-availability (failover) capabilities on a wide range of platforms, supporting several tens of thousands of mission critical sites.

=> Project web site

Ultra Monkey

Ultra Monkey is a project to create load balanced and highly available network services. For example a cluster of web servers that appear as a single web server to end-users. The service may be for end-users across the world connected via the internet, or for enterprise users connected via an intranet.

Ultra Monkey makes use of the Linux operating system to provide a flexible solution that can be tailored to a wide range of needs. From small clusters of only two nodes to large systems serving thousands of connections per second.

=> Project web site

Personally, I’ve worked with both LVS and Red Hat Cluster Suite and I highly recommend these softwares.

Building a Home File Server with FreeNAS ( FreeBSD based NAS Server )

Posted on in Categories File system, FreeBSD, Hardware, Howto, Linux desktop, OS X, Storage, UNIX, Windows last updated January 18, 2008

FreeNAS is a free Network-attached storage (NAS) server. It supports following protocols:

=> CIFS (Samba) – Mac / UNIX / Windows can use CIFS to store files
=> FTP
=> NFS
=> Rsync
=> AFP

You can use local user authentication, and software RAID (0,1,5), with a web-based configuration interface.

I’ve installed it on USB pen which takes less than 32 MB. You can install software on hard disk or flash drive.

My Setup

  • Intel 1.7Ghz Celeron
  • 1GB SDRam
  • 3 x 80GB IDE Hard disk
  • 10/100/1000 Mbps Lan Card
  • Raid 5 with FreeNAS (Highest Read data transaction as I need to read lots of files such as MP3 / Photos and other data)
  • BIOS is configured to boot from USB pen and w/o monitor / keyboard
  • D-Link Gigabit Ethernet 8 Port Switche and Wireless AP router

Building a Home File Server with FreeNAS Software - FreeNAS WEB User Interface
(Fig. 01: FreeNAS Web User Interface Configuration Tool)

The software is easy to use and perfect for building a low cost home file server. I’m currently using this NAS as:
=> Home Music / Media Server
=> Home Backup Server
=> Home Ftp Server etc

Download FreeNAS server

=> Visit official site to grab latest software.

Linux Fibre Channel over Ethernet implementation code released

Posted on in Categories Linux, Linux Scalability, Linux Virtualization, Networking, Open source coding, Storage last updated December 18, 2007

Intel has just released source code for Fibre Channel over Ethernet (FCoE). It provides some Fibre Channel protocol processing as well as the encapsulation of FC frames within Ethernet packets. FCoE will allow systems with an Ethernet adapter and a Fibre Channel Forwarder to login to a Fibre Channel fabric (the FCF is a “gateway” that bridges the LAN and the SAN). That fabric login was previously reserved exclusively for Fibre Channel HBAs. This technology reduces complexity in the data center by aiding network convergence. It is targeted for 10Gps Ethernet NICs but will work on any Ethernet NIC supporting pause frames. Intel will provide a Fibre Channel protocol processing module as well as an Ethernet based transport module. The Open-FC module acts as a LLD for SCSI and the Open-FCoE transport uses net_device to send and receive packets.

This is good news. I think one can compare bandwidth and throughput for copper and fiber Ethernet. If you are going to use copper you need to stay within 15m of the switch. This solution will try to bring down cost. One can connect to 8-10 server to central database server with 10G and there could be few more applications.

=> Open FCoE project home page

How do I find out Linux Disk utilization?

Posted on in Categories CentOS, Debian Linux, File system, Hardware, Howto, Linux, Monitoring, RedHat/Fedora Linux, Storage, Tips, Troubleshooting, Tuning last updated December 17, 2007

I’ve already written about finding out Linux / UNIX cpu utilization using various tools. You can use same iostat command to find out disk utilization and for monitoring system input/output device loading by observing the time the physical disks are active in relation to their average transfer rates.

iostat syntax for disk utilization report

iostat -d -x interval count

  • -d : Display the device utilization report (d == disk)
  • -x : Display extended statistics including disk utilization
  • interval : It is time period in seconds between two samples . iostat 2 will give data at each 2 seconds interval.
  • count : It is the number of times the data is needed . iostat 2 5 will give data at 2 seconds interval 5 times

Display 3 reports of extended statistics at 5 second intervals for disk

Type the following command:
$ iostat -d -x 5 3
Output:

Linux 2.6.18-53.1.4.el5 (moon.nixcraft.in)   12/17/2007

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               1.10    39.82  3.41 13.59   309.50   427.48    43.36     0.17   10.03   1.03   1.75
sdb               0.20    18.32  1.15  6.08   117.36   195.25    43.22     0.51   71.14   1.26   0.91

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00   108.40  1.40 64.40    49.60  1382.40    21.76     0.04    0.67   0.44   2.92
sdb               0.00    37.80  0.00 245.20     0.00  2254.40     9.19    28.91  108.49   1.08  26.36

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00    97.01  1.00 57.29    39.92  1234.33    21.86     0.03    0.58   0.50   2.89
sdb               0.00    38.32  0.00 288.42     0.00  2623.55     9.10    32.97  122.30   1.15  33.27

Where,

  • rrqm/s : The number of read requests merged per second that were queued to the hard disk
  • wrqm/s : The number of write requests merged per second that were queued to the hard disk
  • r/s : The number of read requests per second
  • w/s : The number of write requests per second
  • rsec/s : The number of sectors read from the hard disk per second
  • wsec/s : The number of sectors written to the hard disk per second
  • avgrq-sz : The average size (in sectors) of the requests that were issued to the device.
  • avgqu-sz : The average queue length of the requests that were issued to the device
  • await : The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
  • svctm : The average service time (in milliseconds) for I/O requests that were issued to the device
  • %util : Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%.

How do I interpret the output result for optimization?

First you need to note down following values from the iostat output:

  1. The average service time (svctm)
  2. Percentage of CPU time during which I/O requests were issued (%util)
  3. See if a hard disk reports consistently high reads/writes (r/s and w/s)

If any one of these are high, you need to take one of the following action:

  • Get high speed disk and controller for file system (for example move from SATA I to SAS 15k disk)
  • Tune software or application or kernel or file system for better disk utilization
  • Use RAID array to spread the file system

For example, from about iostat report it appears that /dev/sdb under load. Hope this information will help you diagnose and optimize disk related issues.

Related: How to find out Linux CPU utilization using vmstat, iostat, mpstat and sar commands.

Please note that command and information discussed here almost applies to any other UNIX like variant.

CentOS 5 Apache 2.2.3 files failing to download or corrupted download file issue

Posted on in Categories Apache, CentOS, File system, lighttpd, Linux, Storage, Tips, Troubleshooting last updated December 1, 2007

Recently, I noticed something strange about Apache 2.2.3 version running on CentOS Linux 5 64 bit version. We have centralized NFS server and all 3 web server load balanced using hardware front end (another box running LVS).

All Apache server picks up file via NFS i.e DocumentRoot is set over NFS. The small file such as 2 MB or 5 MB get downloaded correctly but large size files failed to download. Another problem was some clients reported that the file get download but cannot open due to file corruption issue.

After investigation and a little bit googling I came across the solution. You need to disable following two options:

  • EnableMMAP – This directive controls whether the httpd may use memory-mapping if it needs to read the contents of a file during delivery. By default, when the handling of a request requires access to the data within a file — for example, when delivering a server-parsed file using mod_include — Apache memory-maps the file if the OS supports it.
  • EnableSendfile – This directive controls whether httpd may use the sendfile support from the kernel to transmit file contents to the client. By default, when the handling of a request requires no access to the data within a file — for example, when delivering a static file — Apache uses sendfile to deliver the file contents without ever reading the file if the OS supports it.

However, these two directives are known to have problem with a network-mounted DocumentRoot (e.g., NFS or SMB), the kernel may be unable to serve the network file through its own cache. So just open httpd.conf on all boxes and changes the following:
EnableMMAP off
EnableSendfile off

Just restart the web server and voila!
# service httpd restart