Red Hat / CentOS Linux 4: Setup Device Mapper Multipathing

Posted on in Categories CentOS, data center, fedora linux, File system, Hardware, High performance computing, Howto, kernel, Linux, Linux Scalability, Linux Virtualization, Networking, RedHat/Fedora Linux, Tips last updated July 6, 2008

Multipath I/O is a fault-tolerance and performance enhancement technique whereby there is more than one physical path between the CPU in a computer system and its mass storage devices through the buses, controllers, switches, and bridge devices connecting them.

A simple example would be a SCSI disk connected to two SCSI controllers on the same computer or a disk connected to two Fibre Channel ports. Should one controller, port or switch fail, the operating system can route I/O through the remaining controller transparently to the application, with no changes visible to the applications, other than perhaps incremental latency.

This is useful for:

  1. Dynamic load balancing
  2. Traffic shaping
  3. Automatic path management
  4. Dynamic reconfiguration

Linux device-mapper

In the Linux kernel, the device-mapper serves as a generic framework to map one block device onto another. It forms the foundation of LVM2 and EVMS, software RAIDs, dm-crypt disk encryption, and offers additional features such as file-system snapshots.

Device-mapper works by processing data passed in from a virtual block device, that it itself provides, and then passing the resultant data on to another block device.

How do I setup device-mapper multipathing in CentOS / RHEL 4 update 2 or above?

Open /etc/multipath.conf file, enter:
# vi /etc/multipath.conf
Make sure following line exists and commented out:

devnode_blacklist {
        devnode "*"
}

Make sure default_path_grouping_policy option in the defaults section set to failover. Here is my sample config:

defaults {
       multipath_tool  "/sbin/multipath -v0"
       udev_dir        /dev
       polling_interval 10
       default_selector        "round-robin 0"
       default_path_grouping_policy    failover
       default_getuid_callout  "/sbin/scsi_id -g -u -s /block/%n"
       default_prio_callout    "/bin/true"
       default_features        "0"
       rr_min_io              100
       failback                immediate
}

Save and close the file. Type the following command to load drivers:
# modprobe dm-multipath
# modprobe dm-round-robin

Start the service, enter:
# /etc/init.dmultipathd start
multipath is used to detect multiple paths to devices for fail-over or performance reasons and coalesces them:
# multipath -v2
Turn on service:
# /sbin/chkconfig multipathd on
Finally, create device maps from partition tables:
# kpartx -a /dev/mapper/mpath#
You need to use fdisk on the underlying disks such as /dev/sdc.

References:

  • man page kpartx,multipath, udev, dmsetup and hotplug

Load Balancer Open Source Software

Posted on in Categories Hardware, High performance computing, Linux, Linux Scalability, Linux Virtualization, Monitoring, Networking, RedHat/Fedora Linux, Storage last updated March 11, 2008

I’ve worked with a various load balancing systems (LBS). They are complex pieces of hardware and software. In this post I will highlight some of the open source load balancing software. But what is load balancing?
It is nothing but a technique used to share (spared) load / services between two or more servers. For example, busy e-commerce or bank website uses load balancer to increase reliability, throughput, uptime, response time and better resource utilization. You can use following softwares as an advanced load balancing solution for web, cache, dns, mail, ftp, auth servers, VoIP services etc.

Linux Virtual Server (LVS)

LVS is ultimate open source Linux load sharing and balancing software. You can easily build a high-performance and highly available server for Linux using this software. From the project page:

Virtual server is a highly scalable and highly available server built on a cluster of real servers. The architecture of server cluster is fully transparent to end users, and the users interact with the cluster system as if it were only a single high-performance virtual server.

=> Project Web Site

Red Hat Cluster Suite

It is a high availability cluster software implementation from Linux leader Red Hat. It provide two type services:

  1. Application / Service Failover – Create n-node server clusters for failover of key applications and services
  2. IP Load Balancing – Load balance incoming IP network requests across a farm of servers

=> Product web page

The High Availability Linux Project

Linux-HA provides sophisticated high-availability (failover) capabilities on a wide range of platforms, supporting several tens of thousands of mission critical sites.

=> Project web site

Ultra Monkey

Ultra Monkey is a project to create load balanced and highly available network services. For example a cluster of web servers that appear as a single web server to end-users. The service may be for end-users across the world connected via the internet, or for enterprise users connected via an intranet.

Ultra Monkey makes use of the Linux operating system to provide a flexible solution that can be tailored to a wide range of needs. From small clusters of only two nodes to large systems serving thousands of connections per second.

=> Project web site

Personally, I’ve worked with both LVS and Red Hat Cluster Suite and I highly recommend these softwares.

Linux Xen High Availability Clusters Configuration Tutorial

Posted on in Categories File system, Hardware, High performance computing, Linux, Linux Scalability, Linux Virtualization, UNIX, Windows, xen last updated February 6, 2008

Xen is one of the leading Virtualization software. You can use Xen virtualization to implement HA clusters. However, there are few issues you must be aware of while handling failures in a high-availability environment. This article explains configuration options using Xen:

The idea of using virtual machines to build high available clusters is not new. Some software companies claim that virtualization is the answer to your HA problems, off course that’s not true. Yes, you can reduce downtime by migrating virtual machines to another physical machine for maintenance purposes or when you think hardware is about to fail, but if an application crashes you still need to make sure another application instance takes over the service. And by the time your hardware fails, it’s usually already too late to initiate the migration.

So, for each and every application you still need to look at whether you want to have it constantly available, if you can afford the application to be down for some time, or if your users won’t mind having to relogin when one server fails.

=> Using Xen for High Availability Clusters [onlamp.com]

Virtualization: Run Windows and Linux at One Place

Posted on in Categories Howto, Linux, Linux desktop, Linux Virtualization, Windows last updated January 24, 2008

I’ve used VMWARE ESX / Xen paravirtualization, Virtuozzo, Solaris Containers, and FreeBSD Jails as os level virtualization. Virutalbox is another full virtualization solution. Presently, VirtualBox runs on Windows, Linux and Macintosh hosts and supports a large number of guest operating systems including but not limited to Windows (NT 4.0, 2000, XP, Server 2003, Vista), DOS/Windows 3.x, Linux (2.4 and 2.6), and OpenBSD.

Virtualization: Run Windows and Linux at One Place

Rakesh has published a small article about VirtualBox Virtualization software. Both Windows and Linux can be run together simultaneously, and you don’t even need to switch between the two. With the seamless Windows feature of the latest version of VirtualBox virtualization software, you can seamlesssly run both Windows and Linux applications from the same desktop interface. This has been made possible by the combined efforts of VirtualBox and SeamlessRDP that is meant for seamless Windows support for rdesktop.

=> How to run Windows and Linux at one place? [ciol.com]

Linux Fibre Channel over Ethernet implementation code released

Posted on in Categories Linux, Linux Scalability, Linux Virtualization, Networking, Open source coding, Storage last updated December 18, 2007

Intel has just released source code for Fibre Channel over Ethernet (FCoE). It provides some Fibre Channel protocol processing as well as the encapsulation of FC frames within Ethernet packets. FCoE will allow systems with an Ethernet adapter and a Fibre Channel Forwarder to login to a Fibre Channel fabric (the FCF is a “gateway” that bridges the LAN and the SAN). That fabric login was previously reserved exclusively for Fibre Channel HBAs. This technology reduces complexity in the data center by aiding network convergence. It is targeted for 10Gps Ethernet NICs but will work on any Ethernet NIC supporting pause frames. Intel will provide a Fibre Channel protocol processing module as well as an Ethernet based transport module. The Open-FC module acts as a LLD for SCSI and the Open-FCoE transport uses net_device to send and receive packets.

This is good news. I think one can compare bandwidth and throughput for copper and fiber Ethernet. If you are going to use copper you need to stay within 15m of the switch. This solution will try to bring down cost. One can connect to 8-10 server to central database server with 10G and there could be few more applications.

=> Open FCoE project home page

Download of the day: Oracle VM

Posted on in Categories Download of the day, High performance computing, Linux, Linux distribution, Linux Virtualization, Networking, xen last updated December 7, 2007

Virtualization is the process of abstracting computing resources such that multiple operating system and application images can share a single physical server, bringing significant cost-of-ownership and manageability benefits. Through its Oracle VM product, Oracle offers scalable, low-cost server virtualization for heterogeneous applications.

Oracle VM is free server virtualization software that fully supports both Oracle and non-Oracle applications, and is three times more efficient than other server virtualization products.
Continue reading “Download of the day: Oracle VM”

Quick Tip: Send Ctr + Alt + Del to XEN Vps / Guest os

Posted on in Categories CentOS, Howto, Linux, Linux Virtualization, RedHat/Fedora Linux last updated December 4, 2007

If you hit CTRL+ALT+Delete your host will go down under CentOS Linux 5.x. The solution is hit CTRL key thrice. Now it will disable CTRL+ALT+Del for host. Next just hit alt+del to complete guest CTRL+ALT+Del cycle.

Alternatively, you can type the following on host to shutdown vps:
xm shutdown myGuestVpsOsName

XEN Server Status Monitoring Command Cheat Sheet

Posted on in Categories CentOS, Debian Linux, Howto, Linux, Linux Virtualization, Monitoring, RedHat/Fedora Linux, xen last updated November 15, 2007

The xm command is the main command line interface for managing Xen guest domains. The program can be used to create, pause, and shutdown domains. It can also be used to list current domains, enable or pin VCPUs, and attach or detach virtual block devices.

Please note that before running any one of the following command you must run xend ( Xen control daemon aka service) and must be run as privileged user. Running xm command as non root will return an error.

I hope following XEN status monitoring cheat sheet will save your time.
Continue reading “XEN Server Status Monitoring Command Cheat Sheet”

How to: Upgrade Red Hat Enterprise Linux (RHEL) to latest Releases 5.1

Posted on in Categories CentOS, Howto, Linux distribution, Linux Virtualization, RedHat/Fedora Linux, Sys admin, Tips last updated November 8, 2007

RHEL 5.1 has been released. Redhat announced the availability of Red Hat Enterprise Linux 5.1, with integrated virtualization. This release provides the most compelling platform for customers and software developers ever, with its industry-leading virtualization capabilities complementing Red Hat’s newly announced Linux Automation strategy. It offers the industry’s broadest deployment ecosystem, covering standalone systems, virtualized systems, appliances and web-scale “cloud” computing environments.

Besides supporting Linux virtual machines, RHEL 5.1 will also support Windows XP, Windows 2000, Windows Server 2003 and the forthcoming Windows 2008, Crenshaw said. RHEL 5.1 uses Xen for its virtualization.

How do I upgrade to RHEL 5.1?

Login as the root user and simply type the command to fetch all updates via RHN:
# yum update
Depend upon your network condition and software configuration it may take anywhere between 5-20 minutes. Once completed, just reboot the server:
# reboot
Verify that everything is working fine:
# netstat -tulpn
# netstat -nat
# tail -f /var/log/messages
# egrep -i 'error|warn' /var/log/messages
# egrep -i 'error|warn' /path/to/apps/log

Community driven enterprise CentOS Linux users should expect update soon too. You can apply above commends to upgrade your CentOS box.