≡ Menu


Does Linux support IP over Infiniband (IPoIB) multipathing and failover?

Parallel supercomputers and computer clusters needs to use failover and Infiniband multipathing to provide non stop computing. InfiniBand is a switched fabric communications link primarily used in high-performance computing. Its features include quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. It is a superset of the Virtual Interface Architecture.

InfiniBand theoretical throughput (speed)

InfiniBand provides high speed data transfer. For example USB 2.0 provides 480Mb/s data transfer or Gigagbit Ethernet support 1,000Mb/s, while 12X InfiniBand provides 96 Gbit/s in quad configuration. It also supports both copper and optical cabling. Following table shows effective theoretical throughput in different configurations:

1X2 Gbit/s4 Gbit/s8 Gbit/s
4X 8 Gbit/s16 Gbit/s32 Gbit/s
12X24 Gbit/s48 Gbit/s96 Gbit/s

The InfiniBand Project

The InfiniBand Architecture (IBA) is an industry standard that defines a new high-speed switched fabric subsystem designed to connect processor nodes and I/O nodes to form a system area network. This new interconnect method moves away from the local transaction-based I/O model across busses to a remote message-passing model across channels. The architecture is independent of the host operating system (OS) and the processor platform.

IBA provides both reliable and unreliable transport mechanisms in which messages are enqueued for delivery between end systems. Hardware transport protocols are defined that support reliable and unreliable messaging (send/receive), and memory manipulation semantics (e.g., RDMA read/write) without software intervention in the data transfer path.

Linux and infiniband support

Most enterprise Linux distribution (such as RHEL 4.5 / 5, CentOS / Novell Linux) has support for Infiniband (IPoIB), multipathing and failover. Linux kernel v2.6.11 and above has support for IPoIB and related technologies. The OpenFabrics Alliance is creating an open source software stack for InfiniBand and iWARP that includes the "IBVerbs" library.

I may get chance to play with infiniband based devices and Linux in near future :D

Why my Linux server ext3 filesystem go read-only?

From my mailbag:

We have 5 Dell server collocated running CentOS 4.x and 5.x server operating system. Sometime my file system (ext3) goes read-only. I’d like to know what could be causing such a problem?

My guess:
a) Hardware problem / hard disk problem, check harddisk for errors.

b) High disk I/O aka busy I/O retry error can mark low level disk call as failed. This will force ext3 to go into read only mode.

c) High disk I/O on SAN

d) SAN is not configured properly for the path failover.

In all sort of problems ext3 goes read-only to protect the filesystem and further damage. If you are using VMWARE, check out official webpage to download SCSI patches or workaround for vmware problems.

So what could be causing the file system on Linux go read-only?

Apart from above generic problem, any other error can trigger filesystem on Linux go read only. I hope our reader / seasoned Linux admin can help to answer this question. Please share the experiences and advice in the comments.

MySQL Proxy Load balancing and Failover Tutorial

MySQL Proxy is a simple and new program that sits between your client and MySQL server(s) that can monitor, analyze or transform their communication. Its flexibility allows for a wide variety of use cases, including:
a) Load balancing
b) Failover
c) Query analysis
d) Query filtering and modification
e) and many more...

MySQL Proxy tutorial

Oreilly has published a nice tutorial using MySQL proxy application:

MySQL Proxy is a lightweight binary application standing between one or more MySQL clients and a server. The clients connect to the Proxy with the usual credentials, instead of connecting to the server. The Proxy acts as man-in-the-middle between client and server.

In its basic form, the Proxy is just a redirector. It gets an empty bucket from the client (a query), takes it to the server, fills the bucket with data, and passes it back to the client.

If that were all, the Proxy would just be useless overhead. There is a little more I haven't told you yet. The Proxy ships with an embedded Lua interpreter. Using Lua, you can define what to do with a query or a result set before the Proxy passes them along.

MySQL Proxy Load balancing and Failover Tutorial - Logo

Download MySQL proxy

You can download MySQL proxy here