Does Linux support IP over Infiniband (IPoIB) multipathing and failover?

by on September 24, 2007 · 4 comments· LAST UPDATED September 24, 2007

in , ,

Parallel supercomputers and computer clusters needs to use failover and Infiniband multipathing to provide non stop computing. InfiniBand is a switched fabric communications link primarily used in high-performance computing. Its features include quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. It is a superset of the Virtual Interface Architecture.

InfiniBand theoretical throughput (speed)

InfiniBand provides high speed data transfer. For example USB 2.0 provides 480Mb/s data transfer or Gigagbit Ethernet support 1,000Mb/s, while 12X InfiniBand provides 96 Gbit/s in quad configuration. It also supports both copper and optical cabling. Following table shows effective theoretical throughput in different configurations:

InfiniBandSingleDoubleQuad
1X2 Gbit/s4 Gbit/s8 Gbit/s
4X 8 Gbit/s16 Gbit/s32 Gbit/s
12X24 Gbit/s48 Gbit/s96 Gbit/s

The InfiniBand Project

The InfiniBand Architecture (IBA) is an industry standard that defines a new high-speed switched fabric subsystem designed to connect processor nodes and I/O nodes to form a system area network. This new interconnect method moves away from the local transaction-based I/O model across busses to a remote message-passing model across channels. The architecture is independent of the host operating system (OS) and the processor platform.

IBA provides both reliable and unreliable transport mechanisms in which messages are enqueued for delivery between end systems. Hardware transport protocols are defined that support reliable and unreliable messaging (send/receive), and memory manipulation semantics (e.g., RDMA read/write) without software intervention in the data transfer path.

Linux and infiniband support

Most enterprise Linux distribution (such as RHEL 4.5 / 5, CentOS / Novell Linux) has support for Infiniband (IPoIB), multipathing and failover. Linux kernel v2.6.11 and above has support for IPoIB and related technologies. The OpenFabrics Alliance is creating an open source software stack for InfiniBand and iWARP that includes the "IBVerbs" library.

I may get chance to play with infiniband based devices and Linux in near future :D

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 4 comments… read them below or add one }

1 Get your Software September 24, 2007 at 2:31 pm

Great article , I didn’t knew about the (IPoIB) before this.
Do you know if debian have (IPoIB)

Reply

2 Balwinder S Dheeman September 24, 2007 at 6:52 pm

No, I think, Debian, Gentoo and even FreeBSD don’t have InfiniBand/ibverbs support. However, Ubuntu feisty seems to have support packages.

IMHO, you may download and easily build feisty source packages on a Debian lenny/testing machine.

Hope that helps!

Reply

3 quantumsummers November 20, 2008 at 2:56 pm

Gentoo has had support for IPoIB since 2005.

Reply

4 TooMeeK June 9, 2012 at 9:56 am

Leave a Comment

Tagged as: , , , , , , , , , , , ,

Previous post:

Next post: