Parallel supercomputers and computer clusters needs to use failover and Infiniband multipathing to provide non stop computing. InfiniBand is a switched fabric communications link primarily used in high-performance computing. Its features include quality of service and failover, and it is designed to be scalable. The InfiniBand architecture specification defines a connection between processor nodes and high performance I/O nodes such as storage devices. It is a superset of the Virtual Interface Architecture.
InfiniBand theoretical throughput (speed)
InfiniBand provides high speed data transfer. For example USB 2.0 provides 480Mb/s data transfer or Gigagbit Ethernet support 1,000Mb/s, while 12X InfiniBand provides 96 Gbit/s in quad configuration. It also supports both copper and optical cabling. Following table shows effective theoretical throughput in different configurations:
| InfiniBand | Single | Double | Quad |
| 1X | 2 Gbit/s | 4 Gbit/s | 8 Gbit/s |
| 4X | 8 Gbit/s | 16 Gbit/s | 32 Gbit/s |
| 12X | 24 Gbit/s | 48 Gbit/s | 96 Gbit/s |
The InfiniBand Project
The InfiniBand Architecture (IBA) is an industry standard that defines a new high-speed switched fabric subsystem designed to connect processor nodes and I/O nodes to form a system area network. This new interconnect method moves away from the local transaction-based I/O model across busses to a remote message-passing model across channels. The architecture is independent of the host operating system (OS) and the processor platform.
IBA provides both reliable and unreliable transport mechanisms in which messages are enqueued for delivery between end systems. Hardware transport protocols are defined that support reliable and unreliable messaging (send/receive), and memory manipulation semantics (e.g., RDMA read/write) without software intervention in the data transfer path.
Linux and infiniband support
Most enterprise Linux distribution (such as RHEL 4.5 / 5, CentOS / Novell Linux) has support for Infiniband (IPoIB), multipathing and failover. Linux kernel v2.6.11 and above has support for IPoIB and related technologies. The OpenFabrics Alliance is creating an open source software stack for InfiniBand and iWARP that includes the "IBVerbs" library.
I may get chance to play with infiniband based devices and Linux in near future :D
You should follow me on twitter here or grab rss feed to keep track of new changes.
Featured Articles:
- 30 Handy Bash Shell Aliases For Linux / Unix / Mac OS X
- Top 30 Nmap Command Examples For Sys/Network Admins
- 25 PHP Security Best Practices For Sys Admins
- 20 Linux System Monitoring Tools Every SysAdmin Should Know
- 20 Linux Server Hardening Security Tips
- Linux: 20 Iptables Examples For New SysAdmins
- Top 20 OpenSSH Server Best Security Practices
- Top 20 Nginx WebServer Best Security Practices
- 20 Examples: Make Sure Unix / Linux Configuration Files Are Free From Syntax Errors
- 15 Greatest Open Source Terminal Applications Of 2012
- My 10 UNIX Command Line Mistakes
- Top 10 Open Source Web-Based Project Management Software
- Top 5 Email Client For Linux, Mac OS X, and Windows Users
- The Novice Guide To Buying A Linux Laptop
{ 4 comments… read them below or add one }
Great article , I didn’t knew about the (IPoIB) before this.
Do you know if debian have (IPoIB)
No, I think, Debian, Gentoo and even FreeBSD don’t have InfiniBand/ibverbs support. However, Ubuntu feisty seems to have support packages.
IMHO, you may download and easily build feisty source packages on a Debian lenny/testing machine.
Hope that helps!
Gentoo has had support for IPoIB since 2005.
Debian has support for IPoIB.
http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-5.html