≡ Menu

Linux Scalability

Intel has just released source code for Fibre Channel over Ethernet (FCoE). It provides some Fibre Channel protocol processing as well as the encapsulation of FC frames within Ethernet packets. FCoE will allow systems with an Ethernet adapter and a Fibre Channel Forwarder to login to a Fibre Channel fabric (the FCF is a "gateway" that bridges the LAN and the SAN). That fabric login was previously reserved exclusively for Fibre Channel HBAs. This technology reduces complexity in the data center by aiding network convergence. It is targeted for 10Gps Ethernet NICs but will work on any Ethernet NIC supporting pause frames. Intel will provide a Fibre Channel protocol processing module as well as an Ethernet based transport module. The Open-FC module acts as a LLD for SCSI and the Open-FCoE transport uses net_device to send and receive packets.

This is good news. I think one can compare bandwidth and throughput for copper and fiber Ethernet. If you are going to use copper you need to stay within 15m of the switch. This solution will try to bring down cost. One can connect to 8-10 server to central database server with 10G and there could be few more applications.

=> Open FCoE project home page

My previous article related to iSCSI storage and NAS storage brought a couple of questions. An interesting question from my mail bag:

I've 5 Debian Linux servers with HP SAN box. Should I boot from SAN?

No, use centralized network storage for shared data or high availability configuration only. Technically you can boot and configure system. However I don't recommend booting from SAN or any other central server until and unless you need diskless nodes:

[a] Use local storage - Always use local storage for /boot and / (root) filesystem

[b] Keep it simply - Booting from SAN volumes is complicated procedure. Most operating systems are not designed for this kind of configuration. You need to modify scripts and booting procedure.

[c] SAN booting support - Your SAN vendor must support platform booting a Linux server. You need to configure HBA and SAN according to vendor specification. You must totally depend upon SAN vendor for drivers and firmware (HBA Bios) to get thing work properly. General principle - don't put all your eggs in one basket err one vendor ;)

[d] Other factors - Proper fiber channel topology must be used. Make sure Multipathing and redundant SAN links are used. The boot disk LUN is dedicated to a single host. etc

As you can see, complications started to increases, hence I don't recommend booting from SAN.

RHEL / CentOS Support 4GB or more RAM ( memory )

If you have 4 GB or more RAM use the Linux kernel compiled for PAE capable machines. Your machine may not show up total 4GB ram. All you have to do is install PAE kernel package.

This package includes a version of the Linux kernel with support for up to 64GB of high memory. It requires a CPU with Physical Address Extensions (PAE).
The non-PAE kernel can only address up to 4GB of memory. Install the kernel-PAE package if your machine has more than 4GB of memory (>=4GB).

How Do I Install PAE kernel?

To install PAE kernel, use yum command:
# yum install kernel-PAE
Output:

Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
Reading repository metadata in from local files
Parsing package install arguments
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for kernel-PAE to pack into transaction set.
kernel-PAE-2.6.18-8.1.15. 100% |=========================| 207 kB    00:00
---> Package kernel-PAE.i686 0:2.6.18-8.1.15.el5 set to be installed
--> Running transaction check
Dependencies Resolved
=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 kernel-PAE              i686       2.6.18-8.1.15.el5  updates            12 M
Transaction Summary
=============================================================================
Install      1 Package(s)
Update       0 Package(s)
Remove       0 Package(s)
Total download size: 12 M
Is this ok [y/N]: y
Downloading Packages:
(1/1): kernel-PAE-2.6.18- 100% |=========================|  12 MB    00:12
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: kernel-PAE                   ######################### [1/1]
Installed: kernel-PAE.i686 0:2.6.18-8.1.15.el5
Complete!

Just reboot the server and make sure you boot with PAE kernel i.e. 2.6.18-8.1.15.el5PAE:
# reboot

Linux Creating a Partition Size Larger Than 2TB

Frankly speaking, you cannot create a Linux partition larger than 2 TB using the fdisk command. The fdisk won't create partitions larger than 2 TB. This is fine for desktop and laptop users, but on server you need a large partition. For example, you cannot create 3TB or 4TB partition size (RAID based) using the fdisk command. It will not allow you to create a partition that is greater than 2TB. In this tutorial, you will learn more about creating Linux filesystems greater than 2 Terabytes to support enterprise grade operation under any Linux distribution.
[click to continue…]

Yesterday I wrote about increasing local port range with net.ipv4.ip_local_port_range proc file. There is also /proc/sys/kernel/pid_max file, which specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID). The default value for this file, 32768, results in the same range of PIDs as on earlier kernels (< =2.4). The value in this file can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million). [click to continue…]

Linux Local Port Range
If your Linux server is opening lots of outgoing network connections, you need to increase local port range. By default range is small. For example a squid proxy server can come under fire if it runs out of ports.
[click to continue…]

If you try to mount an ext3 Linux filesystem on a SAN from multiple nodes at the same time you will be in serious deep trouble.

SAN based storage allows multiple nodes to connect to same devices at the same time. Ext3/2 are not cluster aware file system. They can lead to a disaster such as kernel panic, server hang, corruption etc.

You need to use something which supports:

  1. Useful in clusters for moderate scale out and shared SAN volumes
  2. Symmetrical Parallel Cluster File System, Journaled
  3. POSIX access controls

Both GFS (RedHat Global File System) and Lustre (a scalable, secure, robust, highly available cluster file system) can be used with SAN based storage allows multiple nodes to connect to same devices at the same time.

Many newbie get confused as Linux offers a number of file systems. This paper (Linux File System Primer) discusses these file systems, why there are so many, and which ones are the best to use for which workloads and data.