≡ Menu

sysctl command

It is no secret that I am a pretty big fan of excellent Linux Software RAID. Creating, assembling and rebuilding small array is fine. But, things started to get nasty when you try to rebuild or re-sync large size array. You may get frustrated when you see it is going to take 22 hours to rebuild the array. You can always increase the speed of Linux Software RAID 0/1/5/6 reconstruction using the following five tips.
[click to continue…]

Core dumps are often used to diagnose or debug errors in Linux or UNIX programs. Core dumps can serve as useful debugging aids for sys admins to find out why Application like Lighttpd, Apache, PHP-CGI or any other program crashed. Many vendors and open source project author requests a core file to troubleshoot a program. A core file is generated when an application program abnormally terminates due to bug, operating system security protection schema, or program simply try to write beyond the area of memory it has allocated, and so on. This article explains how to turn on core file support and track down bugs in programs.
[click to continue…]

Applications that perform a lot of memory accesses (several GBs) may obtain performance improvements by using large pages due to reduced Translation Lookaside Buffer (TLB) misses. HugeTLBfs is memory management feature offered in Linux kernel, which is valuable for applications that use a large virtual address space. It is especially useful for database applications such as MySQL, Oracle and others. Other server software that uses the prefork or similar (e.g. Apache web server) model will also benefit.

[click to continue…]

One of my client has server node located at north America, Asia and Europe data centers. All servers are connected using 1000Mbps links. They transfers lots of data between all nodes over ssh session using scp / sftp. However, performance was horrible. After some research I came across High Performance SSH/SCP - HPN-SSH patch for OpenSSH:

SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links.

Modifying the ssh code to allow the buffers to be defined at run time eliminates this bottleneck. We have created a patch that will remove the bottlenecks in OpenSSH and is fully interoperable with other servers and clients. In addition HPN clients will be able to download faster from non HPN servers, and HPN servers will be able to receive uploads faster from non HPN clients. However, the host receiving the data must have a properly tuned TCP/IP stack.

The amount of improvement any specific user will see is dependent on a number of issues. Transfer rates cannot exceed the capacity of the network nor the throughput of the I/O subsystem including the disk and memory speed. The improvement will also be highly influenced by the capacity of the processor to perform the encryption and decryption. Less computational expensive ciphers will often provide better throughput than more complex ciphers.

You can download HPN-SSH patch here. This patch improved our performance. You also need to tweak Linux TCP/IP networking settings. Here is my sysctl.conf file ( read this TCP tunning Linux guide for detailed explanation) :
# optimization start
# increase TCP max buffer size setable using setsockopt()
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
# increase Linux auto tuning TCP buffer limits
# min, default, and max number of bytes to use
# set max to at least 4MB, or higher if you use very high BDP paths
net.core.rmem_max = 8388608
net.core.wmem_max = 8388608
net.core.netdev_max_backlog = 5000
net.ipv4.tcp_window_scaling = 1
# optimization end

Yesterday I wrote about increasing local port range with net.ipv4.ip_local_port_range proc file. There is also /proc/sys/kernel/pid_max file, which specifies the value at which PIDs wrap around (i.e., the value in this file is one greater than the maximum PID). The default value for this file, 32768, results in the same range of PIDs as on earlier kernels (< =2.4). The value in this file can be set to any value up to 2^22 (PID_MAX_LIMIT, approximately 4 million). [click to continue…]

Generally service such as ssh, screen, expect, telnet etc use pty (pseudo-terminals) in master – slave mode for login and other purposes. If pty setting is too low many users will not able to login to system using ssh or other commands. In this tip I will explain how to increase the maximum number of pseudo-terminals.

pty man page defines pseudo-terminal as follows:

A pseudo-terminal is a pair of virtual character devices that provide a bidirectional communication channel. One end of the channel is called the master; the other end is called the slave. The slave end of the pseudo-terminal provides an interface that behaves exactly like a classical terminal. A process that expects to be connected to a terminal, can open the slave end of a pseudo-terminal and then be driven by a program that has opened the master end. Anything that is written on the master end is provided to the process on the slave end as though it was input typed on a terminal.

List the maximum number of Pseudo-terminals

Just run the following command to list / display the maximum number of Pseudo-terminals under Linux
$ cat /proc/sys/kernel/pty/max


Increase the maximum number of Pseudo-terminals (PTY)

If you have large Linux installation such as University or ISP login service you need to increase the PTYs to allow more login sessions. Open kernel configuration file - /etc/sysctl.conf:
# vi /etc/sysctl.conf
Append following config directive (support 5120 ptys)
kernel.pty.max = 5120
Save and close the file. Reload the changes:
# sysctl -p
Verify that the new maximum number of pseudo-terminals value is changed, enter:
$ cat /proc/sys/kernel/pty/max

Further readings

=> Refer to sysctl, proc, and pty man pages for more information.

While administrating a box, you may wanted to find out what a processes is doing and find out how many file descriptors (fd) are being used. You will surprised to find out that process does open all sort of files:
=> Actual log file

=> /dev files

=> UNIX Sockets

=> Network sockets

=> Library files /lib /lib64

=> Executables and other programs etc

In this quick post, I will explain how to to count how many file descriptors are currently in use on your Linux server system.
[click to continue…]