≡ Menu


Lighttpd Traffic Shaping: Throttle Connections Per Single IP (Rate Limit)

If you do not control or throttle end users, your server may run out of resources. Spammers, abuser and badly written bots can eat up all your bandwidth. A webserver must keep an eye on connections and limit connections per second. This is serving 101. The default is no limit. Lighttpd can limit the throughput for each single connection (per IP) or for all connections. You also need to a use firewall to limit connections per second. In this article I will cover firewall and lighttpd web server settings to throttle end users. The firewall settings can be applied to other web servers such as Apache / Nginx and IIS server behind PF / netfilter based firewall.
[click to continue…]

Comparison: Linux Disk Scheduler

I've already written about changing the I/O scheduler for hard disk under Linux and avoiding sudden outburst of disk I/O using ionice utility. I/O schedulers can have many purposes such as:
Minimize time wasted by hard disk seeks.
Prioritize a certain processes' I/O requests.
Give a share of the disk bandwidth to each running process etc

Google has sponsored Gelato@UNSW to take a close look at the disk schedulers in Linux, particularly when combined with RAID. They have now published their findings:

We benchmarked the four standard Linux disk schedulers using several different tools (see our wiki for full details) and lots of different workloads, on single SCSI and SATA disks, and on hardware and software RAID arrays from two to eight spindles (hardware raid) and up to twenty spindles (software raid), trying RAID levels 0 through 6.

We had to fix some of the benchmarking tools (the fixes are now upstream), and we developed a new one: a Markov Chain based replay tool, which allows a workload to be characterised and then a similar workload generated.

=> Selected comparisons of throughput and latency with the different Linux schedulers (via Google open source blog)

Download Of The Day: OpenSSH 5.1

OpenSSH server and client version 5.1 has just been released and available for download. New features in OpenSSH 5.1:
=> Introduce experimental SSH Fingerprint ASCII Visualisation to ssh(1) and ssh-keygen(1).

=> sshd now support CIDR address/masklen matching.

=> Added an extended test mode (-T) to sshd(8) to request that it write its effective configuration to stdout and exit.

=> ssh(1) now prints the number of bytes transferred and the overall connection throughput for SSH protocol 2 sessions when in verbose mode.

=> Added a MaxSessions option to sshd_config(5) to allow control of the number of multiplexed sessions supported over a single TCP connection.

Download OpenSSH 5.1

=> Visit offical site to grab latest OpenSSH 5.1

mod_compress: Lighttpd Gzip Compression To Improve Download and Browsing Speed

Gzip is the most popular and effective compression method. Most modern web browser supports and accepts compressed data transfer. By gziping response time can reduced by 60-70% as compare to normal web page. The end result is faster web site experience for both dial up (they're not dead yet - I've dial up account for backup purpose) and broadband user. I've already written about speeding up Apache 2.x web access or downloads with mod_deflate.

mod_compress for Lighttpd 1.4.xx

Lighttpd 1.4.xx supports gzip compression using mod_compress. This module can reduces the network load and can improve the overall throughput of the webserver. All major http-clients support compression by announcing it in the Accept-Encoding header as follows:

Accept-Encoding: gzip, deflate

If lighttpd sees this header in the request, it can compress the response using one of the methods listed by the client. The web server notifies the web client of this via the Content-Encoding header in the response:

Content-Encoding: gzip

This is used to negotiate the most suitable compression method. Lighttpd support deflate, gzip and bzip2.

Configure mod_compress

Open your lighttpd.conf file:
# vi /etc/lighttpd/lighttpd.conf
Append mod_compress to server.modules directive:
server.modules += ( "mod_compress" )
Setup compress.cache-dir to stored all cached file:
compress.cache-dir = "/tmp/lighttpdcompress/"
Finally, define mimetypes to get compressed. Following will allow to compress javascript, plain text files, css file,xml file etc:

compress.filetype           = ("text/plain","text/css", "text/xml", "text/javascript" )

Save and close the file. Create /tmp/lighttpdcompress/ file:
# mkdir -p /tmp/lighttpdcompress/
# chown lighttpd:lighttpd /tmp/lighttpdcompress/

Restart lighttpd:
# /etc/init.d/lighttpd restart

How do I enable mod_compress per virtual host?

Use conditional $HTTP host directive, for example turn on compression for theos.in:

$HTTP["host"] =~ "theos\.in" {
  compress.cache-dir = "/var/www/cache/theos.in/"

PHP dynamic compression

Open php.in file:
# vi /etc/php.ini
To compress dynamic content with PHP please enable following two directives:
zlib.output_compression = On
zlib.output_handler = On

Save and close the file. Restart lighttpd:
# service lighttpd restart

Cleaning cache directory

You need to run a shell script for cleaning out cache directory.

See also:

EKA – The world's 4th Fastest Super Computer

In November 2007 EKA was announced, which is privately funded supercomputer. Eka, built by CRL, Pune is the world's 4th fastest supercomputer, and the fastest one that didn't use government funding. This is the same supercomputer referenced in Yahoo!'s recent announcement about cloud computing research at the Hadoop Summit. This article describes some of the technical details of Eka's design and implementation.

My town - pune is home to some of the best supercomputing research and development facility into the area of software and other related technology. C-DAC (a scientific society of the Department of Information Technology, Government of India) has developed and supplied a range of high performance parallel computers, known as the PARAM series of supercomputers. Now CRL - a tata group company build the EKa one of the top 10 supercomputers in the world, Eka is the only system that was fully privately funded. All other systems used government money, so all of them are for captive use. This means that Eka is the only system in the top 10 that is available for commercial use without strings attached.

The computing infrastructure itself consists of:

  1. 1800 blades, 4 cores each. 3Ghz for each core.
  2. HP SFS clusters
  3. 28TB memory
  4. 80TB storage. Simple SATA disks. 5.2Gbps throughput.
  5. Lustre distributed file-system
  6. 20Gbps infiniband DDR. Eka was on the cutting edge of Infiniband technology. They sourced their infiniband hardware from an Israeli company and where amongst the first users of their releases - including beta, and even alpha quality stuff.
  7. Multiple Gigabit ethernets
  8. Linux is the underlying OS. Any Linux will work - RedHat, SuSe, your favorite distribution.

=> Building EKA - The world's fastest privately funded supercomputer

Linux Fibre Channel over Ethernet implementation code released

Intel has just released source code for Fibre Channel over Ethernet (FCoE). It provides some Fibre Channel protocol processing as well as the encapsulation of FC frames within Ethernet packets. FCoE will allow systems with an Ethernet adapter and a Fibre Channel Forwarder to login to a Fibre Channel fabric (the FCF is a "gateway" that bridges the LAN and the SAN). That fabric login was previously reserved exclusively for Fibre Channel HBAs. This technology reduces complexity in the data center by aiding network convergence. It is targeted for 10Gps Ethernet NICs but will work on any Ethernet NIC supporting pause frames. Intel will provide a Fibre Channel protocol processing module as well as an Ethernet based transport module. The Open-FC module acts as a LLD for SCSI and the Open-FCoE transport uses net_device to send and receive packets.

This is good news. I think one can compare bandwidth and throughput for copper and fiber Ethernet. If you are going to use copper you need to stay within 15m of the switch. This solution will try to bring down cost. One can connect to 8-10 server to central database server with 10G and there could be few more applications.

=> Open FCoE project home page

HowTo: Linux Check IDE / SATA Hard Disk Transfer Speed

So how do you find out how fast is your hard disk under Linux? Is it running at SATA I (150 MB/s) or SATA II (300 MB/s) speed without opening computer case or chassis?
[click to continue…]