Lighttpd Traffic Shaping: Throttle Connections Per Single IP (Rate Limit)

by on June 21, 2009 · 15 comments· LAST UPDATED February 9, 2010

in , ,

If you do not control or throttle end users, your server may run out of resources. Spammers, abuser and badly written bots can eat up all your bandwidth. A webserver must keep an eye on connections and limit connections per second. This is serving 101. The default is no limit. Lighttpd can limit the throughput for each single connection (per IP) or for all connections. You also need to a use firewall to limit connections per second. In this article I will cover firewall and lighttpd web server settings to throttle end users. The firewall settings can be applied to other web servers such as Apache / Nginx and IIS server behind PF / netfilter based firewall.

Lignttpd: Limit All Connections

You can limit the throughput for all connections to the given limit in kbyte/s. Open lighttpd.conf file:
# vi lighttpd.conf
Set limit to 1024 kbyte/s:
server.kbytes-per-second=1024
Save and close the file. Reload lighttpd server:
# service lighttpd reload

Lighttpd: Limit Throughput For Each Single Connection

Set limit to 64 kbyte/s for each single connection per IP:
connection.kbytes-per-second=64
Reload lighttpd server:
# service lighttpd reload

How Do I Set a Limit Only For Virtual Host?

You can set limit for virtual host only as follows (limit traffic to theos.in to 64 kbyte/s:

 
    $HTTP["host"] == "theos.in" {
      server.kbytes-per-second = 64
    }
 

How Do I Limit Connections Per Single IP?

You need to use a firewall such as *BSD PF or Linux netfilter firewall.

*BSD PF Firewall Example - Limit Connections Per Single IP

Add following rules to your /etc/pf.conf file. The following rules will protect the webserver against hosts making more than 100 connections in 10 seconds. Any IP which connects faster than this rate will have its address added to the table and have all states originating from it flushed. Any new packets from same IP to web server will be dropped:

webserver_ip="202.54.1.1"
table <abusive_ips> persist
block quick from <abusive_ips>
pass in on $ext_if proto tcp to $webserver_ip port www keep state (max-src-conn-rate 100/10, overload <bad_hosts> flush global)

Another example:

webserver_ip="202.54.1.1"
table <abusive_ips> persist
block in quick from <abusive_ips>
pass in on $ext_if proto tcp to $webserver_ip port www flags S/SA keep state (max-src-conn 100, max-src-conn-rate 15/5, overload <abusive_ips> flush)

Here is what it does:

  • Limits the maximum number of connections per source to 100 (some browsers can open 30-40 connections per IP, so keep this to 100)
  • Next, limit the number of connections per second or span of seconds. For e.g. rate limit the number of connections to 15 in a 5 second span.
  • If anyone breaks our rules add them to our abusive_ips table and block them for making any further connections.
  • Finally, flush keyword kills all states created by the matching rule which originate from the host which exceeds these limits.

Feel free to adjust settings as per your setup.

Linux Netfilter (Iptables) Examples To Limit Connections

The following example will drop incoming connections if IP make more than 10 connection attempts to port 80 within 100 seconds (add rules to your iptables shell script)

IPT=/sbin/iptables
# Max connection in seconds
SECONDS=100
# Max connections per IP
BLOCKCOUNT=10
# ....
# ..
# default action can be DROP or REJECT 
DACTION="DROP"
$IPT -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
$IPT -I INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds ${SECONDS} --hitcount ${BLOCKCOUNT} -j ${DACTION}
# ....
# ..

Again, feel free to adjust settings as per your setup.

Recommend Readings:

  1. Sample PF firewall script.
  2. Sample Iptables firewall script.
  3. The official lighttpd documentation.
  4. Iptables recent patch documentation.
  5. The official pf documentation.
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 15 comments… read them below or add one }

1 Vasi June 22, 2009 at 3:13 am

Excellent information thanks a lot

Reply

2 Swapnil K. Chaudhari June 22, 2009 at 6:08 am

Hi,
I have a query regardig Limiting Connections Per Single IP.

Cosider a scenario of a big company, where there is only one public IP address available and all the users use DHCP or private IP. So if say 100 emplyee in the company access the same web site simultaneously (For checking mails, or some other common information). Web server will see that same public IP is generating 100 connections and put the public IP of company in black list, and employees won’t be able to connect to that web server.

How can we avoid this scenario?

Thanks,
Swapnil.

Reply

3 nixCraft June 22, 2009 at 8:36 am

You can always white list certain IPs / subnets or create class of IPs to exclude throttling.

Reply

4 gadelkareem August 28, 2010 at 2:46 pm

could you give up an example?

Reply

5 kunal June 24, 2009 at 4:47 am

How can i throttle connections per single IP in apache. Earlier there used to be a module named as mod_evassive but now i dont think its available any more.


Kunal

Reply

6 nixCraft June 24, 2009 at 5:20 am

Try,
mod_bandwidth:

“Mod_bandwidth” is a module for the Apache webserver that enable the setting of server-wide or per connection bandwidth limits, based on the directory, size of files and remote IP/domain.

Download : http://www.cohprog.com/mod_bandwidth.html
mod_limitipconn:

It allows web server administrators to limit the number of simultaneous downloads permitted from a single IP address.

Download : http://dominia.org/djao/limitipconn2.html

Reply

7 kunal June 25, 2009 at 2:58 am

Thanks Vivek,

Will definitely give a try to this.


Kunal

Reply

8 geeth August 3, 2009 at 12:14 pm

The blog is really superb stuff, could you please mention the regarding apache module too

Reply

9 Vicent Gonzalez i Castells August 12, 2009 at 7:01 am

Great job. A good information.

Following Swapnil comment, it’s a great problem to create a white list. We can think about to create a white list of universitues IP’s of the world. For me, it’s a good technique to apply into small environments like a enterprise network.

Santi

Reply

10 The Mikeness July 21, 2010 at 9:55 pm

Note that limiting concurrent connections per IP to 100 may block some organizations that use a shoddy HTTP Proxy, as it will usually refuse to hold persistent connections, and there may be dozens or hundreds of users behind the proxy. What the end users will begin to see if you do alot of this sort of filtering without using extreme care is things failing at odd places, too many requests within a few seconds randomly causing some portions of pages to not load completely (like CSS and image files), AJAX requests to fail, etc.

I just though I’d mention that incase this sort of thinking becomes prominent among people running webservers and big sites start to have hard to troubleshoot problems that may end up ruining your reputation amongst your users for being unable to operate a reliable service.

Reply

11 nixCraft June 12, 2011 at 9:38 am

You serve all static content (css/js/images/videos) from other domain or cdn to avoid this issue.

Reply

12 gadelkareem August 28, 2010 at 2:49 pm

What about evasive.max-conns-per-ip = 20 for lighty?

Reply

13 dirk March 18, 2011 at 12:11 pm

evasive.max-conns-per-ip = 14
is a good value for lighttpd. no need to install firewalls.

Reply

14 David September 18, 2011 at 8:49 pm

Excelent, but is there a way you can do the same using apache instead of lighthttp?

Reply

15 JC October 22, 2011 at 8:44 am

There is mod_evasive for Apache.

Reply

Leave a Comment

Tagged as: , , , , , , , ,

Previous post:

Next post: