CentOS / Redhat: Install nginx As Reverse Proxy Load Balancer

by on February 24, 2010 · 24 comments· LAST UPDATED February 1, 2011

in , ,

How do I configure nginx as failover reverse proxy load balancer in front of two Apache web servers under CentOS / RHEL 5.x?

nginx is a Web and Reverse proxy server. Nginx used in front of Apache Web servers. All connections coming from the Internet addressed to one of the Web servers are routed through the nginx proxy server, which may either deal with the request itself or pass the request wholly or partially to the main web servers.

Our Sample Setup

Internet--
         |
    =============                               |---- apache1 (192.168.1.15)
    | ISP Router|                               |
    =============                               |---- apache2 (192.168.1.16)
         |                                      |
         |                                      |---- db1 (192.168.1.17)
         |      |eth0 -> 192.168.1.11 ----------/
         |-lb0==|                        /
         |      |eth1 -> 202.54.1.1 ----/
         |
         |      |eth0 -> 192.168.1.10 ----------\
         |-lb1==|                        /      |---- apache1 (192.168.1.15)
                |eth1 -> 202.54.1.1 ----/       |
                                                |---- apache2 (192.168.1.16)
                                                |
                                                |---- db1 (192.168.1.17)

Where,

  • lb0 - Linux box directly connected to the Internet via eth1. This is master load balancer.
  • lb1 - Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.
  • 202.54.1.1 - This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
  • eth0 is connected to LAN and all other backend software servers are connected via eth0.
  • nginx is installed on both lb0 and lb1. It will listen on 202.54.1.1. You need to configure nginx as reverse proxy server. It will connects to Apache1 and Apache2.
  • Install httpd server on Apache#1 and Apache#2 server. Configure them to listen on 192.168.1.15:80 and 192.168.1.16:80. Do not assign public IP to this box. Only activate eth0 via LAN.
  • Install MySQL / Oracle / PgSQL server on Db#1. Configure db server to listen on 192.168.1.17:$db_server_port. Do not assign public IP to this box. Only activate eth0 via LAN.

In short you need the following hardware:

  • 2 load balancer reverse proxy servers (250GB SATA, 2GB RAM, Single Intel P-D930 or AMD 170s with RHEL 64 bit+keepalived+nginx)
  • 2 Apache web servers (Software RAID-1, SCSI-73GBx2 15k disk, 6GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+Apache 2)
  • 1 backup Apache web servers (Software RAID-1, SCSI-73GBx2 15k disk, 6GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+Apache 2)
  • 1 database server (RAID-10, SCSI-73GBx4 15k disk, 16GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+MySQL 5)
  • 1 Caching server (RAID-1, SCSI-73GBx2 15k disk, 8GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit)
  • 1 offsite backup server (RAID-6, 1TB SATAx4, 4GB RAM, Single Intel/AMD CPU with RHEL 64bit)
  • Slave database, storage, pop3 and SMTP server as per requirements.
  • Internet uplink 100Mbps+ or as per requirements.

Remove Unwanted Software From lb0 and lb1

Type the following commands:
# yum -y groupremove "X Window System"
# x=$(yum list installed | egrep -i 'php|httpd|mysql|bind|dhclient|tftp|inetd|xinetd|ypserv|telnet-server|rsh-server|vsftpd|tcsh' | awk '{ print $1}')
# yum -y remove $x
# yum -y install bind-utils sysstat openssl-devel.x86_64 pcre-devel.x86_64 openssl097a.x86_64
# /usr/sbin/authconfig --passalgo=sha512 --update
# passwd root

The above will remove X windows and other unwanted software from both lb0 and lb1.

Install Nginx On Both lb0 and lb1

Type the following commands to download nginx, enter:
# cd /opt
# wget http://sysoev.ru/nginx/nginx-0.8.33.tar.gz

Untar nginx, enter:
# tar -zxvf nginx-0.8.33.tar.gz
# cd nginx-0.8.33

Configure nginx for 64 bit RHEL / CentOS Linux:
# ./configure --without-http_autoindex_module --without-http_ssi_module --without-http_userid_module --without-http_auth_basic_module --without-http_geo_module --without-http_fastcgi_module --without-http_empty_gif_module --with-openssl=/lib64

Sample outputs:

....
  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"
...

Install the same:
# make
# make install

Create nginx User Account

Type the following commands to create a user account:
# useradd -s /sbin/nologin -d /usr/local/nginx/html -M nginx
# passwd -l nginx

Configure nginx As Reverse Proxy Load Balancer On Both lb0 and lb1

Edit /usr/local/nginx/conf/nginx.conf, enter:
# vi /usr/local/nginx/conf/nginx.conf
Update it as follows:

 
pid               logs/nginx.pid;
user              nginx nginx;
worker_processes  10;
 
events {
    worker_connections  1024;
}
 
http {
  default_type       application/octet-stream;
 
 ## Common options ##
 include options.conf;
 
 ## Proxy settings ##
 include proxy.conf;
 
 ## lb domains ##
 include nixcraft.in.conf;
}

Edit /usr/local/nginx/conf/options.conf, enter:
# vi /usr/local/nginx/conf/options.conf
Update it as follows:

 
 ## Size Limits
  client_body_buffer_size     128K;
  client_header_buffer_size   1M;
  client_max_body_size          1M;
  large_client_header_buffers 8 8k;
 
 ## Timeouts
  client_body_timeout   60;
  client_header_timeout 60;
  expires               24h;
  keepalive_timeout     60 60;
  send_timeout          60;
 
 ## General Options
  ignore_invalid_headers   on;
  keepalive_requests      100;
  limit_zone gulag $binary_remote_addr 5m;
  recursive_error_pages    on;
  sendfile                 on;
  server_name_in_redirect off;
  server_tokens           off;
 
 ## TCP options
  tcp_nodelay on;
  tcp_nopush  on;
 
 ## Compression
  gzip              on;
  gzip_buffers      16 8k;
  gzip_comp_level   6;
  gzip_http_version 1.0;
  gzip_min_length   0;
  gzip_types        text/plain text/css image/x-icon application/x-perl application/x-httpd-cgi;
  gzip_vary         on;
 
 ## Log Format
  log_format  main  '$remote_addr $host $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" "$http_user_agent" '
                    '"$gzip_ratio"';
 

Edit /usr/local/nginx/conf/proxy.conf, enter:

 
 ## Proxy caching options
  proxy_buffering           on;
  proxy_cache_min_uses       3;
  proxy_cache_path          /usr/local/nginx/proxy_temp/ levels=1:2 keys_zone=cache:10m inactive=10m max_size=1000M;
  proxy_cache_valid         any 10m;
  proxy_ignore_client_abort off;
  proxy_intercept_errors    on;
  proxy_next_upstream       error timeout invalid_header;
  proxy_redirect            off;
  proxy_set_header          X-Forwarded-For $remote_addr;
  proxy_connect_timeout     60;
  proxy_send_timeout        60;
  proxy_read_timeout        60;
 

Edit /usr/local/nginx/conf/nixcraft.in.conf, enter:

 
## Connect to backend servers via LAN ##
## Reverse Proxy Load Balancer Logic ##
upstream nixcraft  {
      server 192.168.1.15 weight=10 max_fails=3 fail_timeout=30s;
      server 192.168.1.16 weight=10 max_fails=3 fail_timeout=30s;
      # only comes alive when above two fails
      server 192.168.1.23 weight=1 backup;
}
 
server {
      access_log  logs/access.log main;
      error_log   logs/error.log;
      index       index.html;
      root        /usr/local/nginx/html;
      server_name nixcraft.in www.nixcraft.in subdomain.nixcraft.in;
 
     ## Only requests to our Host are allowed
      if ($host !~ ^(nixcraft.in|www.nixcraft.in|subdomain.nixcraft.in)$ ) {
         return 444;
      }
 
     ## redirect www to nowww
     # if ($host = 'www.nixcraft.in' ) {
     #    rewrite  ^/(.*)$  http://nixcraft.in/$1  permanent;
     # }
 
     ## Only allow these request methods
     if ($request_method !~ ^(GET|HEAD|POST)$ ) {
         return 444;
     }
 
     ## PROXY - Web
      location / {
        proxy_pass  http://nixcraft;
        proxy_cache            cache;
        proxy_cache_valid      200 24h;
        proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
        proxy_ignore_headers   Expires Cache-Control;
 
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
      }
 
     # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
}

Start nginx web server:
# /usr/local/nginx/sbin/nginx
# netstat -tulpn | grep :80
# echo ' /usr/local/nginx/sbin/nginx' >> /etc/rc.local

Fire a webbrowser and type domain name such as nixcraft.in:
http://nixcraft.in

References:

Stay tuned, for rest of the firewall, security, SELinux, database, helper scripts and optimization configuration.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 24 comments… read them below or add one }

1 bane February 25, 2010 at 10:44 am

How do you make to both apache servers display same web content, via some shared storage (NFS), or some kind sync between 2 apache servers ?

PS. Great howto!

Reply

2 nixCraft February 25, 2010 at 10:48 am

@bane: yes you can use NFS or other file sharing protocols. I will cover some helper script which sync Apache root.

Reply

3 Curtis February 25, 2010 at 10:51 pm

@Vivek – thanks for the write up :)
@bane – you can try use `unison` for two-way replication over SSH with a cronjob, but there are obviously some draw-backs to using this.

Reply

4 Scott Behrens April 9, 2010 at 3:56 pm

What would I need to modify if I did not want to use the load balancing bits? I am just doing this for one webserver on the back-end.

Reply

5 nixCraft April 9, 2010 at 5:22 pm

@ Scott,

Just add one IP to upstream without max_fails, fail_timeout etc. If your backend is at 127.0.0.1:8015

upstream nixcraft  {
 server 127.0.0.1:8015;
}

Rest is same.

Reply

6 nick April 15, 2010 at 2:07 pm

Good evening

The load balancer routes traffic to web servers in cycle or has any options to check the least loaded server?

regards

Reply

7 nixCraft April 15, 2010 at 3:05 pm

Requests are distributed according to the servers in round-robin manner with respect of the server weight. If you want to send requests to the least-busy backend server, rather than distributing requests round-robin, use ngx_http_upstream_fair_module.

HTH

Reply

8 Anita Roshan January 27, 2011 at 11:49 pm

I am new to nginx. How do I get specifics on how to use ngx_http_upstream_fair_module. Any pointers would be appreciated.

Thanks
Anita

Reply

9 nick April 15, 2010 at 5:48 pm

thanks much for the quick reply :)

Reply

10 XxRa3eDxX April 25, 2010 at 10:26 am

Hello

Thanks for you Vivek

Can you help me with Virtual host ip based ??

i have more than 100 website ..

95 with 1 ip = 123.456.789.10
1 with 1 ip = 123.456.789.11
2 with 1 ip = 123.456.789.12
3 with 1 ip = 123.456.789.13
4 with 1 ip = 123.456.789.14
5 with 1 ip = 123.456.789.15

I can do all of this with 1 ip of them ..

i try to user multple location like

location / {
proxy_pass http://123.456.789.10:8080/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
root /usr/share/nginx/html;
index index.html index.htm index.php;
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}

}

and i use mod_rpaf

But i can’t do this ^_^

Thanks for you

Reply

11 Rushi September 20, 2010 at 9:53 am

Since you’re forwarding IP addresses using extra headers using X-Forwarded-For and X-Real-IP, what do you do on apache to log the real ip address? I assume you have to change the LogFormat directive?

Reply

12 nixCraft September 20, 2010 at 10:55 am

You can change logformat or install Apache module called mod-rpaf (recommended) .

Reply

13 Yuzi October 29, 2010 at 4:34 am

Hi,

Thanks very much for the writeup. Does this setting handle session?

Thanks.

Reply

14 nixCraft October 30, 2010 at 7:25 am

nginx does support IP-hash based sticky sessions.

Reply

15 noblody January 18, 2011 at 2:41 pm

I think nginx does support IP-hash based sticky sessions by using
upstream backend {
ip_hash;
server backend1.example.com;
server backend2.example.com;
}

Reply

16 vangel January 5, 2011 at 5:30 pm

holy **** exactly what I was looking for. weird i dint find it before when i googled with site:

Reply

17 Marc February 14, 2011 at 1:48 pm

Hi, I’m trying to set up nginx like this but nginx keeps routing the request for the internal app server over eth0 (WAN) in stead of eth1 (LAN). proxy_bind also doesn’t seem to work. Do I need to do some iptables wizardry to get nginx to translate the incoming request on eth0 to a request on eth1?

Reply

18 nixCraft February 14, 2011 at 1:59 pm

No iptables is not required. You must have some sort of networking config issue or may be existing iptables is preventing the same. I suggest you use our forum and post the output of the following command including your nginx.conf:

netstat -tulpn
iptable -L -v -n

Also check nginx error and access log file.

Reply

19 Marc February 14, 2011 at 4:51 pm
20 Muhammad Nur Husni March 3, 2011 at 9:07 am

Very nice………

Reply

21 A.Jesin May 13, 2011 at 9:37 am

Why is there so much complication involved ?
Can’t we just use a server running apache with mod_proxy_balancer and two normal webservers from which content is served ?

Reply

22 lukman December 6, 2011 at 6:56 am

nice tutor bro

thanks a lot

Reply

23 lukman December 17, 2011 at 6:27 am

hi vivek ,

i have 3 virtual machine using centos 6 on local machine
1) 192.168.168.129 -> nginx as lb0 im not using public just test on local machine
backend -> 192.168.168.130 -> back end (apache+mysql)
backend -> 192.168.168.131 -> back end (apache+mysql)

using this script my machine is running properly but when i did changed with the step your did mmm my lb was not working need your assist :

here’s the script without problem
#nginx configuration
upstream localhost
{
# ip_hash;
server 192.168.168.130:80 max_fails=10 fail_timeout=180s;
server 192.168.168.131:80 max_fails=10 fail_timeout=180s;
# only comes alive when above two fails
#server 192.168.1.23 weight=1 backup;
}

server
{
access_log logs/access.log main;
error_log logs/error.log;
index index.html;
root /usr/local/nginx/html;

listen 80;
# listen 192.168.168.29;
#server_name default;
server_name 192.168.168.129:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 5;
proxy_send_timeout 10;
proxy_read_timeout 15;

location / {
proxy_pass http://localhost;
proxy_cache off;
}

}

log erros in nginx with the config script as your wrote:
2011/12/16 22:39:39 [error] 1745#0: *7 connect() failed (113: No route to host) while connecting to upstream, client: 192.168.168.46, server: 192.168.168.129, request: “GET / HTTP/1.1″, upstream: “http://192.168.168.30:80/”, host: “192.168.168.129”

Reply

24 Ali February 26, 2013 at 10:49 am

Hi;

I have a network vip say 10.10.10.1 and behind this network vip are 2 live hosts. Each host is running a keepalive daemon which is being monitored by the network VIP. Subject to the keepalive it routes traffic to the hosts. I am using RP devices for with failover as follows:

upstream cai_8500 {
server 10.20.11.100:8500 weight=10 max_fails=1 fail_timeout=15s;
server 10.40.107.100:8500 weight=10 max_fails=1 fail_timeout=15s;

}

I want to be able to use the network VIP to do the load balancing or failover. If I want to do any maintenance work on any of the hosts, I simply stop the “keepalive” daemon and the network VIP does not send any traffic to it. The problem I’m having with RP is that it keeps sending traffic as its using the host ip and not the VIP. Any ideas what I can do to resolve this issue please?

Thanks
Ali.

Reply

Leave a Comment

Tagged as: , , , , , , , , , ,

Previous Faq:

Next Faq: