CentOS / Redhat: Install nginx As Reverse Proxy Load Balancer

Posted on in Categories , , last updated February 1, 2011

How do I configure nginx as failover reverse proxy load balancer in front of two Apache web servers under CentOS / RHEL 5.x?

nginx is a Web and Reverse proxy server. Nginx used in front of Apache Web servers. All connections coming from the Internet addressed to one of the Web servers are routed through the nginx proxy server, which may either deal with the request itself or pass the request wholly or partially to the main web servers.

Our Sample Setup

    =============                               |---- apache1 (
    | ISP Router|                               |   
    =============                               |---- apache2 (
         |                                      | 
         |                                      |---- db1 (
         |      |eth0 -> ----------/
         |-lb0==|                        /
         |      |eth1 -> ----/
         |      |eth0 -> ----------\
         |-lb1==|                        /      |---- apache1 (
                |eth1 -> ----/       |
                                                |---- apache2 (
                                                |---- db1 (


  • lb0 – Linux box directly connected to the Internet via eth1. This is master load balancer.
  • lb1 – Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.
  • – This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
  • eth0 is connected to LAN and all other backend software servers are connected via eth0.
  • nginx is installed on both lb0 and lb1. It will listen on You need to configure nginx as reverse proxy server. It will connects to Apache1 and Apache2.
  • Install httpd server on Apache#1 and Apache#2 server. Configure them to listen on and Do not assign public IP to this box. Only activate eth0 via LAN.
  • Install MySQL / Oracle / PgSQL server on Db#1. Configure db server to listen on$db_server_port. Do not assign public IP to this box. Only activate eth0 via LAN.

In short you need the following hardware:

  • 2 load balancer reverse proxy servers (250GB SATA, 2GB RAM, Single Intel P-D930 or AMD 170s with RHEL 64 bit+keepalived+nginx)
  • 2 Apache web servers (Software RAID-1, SCSI-73GBx2 15k disk, 6GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+Apache 2)
  • 1 backup Apache web servers (Software RAID-1, SCSI-73GBx2 15k disk, 6GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+Apache 2)
  • 1 database server (RAID-10, SCSI-73GBx4 15k disk, 16GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit+MySQL 5)
  • 1 Caching server (RAID-1, SCSI-73GBx2 15k disk, 8GB RAM, Dual Intel Xeon or AMD 64 bit CPU with RHEL 64 bit)
  • 1 offsite backup server (RAID-6, 1TB SATAx4, 4GB RAM, Single Intel/AMD CPU with RHEL 64bit)
  • Slave database, storage, pop3 and SMTP server as per requirements.
  • Internet uplink 100Mbps+ or as per requirements.

Remove Unwanted Software From lb0 and lb1

Type the following commands:
# yum -y groupremove "X Window System"
# x=$(yum list installed | egrep -i 'php|httpd|mysql|bind|dhclient|tftp|inetd|xinetd|ypserv|telnet-server|rsh-server|vsftpd|tcsh' | awk '{ print $1}')
# yum -y remove $x
# yum -y install bind-utils sysstat openssl-devel.x86_64 pcre-devel.x86_64 openssl097a.x86_64
# /usr/sbin/authconfig --passalgo=sha512 --update
# passwd root

The above will remove X windows and other unwanted software from both lb0 and lb1.

Install Nginx On Both lb0 and lb1

Type the following commands to download nginx, enter:
# cd /opt
# wget http://sysoev.ru/nginx/nginx-0.8.33.tar.gz

Untar nginx, enter:
# tar -zxvf nginx-0.8.33.tar.gz
# cd nginx-0.8.33

Configure nginx for 64 bit RHEL / CentOS Linux:
# ./configure --without-http_autoindex_module --without-http_ssi_module --without-http_userid_module --without-http_auth_basic_module --without-http_geo_module --without-http_fastcgi_module --without-http_empty_gif_module --with-openssl=/lib64

Sample outputs:

  nginx path prefix: "/usr/local/nginx"
  nginx binary file: "/usr/local/nginx/sbin/nginx"
  nginx configuration prefix: "/usr/local/nginx/conf"
  nginx configuration file: "/usr/local/nginx/conf/nginx.conf"
  nginx pid file: "/usr/local/nginx/logs/nginx.pid"
  nginx error log file: "/usr/local/nginx/logs/error.log"
  nginx http access log file: "/usr/local/nginx/logs/access.log"
  nginx http client request body temporary files: "client_body_temp"
  nginx http proxy temporary files: "proxy_temp"
  nginx http fastcgi temporary files: "fastcgi_temp"

Install the same:
# make
# make install

Create nginx User Account

Type the following commands to create a user account:
# useradd -s /sbin/nologin -d /usr/local/nginx/html -M nginx
# passwd -l nginx

Configure nginx As Reverse Proxy Load Balancer On Both lb0 and lb1

Edit /usr/local/nginx/conf/nginx.conf, enter:
# vi /usr/local/nginx/conf/nginx.conf
Update it as follows:

pid               logs/nginx.pid;
user              nginx nginx;
worker_processes  10;
events {
    worker_connections  1024;
http {
  default_type       application/octet-stream;
 ## Common options ##
 include options.conf;
 ## Proxy settings ##
 include proxy.conf;
 ## lb domains ##
 include nixcraft.in.conf;

Edit /usr/local/nginx/conf/options.conf, enter:
# vi /usr/local/nginx/conf/options.conf
Update it as follows:

 ## Size Limits
  client_body_buffer_size     128K;
  client_header_buffer_size   1M;
  client_max_body_size          1M;
  large_client_header_buffers 8 8k;
 ## Timeouts
  client_body_timeout   60;
  client_header_timeout 60;
  expires               24h;
  keepalive_timeout     60 60;
  send_timeout          60;
 ## General Options
  ignore_invalid_headers   on;
  keepalive_requests      100;
  limit_zone gulag $binary_remote_addr 5m;
  recursive_error_pages    on;
  sendfile                 on;
  server_name_in_redirect off;
  server_tokens           off;
 ## TCP options
  tcp_nodelay on;
  tcp_nopush  on;
 ## Compression
  gzip              on;
  gzip_buffers      16 8k;
  gzip_comp_level   6;
  gzip_http_version 1.0;
  gzip_min_length   0;
  gzip_types        text/plain text/css image/x-icon application/x-perl application/x-httpd-cgi;
  gzip_vary         on;
 ## Log Format
  log_format  main  '$remote_addr $host $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" "$http_user_agent" '

Edit /usr/local/nginx/conf/proxy.conf, enter:

 ## Proxy caching options
  proxy_buffering           on;
  proxy_cache_min_uses       3;
  proxy_cache_path          /usr/local/nginx/proxy_temp/ levels=1:2 keys_zone=cache:10m inactive=10m max_size=1000M;
  proxy_cache_valid         any 10m;
  proxy_ignore_client_abort off;
  proxy_intercept_errors    on;
  proxy_next_upstream       error timeout invalid_header;
  proxy_redirect            off;
  proxy_set_header          X-Forwarded-For $remote_addr;
  proxy_connect_timeout     60;
  proxy_send_timeout        60;
  proxy_read_timeout        60;

Edit /usr/local/nginx/conf/nixcraft.in.conf, enter:

## Connect to backend servers via LAN ##
## Reverse Proxy Load Balancer Logic ##
upstream nixcraft  {
      server weight=10 max_fails=3 fail_timeout=30s;
      server weight=10 max_fails=3 fail_timeout=30s;
      # only comes alive when above two fails      
      server weight=1 backup;
server {
      access_log  logs/access.log main;
      error_log   logs/error.log;
      index       index.html;
      root        /usr/local/nginx/html;
      server_name nixcraft.in www.nixcraft.in subdomain.nixcraft.in;
     ## Only requests to our Host are allowed
      if ($host !~ ^(nixcraft.in|www.nixcraft.in|subdomain.nixcraft.in)$ ) {
         return 444;
     ## redirect www to nowww
     # if ($host = 'www.nixcraft.in' ) {
     #    rewrite  ^/(.*)$  http://nixcraft.in/$1  permanent;
     # }
     ## Only allow these request methods
     if ($request_method !~ ^(GET|HEAD|POST)$ ) {
         return 444;
     ## PROXY - Web
      location / {
        proxy_pass  http://nixcraft;
        proxy_cache            cache;
        proxy_cache_valid      200 24h;
        proxy_cache_use_stale  error timeout invalid_header updating http_500 http_502 http_503 http_504;
        proxy_ignore_headers   Expires Cache-Control;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
     # redirect server error pages to the static page /50x.html
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;

Start nginx web server:
# /usr/local/nginx/sbin/nginx
# netstat -tulpn | grep :80
# echo ' /usr/local/nginx/sbin/nginx' >> /etc/rc.local

Fire a webbrowser and type domain name such as nixcraft.in:


Stay tuned, for rest of the firewall, security, SELinux, database, helper scripts and optimization configuration.

24 comment

  1. How do you make to both apache servers display same web content, via some shared storage (NFS), or some kind sync between 2 apache servers ?

    PS. Great howto!

  2. @Vivek – thanks for the write up :)
    @bane – you can try use `unison` for two-way replication over SSH with a cronjob, but there are obviously some draw-backs to using this.

  3. What would I need to modify if I did not want to use the load balancing bits? I am just doing this for one webserver on the back-end.

  4. @ Scott,

    Just add one IP to upstream without max_fails, fail_timeout etc. If your backend is at

    upstream nixcraft  {

    Rest is same.

  5. Good evening

    The load balancer routes traffic to web servers in cycle or has any options to check the least loaded server?


    1. Requests are distributed according to the servers in round-robin manner with respect of the server weight. If you want to send requests to the least-busy backend server, rather than distributing requests round-robin, use ngx_http_upstream_fair_module.


      1. I am new to nginx. How do I get specifics on how to use ngx_http_upstream_fair_module. Any pointers would be appreciated.


  6. Hello

    Thanks for you Vivek

    Can you help me with Virtual host ip based ??

    i have more than 100 website ..

    95 with 1 ip = 123.456.789.10
    1 with 1 ip = 123.456.789.11
    2 with 1 ip = 123.456.789.12
    3 with 1 ip = 123.456.789.13
    4 with 1 ip = 123.456.789.14
    5 with 1 ip = 123.456.789.15

    I can do all of this with 1 ip of them ..

    i try to user multple location like

    location / {
    proxy_pass http://123.456.789.10:8080/;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 90;
    proxy_send_timeout 90;
    proxy_read_timeout 90;
    client_max_body_size 10m;
    client_body_buffer_size 128k;
    proxy_buffer_size 4k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;
    root /usr/share/nginx/html;
    index index.html index.htm index.php;

    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
    root html;


    and i use mod_rpaf

    But i can’t do this ^_^

    Thanks for you

  7. Since you’re forwarding IP addresses using extra headers using X-Forwarded-For and X-Real-IP, what do you do on apache to log the real ip address? I assume you have to change the LogFormat directive?

      1. I think nginx does support IP-hash based sticky sessions by using
        upstream backend {
        server backend1.example.com;
        server backend2.example.com;

  8. Hi, I’m trying to set up nginx like this but nginx keeps routing the request for the internal app server over eth0 (WAN) in stead of eth1 (LAN). proxy_bind also doesn’t seem to work. Do I need to do some iptables wizardry to get nginx to translate the incoming request on eth0 to a request on eth1?

    1. No iptables is not required. You must have some sort of networking config issue or may be existing iptables is preventing the same. I suggest you use our forum and post the output of the following command including your nginx.conf:

      netstat -tulpn
      iptable -L -v -n

      Also check nginx error and access log file.

  9. Why is there so much complication involved ?
    Can’t we just use a server running apache with mod_proxy_balancer and two normal webservers from which content is served ?

  10. hi vivek ,

    i have 3 virtual machine using centos 6 on local machine
    1) -> nginx as lb0 im not using public just test on local machine
    backend -> -> back end (apache+mysql)
    backend -> -> back end (apache+mysql)

    using this script my machine is running properly but when i did changed with the step your did mmm my lb was not working need your assist :

    here’s the script without problem
    #nginx configuration
    upstream localhost
    # ip_hash;
    server max_fails=10 fail_timeout=180s;
    server max_fails=10 fail_timeout=180s;
    # only comes alive when above two fails
    #server weight=1 backup;

    access_log logs/access.log main;
    error_log logs/error.log;
    index index.html;
    root /usr/local/nginx/html;

    listen 80;
    # listen;
    #server_name default;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 5;
    proxy_send_timeout 10;
    proxy_read_timeout 15;

    location / {
    proxy_pass http://localhost;
    proxy_cache off;


    log erros in nginx with the config script as your wrote:
    2011/12/16 22:39:39 [error] 1745#0: *7 connect() failed (113: No route to host) while connecting to upstream, client:, server:, request: “GET / HTTP/1.1”, upstream: “”, host: “”

  11. Hi;

    I have a network vip say and behind this network vip are 2 live hosts. Each host is running a keepalive daemon which is being monitored by the network VIP. Subject to the keepalive it routes traffic to the hosts. I am using RP devices for with failover as follows:

    upstream cai_8500 {
    server weight=10 max_fails=1 fail_timeout=15s;
    server weight=10 max_fails=1 fail_timeout=15s;


    I want to be able to use the network VIP to do the load balancing or failover. If I want to do any maintenance work on any of the hosts, I simply stop the “keepalive” daemon and the network VIP does not send any traffic to it. The problem I’m having with RP is that it keeps sending traffic as its using the host ip and not the VIP. Any ideas what I can do to resolve this issue please?


Leave a Comment