CentOS / Redhat Linux: Install Keepalived To Provide IP Failover For Web Cluster

by on February 21, 2010 · 30 comments· LAST UPDATED February 1, 2011

in , ,

Keepalived provides a strong and robust health checking for LVS clusters. It implements a framework of health checking on multiple layers for server failover, and VRRPv2 stack to handle director failover. How do I install and configure Keepalived for reverse proxy server such as nginx or lighttpd?

If your are using a LVS director to loadbalance a server pool in a production environment, you may want to have a robust solution for healthcheck & failover. This will also work with reverse proxy server such as nginx.

Our Sample Setup

Internet--
         |
    =============
    | ISP Router|
    =============
         |
         |
         |      |eth0 -> 192.168.1.11 (connected to lan)
         |-lb0==|
         |      |eth1 -> 202.54.1.1 (vip master)
         |
         |      |eth0 -> 192.168.1.10 (connected to lan)
         |-lb1==|
                |eth1 -> 202.54.1.1 (vip backup)

Where,

  • lb0 - Linux box directly connected to the Internet via eth1. This is master load balancer.
  • lb1 - Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.
  • 202.54.1.1 - This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
  • eth0 is connected to LAN and all other backend software such as Apache, MySQL and so on.

You need to install the following softwares on both lb0 and lb1:

  • keepalived for IP failover.
  • iptables to filter traffic
  • nginx or lighttpd revers proxy server.

DNS settings should be as follows:

  1. nixcraft.in - Our sample domain name.
  2. lb0.nixcraft.in - 202.54.1.11 (real ip assigned to eth1)
  3. lb1.nixcraft.in - 202.54.1.12 (real ip assigned to eth1)
  4. www.nixcraft.in - 202.54.1.1 (VIP for web server) do not assign this IP to any interface.

Install Keepalived

Visit keepalived.org to grab latest source code. You can use the wget command to download the same (you need to install keepalived on both lb0 and lb1):
# cd /opt
# wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz
# tar -zxvf keepalived-1.1.19.tar.gz
# cd keepalived-1.1.19

Install Kernel Headers

You need to install the following packages:

  1. Kernel-headers - includes the C header files that specify the interface between the Linux kernel and userspace libraries and programs. The header files define structures and constants that are needed for building most standard programs and are also needed for rebuilding the glibc package.
  2. kernel-devel - this package provides kernel headers and makefiles sufficient to build modules against the kernel package.

Make sure kernel-headers and kernel-devel packages are installed. If not type the following install the same:
# yum -y install kernel-headers kernel-devel

Compile keepalived

Type the following command:
# ./configure --with-kernel-dir=/lib/modules/$(uname -r)/build
Sample outputs:

checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
...
.....
..
config.status: creating keepalived/check/Makefile
config.status: creating keepalived/libipvs-2.6/Makefile
Keepalived configuration
------------------------
Keepalived version       : 1.1.19
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
Use VRRP Framework       : Yes
Use Debug flags          : No

Compile and install the same:
# make && make install

Create Required Softlinks

Type the following commands to create service and run it at RHEL / CentOS run level #3 :
# cd /etc/sysconfig
# ln -s /usr/local/etc/sysconfig/keepalived .
# cd /etc/rc3.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived S100keepalived
# cd /etc/init.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived .

Configuration

Your main configuration directory is located at /usr/local/etc/keepalived and configuration file name is keepalived.conf. First, make backup of existing configuration:
# cd /usr/local/etc/keepalived
# cp keepalived.conf keepalived.conf.bak

Edit keepalived.conf as follows on lb0:

vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 101
        authentication {
            auth_type PASS
            auth_pass Add-Your-Password-Here
        }
        virtual_ipaddress {
                202.54.1.1/29 dev eth1
        }
}

Edit keepalived.conf as follows on lb1 (note priority set to 100 i.e. backup load balancer):

vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 100
        authentication {
            auth_type PASS
            auth_pass Add-Your-Password-Here
        }
        virtual_ipaddress {
                202.54.1.1/29 dev eth1
        }
}

Save and close the file. Finally start keepalived on both lb0 and lb1 as follows:
# /etc/init.d/keepalived start

Verify: Keepalived Working Or Not

/var/log/messages will keep track of VIP:
# tail -f /var/log/messages
Sample outputs:

Feb 21 04:06:15 lb0 Keepalived_vrrp: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:06:20 lb0 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1

Verify that VIP assigned to eth1:
# ip addr show eth1
Sample outputs:

3: eth1:  mtu 1500 qdisc pfifo_fast qlen 10000
    link/ether 00:30:48:30:30:a3 brd ff:ff:ff:ff:ff:ff
    inet 202.54.1.11/29 brd 202.54.1.254 scope global eth1
    inet 202.54.1.1/29 scope global secondary eth1

ping failover test

Open UNIX / Linux / OS X desktop terminal and type the following command to ping to VIP:
# ping 202.54.1.1
Login to lb0 and halt the server or take down networking:
# halt
Within seconds VIP should move from lb0 to lb1 and you should not see any drops in ping. On lb1 you should get the following in /var/log/messages:

Feb 21 04:10:07 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) forcing a new MASTER election
Feb 21 04:10:08 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Feb 21 04:10:09 lb1 Keepalived_healthcheckers: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1

Conclusion

Your server is now configured with IP failover. However, you need to install and configure the following software in order to configure webserver and security:

  1. nginx or lighttpd
  2. iptables

Stay tuned, for more information on above configuration.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 30 comments… read them below or add one }

1 jaysunn February 21, 2010 at 2:49 pm

This is awesome. I am planning to try this when I have some free time. Great schematic work. Are you running this setup in production at all?

Jaysunn

Reply

2 nixCraft February 21, 2010 at 3:19 pm

Are you running this setup in production at all?

Yes, we are running this kind of setup in production. One of my client serves over 1 to 1.2 millions of page view per day, at peak traffic touch 50-70Mbps. We have over 12 Apache webserver nodes behind nginx+keepalived.

Stay tunned for rest of the articles in series which will cover nginx / lighttpd and firewall setup.

HTH

Reply

3 zoren February 28, 2012 at 7:00 am

hi im desperate on finding a solution for this..i was instructed by someone to look for a multiple linux configuration that has or maybe shows load balancing and failover..
can u help me Vivek Gite regarding this? or anyone who has knowledge about it coz im not even familiar with this terminology..and i dont understand it even more but i need to find it out..please help.

Reply

4 Kevin Green February 21, 2010 at 9:04 pm

Looking forward for the tutorial covering nginx / lighttpd :D

Reply

5 Srinivas Kotaru February 22, 2010 at 12:36 pm

This is really nice. I’m also looking forward to get more articles on reverse proxy kind of setup’s

Srinivas

Reply

6 Bhaskar Chowdhury February 23, 2010 at 1:53 pm

Cool!

keep up the good work Vivek :)

Thanks
Bhaskar

Reply

7 Ren's May 12, 2010 at 6:24 pm

Hello !

Great tutorial but it doesn’t work with me ! :/

I follow it and I have the following error :

[root@ keepalived]# /etc/init.d/keepalived start
Starting keepalived: /bin/bash: keepalived: command not found
[FAILED]

Ok, I modify the script as follow :

start() {
echo -n $”Starting $prog: ”
daemon /usr/local/sbin/keepalived ${KEEPALIVED_OPTIONS}

After, it starts but the VIP (192.168.1.99) is not configured on my ethernet :/

Here the message log :

May 12 20:13:40 Keepalived: Starting Keepalived v1.1.19 (05/12,2010)
May 12 20:13:40 Keepalived: Starting Healthcheck child process, pid=9944
May 12 20:13:40 Keepalived: Starting VRRP child process, pid=9946
May 12 20:13:40 Keepalived_healthcheckers: Netlink reflector reports IP 192.168.1.246 added
May 12 20:13:40 Keepalived_healthcheckers: Registering Kernel netlink reflector
May 12 20:13:40 Keepalived_healthcheckers: Registering Kernel netlink command channel
May 12 20:13:40 Keepalived_vrrp: Netlink reflector reports IP 192.168.1.246 added
May 12 20:13:40 Keepalived_vrrp: Registering Kernel netlink reflector
May 12 20:13:40 Keepalived_vrrp: Registering Kernel netlink command channel
May 12 20:13:40 Keepalived_vrrp: Registering gratutious ARP shared channel
May 12 20:13:40 Keepalived_healthcheckers: Configuration is using : 3029 Bytes
May 12 20:13:40 Keepalived_vrrp: Configuration is using : 55023 Bytes
May 12 20:13:40 Keepalived_vrrp: Using LinkWatch kernel netlink reflector…
May 12 20:13:40 Keepalived_healthcheckers: Using LinkWatch kernel netlink reflector…

I tried on Centos 5 and on Fedora. It doesn’t work on both…
I try to install with yum on Fedora, it works fine :/ … I need to work on RHEL…

Any ideas to help me ?

Reply

8 Ren's May 12, 2010 at 7:25 pm

Ok it works fine now, I found the solution ! Yeahhhh xD !

Keepalived seems searching the config file in /etc/keepalived…

ln -s /usr/local/etc/keepalived/ /etc/keepalived

And I modified my startup script with the path to keepalived.

Reply

9 maros May 31, 2010 at 9:00 pm

It is bad idea to setup keepalived startup on CentOS via linking like this:

# cd /etc/rc3.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived S100keepalived

This causes trouble with IP assignments when machine is booting because the network is started AFTER the keepalived daemon. The best way is setup startup of keepalived in standart way: ‘chkconfig keepalived on’

Reply

10 charleshb June 22, 2010 at 5:00 pm

Is the setup described above suitable for simple IP failover without load balancer and LVS?

Reply

11 nixCraft June 22, 2010 at 6:33 pm

Yes, it does failover without LB or anything else.

Reply

12 Carlos June 22, 2010 at 8:35 pm

My setup is just that, simple failover, no LB

Reply

13 charleshb June 24, 2010 at 8:19 pm

I must be missing something…when I start keepalived it shows the VIP assigned to eth1, but I have no route for the gateway. If I try to add the route manually I get SIOCADDRT: Network is unreachable. If I add the same IP manually I can add a route and ping the gateway. Any pointers?

Reply

14 Carlos June 22, 2010 at 8:33 pm

I tried this guide out along with the recommended steps from the comments and it worked beautifully on my centos 5.5 environment. I also verified from the Cisco switch that the ARP table changed when the backup node took over as it’s supposed to:
XXXXXX-3560g-11#sh arp | include 10.XXX.XXX.XXX
Internet 10.XXX.XXX.XXX 0 XXXX.XXXX.1328 ARPA VlanXX

XXXXXX-3560g-11#sh arp | include 10.XXX.XXX.XX
Internet 10.XXX.XXX.XXX 0 XXXX.XXXX.5094 ARPA VlanXX

Reply

15 Jean July 2, 2010 at 5:41 am

You can also check out wackamole. It is on top of a spread service. With DNS Round Robin you got a failover and a simple performance cluster.

Reply

16 Reynold P J March 20, 2011 at 1:11 pm

Thanks for the wonderful post :)

Reply

17 apaajha August 5, 2011 at 8:14 am

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
authentication {
auth_type PASS
auth_pass Add-Your-Password-Here
}
virtual_ipaddress {
202.54.1.1/29 dev eth1
}
}

I think you should have a typo on interface eth0 the correct is interface eth1

CMIIW

Reply

18 Saurabh November 7, 2011 at 8:49 pm

beautiful link.. crystal clear explanation… kudos.. Thanks a lot.

Reply

19 lukman December 6, 2011 at 3:06 pm

awesome brooooooo keep on posting yummy

Reply

20 Harsimran February 6, 2012 at 10:59 am

Is this solution work with apache httpd?

Reply

21 mccs March 6, 2012 at 11:27 am

don’t forget when you have a network servicelisting on a port that you probaly need commands to restart the service. like i need to on named on centos

notify_backup “/sbin/service named restart”
notify_master “/sbin/service named restart”
# notify_fault “/sbin/service named restart”

Reply

22 josue May 11, 2012 at 7:31 pm

someone could tell me if you can use Keepalived to a cluster of streaming servers.

Reply

23 MADC0D3R May 30, 2012 at 8:39 pm

I currently have keepalived configured to handle both load balancing and failover, I want to turn off load balancing and just do failover. I’m fairly new to Linux, so any help would be appreciated.

Reply

24 Patrick February 20, 2013 at 3:02 pm

Tried this on a Centos 6 box.
Fixed:
start() {
echo -n $”Starting $prog: ”
daemon /usr/local/sbin/keepalived ${KEEPALIVED_OPTIONS}

and

ln -s /usr/local/etc/keepalived/ /etc/keepalived

But keepalived logs to /var/log/messages:
Feb 20 15:47:00 vps modprobe: FATAL: Module ip_vs not found.
Feb 20 15:47:00 vps Keepalived_vrrp[1304]: IPVS: Can’t initialize ipvs: Protocol not available

Any idea? My box is a virtuozzo container. My this be the problem?

Reply

25 Sandeep May 28, 2013 at 9:35 am

Hi Vivek,

We are configured keepalived with lighttpd but we are facing some issue, can you please help me to resolve issue. Following are details of issue:

We have 3 machines installed with keepalived and when we ran the command “ip addr sh eth0″ we are able to virtual IP on any one machine. Until here every thing is clear.
But some times even though virtual IP is assigned to any one of the machine we are unable ping or access the virtual IP and domain hosted on it from external subnet. From same subnet we are able to ping and access the virtual IP and the domain hosted on it.
We are unable to get any clue regarding the issue.
The only solution we are using to resolve this issue is restarting keepalived in all machines.
Can you please help me to resolve the issue?

Thanks,
Sandeep

Reply

26 Lahiru May 29, 2013 at 11:53 am

Hi,
I have configured the keepalived. And its working correctly. But i want to float the IP with the service (as an example Tom cat service). If service is down state then the VIP must float in to the other backup. How can I configure this.

Thanks.
Lahiru

Reply

27 lee July 3, 2013 at 10:26 am

Hi,
nice clear tutorial, and confirms that my setup should be working. wonder if anyone can help with a problem i’m having, google searching hasn’t helped.
I’ve got keepalived setup on 2 Ubuntu 12.04 servers. with 4 physical interfaces, and two vlan interfaces, eth0, eth1, eth2, eth3, plus eth2.10 and eth2.20. I have failover ip’s set on every interface, with all vrrp instances together in a single vrrp group. if I take down any interface on the master, every failover ip gets successfully moved over to the backup server. I can see in the logs that garps are sent out for each ip/interface on the backup server. I can ping every failover ip address without interruption. so far so good. but before failover, I can ping from a separate pc on any one of these subnets, to other pc’s on any of the other subnets, however, after failover, I can’t ping across the backup keepalived server to these other pc’s. I get absolutely nothing until the primary server is back up and running and has taken back the failover ip’s.
anyone got any idea why this would be happening?

thanks
lee.

Reply

28 nixCraft July 3, 2013 at 12:20 pm

Can you use service installed on the backup system? If so check for the firewall logs.

Reply

29 Fabiano November 23, 2013 at 12:00 am

Hi, Nice tutorial! I´m need something like this, but working in one Linux Box, but with two links connnected on them, it´s possible to do in this way?
E.g:

Linux Box |—- eth0 LAN
|—- eth1 Internet 1
|— eth2 Internet 2
Thanks

Reply

30 Khizer Abdul Sattar March 8, 2014 at 8:02 am

[root@localhost ~]# cp /usr/local/sbin/keepalived /usr/sbin/

[root@localhost ~]# /etc/init.d/keepalived start
Starting keepalived: [ OK ]

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , ,

Previous Faq:

Next Faq: