CentOS / Redhat Linux: Install Keepalived To Provide IP Failover For Web Cluster

Keepalived provides a strong and robust health checking for LVS clusters. It implements a framework of health checking on multiple layers for server failover, and VRRPv2 stack to handle director failover. How do I install and configure Keepalived for reverse proxy server such as nginx or lighttpd?

If your are using a LVS director to loadbalance a server pool in a production environment, you may want to have a robust solution for healthcheck & failover. This will also work with reverse proxy server such as nginx.

Our Sample Setup

Internet--
         |
    =============
    | ISP Router|
    =============
         |
         |
         |      |eth0 -> 192.168.1.11 (connected to lan) 
         |-lb0==|
         |      |eth1 -> 202.54.1.1 (vip master)
         |
         |      |eth0 -> 192.168.1.10 (connected to lan)
         |-lb1==|
                |eth1 -> 202.54.1.1 (vip backup)

Where,

  • lb0 – Linux box directly connected to the Internet via eth1. This is master load balancer.
  • lb1 – Linux box directly connected to the Internet via eth1. This is backup load balancer. This will become active if master networking failed.
  • 202.54.1.1 – This ip moves between lb0 and lb1 server. It is called virtual IP address and it is managed by keepalived.
  • eth0 is connected to LAN and all other backend software such as Apache, MySQL and so on.

You need to install the following softwares on both lb0 and lb1:

  • keepalived for IP failover.
  • iptables to filter traffic
  • nginx or lighttpd revers proxy server.

DNS settings should be as follows:

  1. nixcraft.in – Our sample domain name.
  2. lb0.nixcraft.in – 202.54.1.11 (real ip assigned to eth1)
  3. lb1.nixcraft.in – 202.54.1.12 (real ip assigned to eth1)
  4. www.nixcraft.in – 202.54.1.1 (VIP for web server) do not assign this IP to any interface.

Install Keepalived

Visit keepalived.org to grab latest source code. You can use the wget command to download the same (you need to install keepalived on both lb0 and lb1):
# cd /opt
# wget http://www.keepalived.org/software/keepalived-1.1.19.tar.gz
# tar -zxvf keepalived-1.1.19.tar.gz
# cd keepalived-1.1.19

Install Kernel Headers

You need to install the following packages:

  1. Kernel-headers – includes the C header files that specify the interface between the Linux kernel and userspace libraries and programs. The header files define structures and constants that are needed for building most standard programs and are also needed for rebuilding the glibc package.
  2. kernel-devel – this package provides kernel headers and makefiles sufficient to build modules against the kernel package.

Make sure kernel-headers and kernel-devel packages are installed. If not type the following install the same:
# yum -y install kernel-headers kernel-devel

Compile keepalived

Type the following command:
# ./configure --with-kernel-dir=/lib/modules/$(uname -r)/build
Sample outputs:

checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables... 
checking for suffix of object files... o
...
.....
..
config.status: creating keepalived/check/Makefile
config.status: creating keepalived/libipvs-2.6/Makefile

Keepalived configuration
------------------------
Keepalived version       : 1.1.19
Compiler                 : gcc
Compiler flags           : -g -O2
Extra Lib                : -lpopt -lssl -lcrypto 
Use IPVS Framework       : Yes
IPVS sync daemon support : Yes
Use VRRP Framework       : Yes
Use Debug flags          : No

Compile and install the same:
# make && make install

Create Required Softlinks

Type the following commands to create service and run it at RHEL / CentOS run level #3 :
# cd /etc/sysconfig
# ln -s /usr/local/etc/sysconfig/keepalived .
# cd /etc/rc3.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived S100keepalived
# cd /etc/init.d/
# ln -s /usr/local/etc/rc.d/init.d/keepalived .

Configuration

Your main configuration directory is located at /usr/local/etc/keepalived and configuration file name is keepalived.conf. First, make backup of existing configuration:
# cd /usr/local/etc/keepalived
# cp keepalived.conf keepalived.conf.bak

Edit keepalived.conf as follows on lb0:

vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 101
        authentication {
            auth_type PASS
            auth_pass Add-Your-Password-Here
        }
        virtual_ipaddress {
                202.54.1.1/29 dev eth1
        }
}

Edit keepalived.conf as follows on lb1 (note priority set to 100 i.e. backup load balancer):

vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 100
        authentication {
            auth_type PASS
            auth_pass Add-Your-Password-Here
        }
        virtual_ipaddress {
                202.54.1.1/29 dev eth1
        }
}

Save and close the file. Finally start keepalived on both lb0 and lb1 as follows:
# /etc/init.d/keepalived start

Verify: Keepalived Working Or Not

/var/log/messages will keep track of VIP:
# tail -f /var/log/messages
Sample outputs:

Feb 21 04:06:15 lb0 Keepalived_vrrp: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:06:20 lb0 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1

Verify that VIP assigned to eth1:
# ip addr show eth1
Sample outputs:

3: eth1:  mtu 1500 qdisc pfifo_fast qlen 10000
    link/ether 00:30:48:30:30:a3 brd ff:ff:ff:ff:ff:ff
    inet 202.54.1.11/29 brd 202.54.1.254 scope global eth1
    inet 202.54.1.1/29 scope global secondary eth1

ping failover test

Open UNIX / Linux / OS X desktop terminal and type the following command to ping to VIP:
# ping 202.54.1.1
Login to lb0 and halt the server or take down networking:
# halt
Within seconds VIP should move from lb0 to lb1 and you should not see any drops in ping. On lb1 you should get the following in /var/log/messages:

Feb 21 04:10:07 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) forcing a new MASTER election
Feb 21 04:10:08 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs.
Feb 21 04:10:09 lb1 Keepalived_healthcheckers: Netlink reflector reports IP 202.54.1.1 added
Feb 21 04:10:09 lb1 Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 202.54.1.1

Conclusion

Your server is now configured with IP failover. However, you need to install and configure the following software in order to configure webserver and security:

  1. nginx or lighttpd
  2. iptables

Stay tuned, for more information on above configuration.

🐧 If you liked this page, please support my work on Patreon or with a donation.
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source/DevOps topics:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
40 comments… add one
  • lee Jul 3, 2013 @ 10:26

    Hi,
    nice clear tutorial, and confirms that my setup should be working. wonder if anyone can help with a problem i’m having, google searching hasn’t helped.
    I’ve got keepalived setup on 2 Ubuntu 12.04 servers. with 4 physical interfaces, and two vlan interfaces, eth0, eth1, eth2, eth3, plus eth2.10 and eth2.20. I have failover ip’s set on every interface, with all vrrp instances together in a single vrrp group. if I take down any interface on the master, every failover ip gets successfully moved over to the backup server. I can see in the logs that garps are sent out for each ip/interface on the backup server. I can ping every failover ip address without interruption. so far so good. but before failover, I can ping from a separate pc on any one of these subnets, to other pc’s on any of the other subnets, however, after failover, I can’t ping across the backup keepalived server to these other pc’s. I get absolutely nothing until the primary server is back up and running and has taken back the failover ip’s.
    anyone got any idea why this would be happening?

    thanks
    lee.

    • 🐧 nixCraft Jul 3, 2013 @ 12:20

      Can you use service installed on the backup system? If so check for the firewall logs.

  • Fabiano Nov 23, 2013 @ 0:00

    Hi, Nice tutorial! I´m need something like this, but working in one Linux Box, but with two links connnected on them, it´s possible to do in this way?
    E.g:

    Linux Box |—- eth0 LAN
    |—- eth1 Internet 1
    |— eth2 Internet 2
    Thanks

  • Khizer Abdul Sattar Mar 8, 2014 @ 8:02

    [root@localhost ~]# cp /usr/local/sbin/keepalived /usr/sbin/

    [root@localhost ~]# /etc/init.d/keepalived start
    Starting keepalived: [ OK ]

  • iyus simatupang May 23, 2014 @ 10:13

    dear sir,

    first of all, i have exchange mail server(CAS) and create loadbalance using network loadbalance role that build in windows server, what i want to ask you is how to do i set up keepalived loadbalance to loadbalance my mail server.

    fyi, i’ve been setup my private NIC on both server, but i don’t know how to configuration if i’m using keepalived.

    thank’s

    Iyus Simatupang

  • Roberto Jun 16, 2014 @ 14:03

    We have serious problem with Keepalived that as now cannot we solve.

    We installed and configured Keepalived on 2 HA firewalls Virtual Machines (VMWare ESXi infrastructure). Suddenly, Keepalived BACKUP instance (secondary), probably for a little unresponsiveness of network connection, go in “Transition to MASTER STATE” (read from /var/log/messages), immediately see MASTER (Received higher prio advert) and goes to Backup state “Entering BACKUP STATE”. During this sudden transition, VIP remain only on the MASTER but communications on networks managed by MASTER are lost. No communications take place until we restart Keepalived service on the MASTER. So restarting service manually works fine but surviving to this very little BACKUP fluctuation does not works. As this transition was partial (we do not see complete transition messages as when we do service restart) and/or arp advertising does not work correctly.

    Can somone help on this issue ? there is a possibilities to communicate with keealived developers group to see if this is a bug ?

    We use CentOS 6.5 with Keepalived v1.2.7 (02/21,2013) installed with you on base repository.

    Thank you very much

    we like a lot keepalived, but this is an important issue ! We have /var/log/messages parts for a normal restart and for this VM snapshot strange generated unclean transition but we do not know to correctly interpret and find what was wrong.

  • Stome Jun 26, 2014 @ 4:02

    Dear NixCraft,
    Suppose we have 3-server want to be fail-over with each other it mean they are all primary.
    Ex: i had 3-server (srv01 , srv02, srv03)
    srv01 : should primary ip and have to slave (srv02 and srv03 are slave for srv01)
    srv02 : should primary and have to slave (srv01 and sv03 are slave for srv02)
    srv03 : should primary and have to slave (srv01 and srv02 are slave for srv03)

    Thanks
    Waiting for your reply

  • Jenish Dec 22, 2014 @ 20:11

    I followed the above steps on two centos VMs to share same VIPs across them.

    Keepalived working there now.

    But when I try to ping VIP from third machine it is not able to ping VIP.

    Can you please let me know if I need to do anything specific to make this VIP to be visible to the host machine which is hosting these two VMs?

    Jenish

  • bish May 11, 2015 @ 0:27

    Why build keepalived fro scratch instead of grabbing the RPM?

    1) You prefer an older version (1.1.19 vs 1.2.13) ?

    2) You prefer to make every software config completely based on what libs you may have installed at the moment of configuration, instead of grabbing any missing libs and ensuring a consistent, auditable, repeatable and, best yet, Supportable install?

    3) You just didn’t look for a built version? (this one’s a trap)

    Hand-building everything under the sun is awesome, and arch and slack need you.

  • Jozef Oct 20, 2016 @ 8:45

    Hello, I am using keepalived with mysql but I am getting lot of warnings :
    version Keepalived v1.2.13 (05/14,2015)
    qde4bd 2016-10-19T02:14:22.425677+02:00 Keepalived_vrrp: Process [136284] didn’t respond to SIGTERM
    qde4bd 2016-10-19T02:14:27.420972+02:00 Keepalived_vrrp: Process [136289] didn’t respond to SIGTERM
    qde4bd 2016-10-19T02:20:37.435179+02:00 Keepalived_vrrp: Process [138530] didn’t respond to SIGTERM
    qde4bd 2016-10-19T02:20:42.435442+02:00 Keepalived_vrrp: Process [138534] didn’t respond to SIGTERM
    qde4bd 2016-10-19T02:26:42.486159+02:00 Keepalived_vrrp: Process [141268] didn’t respond to SIGTERM
    qde4bd 2016-10-19T02:26:47.486874+02:00 Keepalived_vrrp: Process [141274] didn’t respond to SIGTERM
    qde4bd 2016-10-19T02:26:52.487736+02:00 Keepalived_vrrp: Process [141278] didn’t respond to SIGTERM
    I can provide more details if you interested
    How I can solve this problem ?

  • Vinayak Pandey Jan 10, 2017 @ 8:10

    I have gone through this and both the servers are receiving vrrp packets from themselves only, even after disabling firewall and selinux. Both of them are receiving the VIP and have become master. How can I solve this issue?

    • 🐧 Vivek Gite Jan 10, 2017 @ 9:04

      Check for priority. It should be different on both servers.

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.