Monitor and restart Apache or lighttpd webserver when daemon is killed

Posted on in Categories Apache, CentOS, Debian Linux, FreeBSD, GNU/Open source, Howto, lighttpd, Linux, Monitoring, MySQL, Networking, Postfix, RedHat/Fedora Linux, Security, Shell scripting, Suse Linux, Sys admin, UNIX last updated November 21, 2006

When you cannot monitor your server for service availability, it is better to take help of automated monitor and restart utility. Last 4 days I was away from my server as I was enjoying my vacation. During this time due to load my lighttpd webserver died but it was restarted automatically within 2 minutes. I had utility configured for monitoring services on a Linux system called monit. It offers all features you ever needed for system monitoring and perform error recovery for UNIX like system.

Before monit I had my own shell and perl script for monitoring service. If service failed script will try to restart service and send an automated email to me. However monit is a superior solution.

monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. For example, monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.

Monit logo

You may also use monit to monitor files, directories and devices on localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons you can monitor the md5 checksum of files that should not change.

Personally, I always install and configure monit on all boxes which are under my control.

Install monit under Debian or Ubuntu Linux

Use apt-get command to install monit
# apt-get install monitOR$ sudo apt-get install monit

Install monit under Red Hat enterprise Linux / CentOS Linux (source code installation)

Many distributions include monit. However monit is not included in official Red hat enterprise Linux. Just download monit source code from official web site using wget command:
# cd /opt
# wget http://www.tildeslash.com/monit/dist/monit-4.8.2.tar.gz
Untar monit
# tar -zxvf monit-4.8.2.tar.gz
# cd monit-4.8.2

Configure and compile monit:

# ./configure
# make

Install monit

# make install

Copy monit configuration file:

# cp monitrc /etc/monitrc

By default monit is located at /usr/local/bin/monit

How do I Configure monit?

monitrc is name of monit configuration file and it is by default located at /etc/monitrc location. However each distribution places file in different location: .
=> Source code installation : /etc/monitrc
=> Debian / Unentu Linux installation : /etc/monit/monitrc

Open monit configuration file and setup values as follows:
# vi /etc/monitrc

a) Run it as daemon and check the services (such as web, mysql, sshd) at 2-minute
intervals.
set daemon 120

b) Set syslog logging with the ‘daemon’ facility:
set logfile syslog facility log_daemon

c) Set mail server name to send email alert
set mailserver mail.cyberciti.biz
Set email format such as from email
set mail-format { from: [email protected]
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
}

d) Now most important part, restart lighttpd or apache web server if failed or killed by Linux kernel due to any causes:
check process lighttpd with pidfile /var/run/lighttpd.pid
group lighttpd
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if failed host 75.126.43.232 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Where,

  • check process lighttpd with pidfile /var/run/lighttpd.pid : You are specifying lighttpd pid file and daemon name
  • group lighttpd: Specify group name, which is allowed or used to start/restart lighttpd
  • start program = “/etc/init.d/lighttpd start” : Command to start lighttpd server
  • stop program = “/etc/init.d/lighttpd stop” : Command to stop lighttpd server
  • if failed host 127.0.0.1 port 80 : Server IP address and port number (80)
  • protocol http then restart : If above IP and port failed restart the webserver
  • if 5 restarts within 5 cycles then timeout : Try to restart 5 times; if monit cannot restart webserver 5 times; just time out to avoid race condition.

Here is my mysql server restart configuration directives:
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout

Here is my sshd server configuration directives:
check process sshd with pidfile /var/run/sshd.pid
start program "/etc/init.d/sshd start"
stop program "/etc/init.d/sshd stop"
if failed host 127.0.0.1 port 22 protocol ssh then restart
if 5 restarts within 5 cycles then timeout

Here is my Apache serverrestart configuration directives:
check process httpd with pidfile /var/run/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Replace IP address 127.0.0.1 with your actual IP address. If you are using Debian just start monit:
# /etc/init.d/monit start

If you are using Red Hat Enterprise Linux, start monit from /etc/inittab file:
Open /etc/inittab file:
# vi /etc/inittab
Append following line:
mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc

Now start monit:
# init -qOR
# telinit -q

You can verify that monit is started from /var/log/messages log file:
# tail -f /var/log/messagesOutput:

Nov 21 04:39:21 server monit[8759]: Starting monit daemon
Nov 21 04:39:21 server monit[8759]: Monit started

If lighttpd died, you will see something as follows in log file:

Nov 21 04:45:13 server monit[8759]: 'lighttpd' process is not running
Nov 21 04:45:13 server monit[8759]: 'lighttpd' trying to restart
Nov 21 04:45:13 server monit[8759]: 'lighttpd' start: /etc/init.d/lighttpd

You may use monit to monitor daemon processes or similar programs running on localhost or started from /etc/init.d/ location such as
=> Apache Web Server
=> SSH Server
=> Postfix/Sendmail MTA
=> MySQL etc

Further readings

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

Share this on (or read 49 comments/add one below):

49 comment

  1. never took the time to read through the documentation. But after reading this article I say it sounds easy enough.

    I will give it a try and probably abandon my custom setup. Thanks! ๐Ÿ™‚

  2. Thanx for this tutorial. was very helpful and easy to understand.
    But i noticed one flaw/bug with monit:
    Assume that you have monit setup to monitor apache and mysql. And the section of apache comes first in the monit.conf file followed by the mysql section.
    If only apache down, monit then ONLY restarts apache (since its first in the monit.conf file). But if ONLY mysql is down(apache still running), then monit first shut’s down and restart apache and the starts mysql. So in this way i cannot make monit ONLY restart the daemon that is down.

    Any work around for this ??

  3. This document is simple and it is easy to set up. One question though.
    Can monit be set up in such way: if it can not restart a service, then send an alert to another email address, which is a pager? The line I am looking at is:

    if 5 restarts within 5 cycles then timeout

    Thanks for help in advance.
    Ira

  4. I have a ” something.x86 ” file that I’m running as a executable

    But I can’t find any “something.pid” anywhere

    How do I check if ” something.x86 ” is running and how do I set it up to restart if it’s not

    Best regards
    jon ivar

  5. us ps -aux command:
    ps -aux | grep -i 'something'

    2nd column shows PID. Generally /var/run/something stores the PID. If it is your own custom-made program, consider writing sys v init script; so that you can integrate your program with monit.

    HTH

  6. Very useful article. I was up and running on SUSE in only a few minutes.

    I completely forgot about monit and was looking for a solution to restart daemons just to find your article and realize I already know about it.

    Monit is no enterprise grade solution though, I think the latest nagios is quite nice and allows daemon restarts, custom scripts, etc. Another one I’ve seen around is zabbix. Maybe someone will find these useful.

  7. I have installed monit on my server. But here a problem happened, such if I wants to run a process,such as “/root/appname -m ‘pram1′”, it wokrs, but if I write it in a executable Shell file, it cannot be executed. Why?

  8. It’s me, the last problem is resolved, it has something to do with working path. But here other question.: why I cann’t excute a shell files, the content like this:
    1 cd /root/test
    2 ./appname -m “param1”

  9. I had the same problem as Marcus Herou. My lighttpd is showing as “Connection failed” and when I look in my access.log I see that lighttpd keeps returning 400. I have no idea why.

  10. Hi,

    I am a newbie and am having problems getting monit to work. I have gotten all the way to the following command but I get stuck after this:

    mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc

    I inserted this command into the inittab file but after that I can not simply copy and paste “init -q” into the shell terminal. I get an —INSERT— prompt and then anything I type after doesnt matter.

    I know very little about Unix commands other than copying and pasting and what I can dig up on a quick google search. Im figuring that I am doing something or dont know something very elementary and would just like to get this to work.

    Thank you guys/gals in advance

  11. I have installed on redhat Centos 5..I have changed the inittab file… but after that when I used # tail -f /var/log/messages command …I dont get the output …as describe here…actually I dont get any output related with monit … any help is appreciated … I need to install it badly… is there something I am missing..

  12. Hello
    first of all thanks for great manual. THe problem is, that i did as i descrbed here, everything was ok … until this

    Now start monit:
    # init -qOR
    # telinit -q

    Goting such output
    [[email protected] etc]# init -q
    init: invalid option -- q
    Usage: init 0123456SsQqAaBbCcUu
    [[email protected] etc]# telinit -q
    telinit: invalid option -- q
    Usage: telinit 0123456SsQqAaBbCcUu

    and of course all steps after this

    f.e. for this command

    # tail -f /var/log/messages
    server stop for a minute, … i stop it manually …
    later check that my messages file become ~16mb

    so, for now Monit don’t monitor, don’t send any alert … and don’t know what to do ?

    try to find any solution, or may be unistall it

  13. Forgot to mention that after this

    got such output

    Jan 22 13:32:54 server sshd[10743]: subsystem request for sftp
    Jan 22 13:46:45 server proftpd[11262]: server.My-Server.org (87.241.187.91[87.241.187.91]) - FTP session opened.
    Jan 22 15:02:44 server sshd[13648]: Accepted password for root from ::ffff:87.241.177.184 port 14969 ssh2
    Jan 22 15:16:32 server sshd[18913]: Accepted password for root from ::ffff:87.241.177.184 port 15068 ssh2
    Jan 22 15:16:33 server sshd[18913]: subsystem request for sftp
    Jan 22 15:19:03 server sshd[18985]: Accepted password for root from ::ffff:87.241.177.184 port 15069 ssh2
    Jan 22 15:19:04 server sshd[18985]: subsystem request for sftp
    Jan 22 15:23:33 server sshd[19137]: Accepted password for root from ::ffff:87.241.177.184 port 15070 ssh2
    Jan 22 15:23:33 server sshd[19137]: subsystem request for sftp
    Jan 22 15:44:14 server sshd[19743]: Accepted password for root from ::ffff:87.241.177.184 port 15308 ssh2

    btw 87.241.177.184 isn’t my iP ๐Ÿ™

  14. @ Mohd. Anas –> VSFTP can’t be monitored using MONIT. Reason is as what you already know… it doesn’t create PID file ๐Ÿ˜‰
    And MONIT uses PID file to learn about running process…. ๐Ÿ™‚

    @ Vivek –> I am a big NixCraft fan. Sorry to advice you but please update the document for monitoring and managing services from a web interface.

    # vi /etc/monitrc
    set httpd port 2812 # To bind monit on port 2812 of apache
    allow admin:monit # require user ‘admin’ with password ‘monit’

    Restart monit to apply changes then open http://:2812 in web-browser.

  15. Searched the net for days until I’ve found this thread and understand I must replace the localhost with the server’s IP for Apache monitoring to work… but, better late then never ๐Ÿ™‚
    Thanks for the intel!

  16. Hello All,
    I have configured as told by above manual , however i don’t know how to check that weather its working or not.
    can somebody guide me. how can i test that monitor is working or nit.

  17. It appears the new version 5.1.1 has a bit of a different structure than explainewd here.
    I get some errors with my configuration like mo repeated to fast disabling for 5 minutes, and an unmonitor error.

    It does however, appear to be functioning, as I can watch what users login, in real time, and various SPI activity. However, I don;t know if it is monitoring the service properly to restart it, when needed.

  18. Hi,

    I want to configuer monit to only generate alert mail for service monitoring and want to monitor only service.
    How to configure it……..?
    Because when i see monit status it showas service and local system statistics.

    Thanks
    Rocky

  19. Just because the port is open and the pid exists does not mean that a service is ‘running’ the true test for responsiveness and usability is as close to using the thing as possible… though I appreciate the tool I’ll stick with curl to check the actual site.

Leave a Comment