≡ Menu

Monitor and restart Apache or lighttpd webserver when daemon is killed

When you cannot monitor your server for service availability, it is better to take help of automated monitor and restart utility. Last 4 days I was away from my server as I was enjoying my vacation. During this time due to load my lighttpd webserver died but it was restarted automatically within 2 minutes. I had utility configured for monitoring services on a Linux system called monit. It offers all features you ever needed for system monitoring and perform error recovery for UNIX like system.

Before monit I had my own shell and perl script for monitoring service. If service failed script will try to restart service and send an automated email to me. However monit is a superior solution.

monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. For example, monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.

Monit logo

You may also use monit to monitor files, directories and devices on localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons you can monitor the md5 checksum of files that should not change.

Personally, I always install and configure monit on all boxes which are under my control.

Install monit under Debian or Ubuntu Linux

Use apt-get command to install monit
# apt-get install monitOR$ sudo apt-get install monit

Install monit under Red Hat enterprise Linux / CentOS Linux (source code installation)

Many distributions include monit. However monit is not included in official Red hat enterprise Linux. Just download monit source code from official web site using wget command:
# cd /opt
# wget http://www.tildeslash.com/monit/dist/monit-4.8.2.tar.gz
Untar monit
# tar -zxvf monit-4.8.2.tar.gz
# cd monit-4.8.2

Configure and compile monit:

# ./configure
# make

Install monit

# make install

Copy monit configuration file:

# cp monitrc /etc/monitrc

By default monit is located at /usr/local/bin/monit

How do I Configure monit?

monitrc is name of monit configuration file and it is by default located at /etc/monitrc location. However each distribution places file in different location: .
=> Source code installation : /etc/monitrc
=> Debian / Unentu Linux installation : /etc/monit/monitrc

Open monit configuration file and setup values as follows:
# vi /etc/monitrc

a) Run it as daemon and check the services (such as web, mysql, sshd) at 2-minute
intervals.
set daemon 120

b) Set syslog logging with the 'daemon' facility:
set logfile syslog facility log_daemon

c) Set mail server name to send email alert
set mailserver mail.cyberciti.biz
Set email format such as from email
set mail-format { from: alert@nixcraft.in
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
}

d) Now most important part, restart lighttpd or apache web server if failed or killed by Linux kernel due to any causes:
check process lighttpd with pidfile /var/run/lighttpd.pid
group lighttpd
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if failed host 75.126.43.232 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Where,

  • check process lighttpd with pidfile /var/run/lighttpd.pid : You are specifying lighttpd pid file and daemon name
  • group lighttpd: Specify group name, which is allowed or used to start/restart lighttpd
  • start program = "/etc/init.d/lighttpd start" : Command to start lighttpd server
  • stop program = "/etc/init.d/lighttpd stop" : Command to stop lighttpd server
  • if failed host 127.0.0.1 port 80 : Server IP address and port number (80)
  • protocol http then restart : If above IP and port failed restart the webserver
  • if 5 restarts within 5 cycles then timeout : Try to restart 5 times; if monit cannot restart webserver 5 times; just time out to avoid race condition.

Here is my mysql server restart configuration directives:
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout

Here is my sshd server configuration directives:
check process sshd with pidfile /var/run/sshd.pid
start program "/etc/init.d/sshd start"
stop program "/etc/init.d/sshd stop"
if failed host 127.0.0.1 port 22 protocol ssh then restart
if 5 restarts within 5 cycles then timeout

Here is my Apache serverrestart configuration directives:
check process httpd with pidfile /var/run/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Replace IP address 127.0.0.1 with your actual IP address. If you are using Debian just start monit:
# /etc/init.d/monit start

If you are using Red Hat Enterprise Linux, start monit from /etc/inittab file:
Open /etc/inittab file:
# vi /etc/inittab
Append following line:
mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc

Now start monit:
# init -qOR
# telinit -q

You can verify that monit is started from /var/log/messages log file:
# tail -f /var/log/messagesOutput:

Nov 21 04:39:21 server monit[8759]: Starting monit daemon
Nov 21 04:39:21 server monit[8759]: Monit started

If lighttpd died, you will see something as follows in log file:

Nov 21 04:45:13 server monit[8759]: 'lighttpd' process is not running
Nov 21 04:45:13 server monit[8759]: 'lighttpd' trying to restart
Nov 21 04:45:13 server monit[8759]: 'lighttpd' start: /etc/init.d/lighttpd

You may use monit to monitor daemon processes or similar programs running on localhost or started from /etc/init.d/ location such as
=> Apache Web Server
=> SSH Server
=> Postfix/Sendmail MTA
=> MySQL etc

Further readings

Tweet itFacebook itGoogle+ itPDF itFound an error/typo on this page?

Comments on this entry are closed.

  • hoba November 29, 2006, 1:28 pm

    never took the time to read through the documentation. But after reading this article I say it sounds easy enough.

    I will give it a try and probably abandon my custom setup. Thanks! :-)

  • Matthew Musgrove November 29, 2006, 6:18 pm

    Your untar command is uncorrect. It should be:
    tar -zxvf monit-4.8.2.tar.gz

  • nixCraft November 29, 2006, 7:10 pm

    Matthew, Thanks for heads up!

    It was a typo…

  • Jeff December 1, 2006, 5:48 pm

    I think “inittab -q” should actually be “telinit q”.

    Thanks.

  • nixCraft December 1, 2006, 6:28 pm

    Jeff,

    Yup, it was typo. You need to use init -q or telinit -q

    Appreciate your post.

  • Brian December 28, 2006, 2:18 am

    Does it work with Suse linux?

  • nixCraft December 28, 2006, 5:45 am

    Brian,

    Yes it does works with Sues Linux and all other UNIX like oses.

  • rick January 26, 2007, 3:08 am

    how do you configure monit to monitor apache’s load such that if reaches 5.00 and above it will automatically restart apache?

    thanks in advance

  • apache February 26, 2007, 2:41 pm

    Another option, which should be a bit simpler, is to use a cron script to check the health of Apache and MySQL.

  • thecoolone March 4, 2007, 7:24 am

    Thanx for this tutorial. was very helpful and easy to understand.
    But i noticed one flaw/bug with monit:
    Assume that you have monit setup to monitor apache and mysql. And the section of apache comes first in the monit.conf file followed by the mysql section.
    If only apache down, monit then ONLY restarts apache (since its first in the monit.conf file). But if ONLY mysql is down(apache still running), then monit first shut’s down and restart apache and the starts mysql. So in this way i cannot make monit ONLY restart the daemon that is down.

    Any work around for this ??

  • nixCraft March 5, 2007, 2:57 pm

    thecoolone,

    I’ve not noticed such a bug. Try to use latest version.

  • Skullz March 6, 2007, 9:50 pm

    any body tell me how can i get site domin name and password please brothers please am comming newly in hacking so please brothers help me am your brother please help me

  • Ira June 8, 2007, 5:25 pm

    This document is simple and it is easy to set up. One question though.
    Can monit be set up in such way: if it can not restart a service, then send an alert to another email address, which is a pager? The line I am looking at is:

    if 5 restarts within 5 cycles then timeout

    Thanks for help in advance.
    Ira

  • nixCraft June 8, 2007, 5:48 pm

    This is default. Monit will raise an email alert when event is timeout.

    You can also try as follows:
    check process httpd with pidfile /var/run/httpd.pid
    alert pager@service.airtel.in only on { timeout, nonexist }

    monit man page has all the example, please refer the same.

  • Ira June 8, 2007, 8:36 pm

    Thanks so much. Read through man page more.

    Ira

  • jon ivar July 2, 2007, 8:40 pm

    I have a ” something.x86 ” file that I’m running as a executable

    But I can’t find any “something.pid” anywhere

    How do I check if ” something.x86 ” is running and how do I set it up to restart if it’s not

    Best regards
    jon ivar

  • nixCraft July 2, 2007, 10:43 pm

    us ps -aux command:
    ps -aux | grep -i 'something'

    2nd column shows PID. Generally /var/run/something stores the PID. If it is your own custom-made program, consider writing sys v init script; so that you can integrate your program with monit.

    HTH

  • jon ivar July 3, 2007, 8:42 pm

    thanks vivek

  • Zubir July 4, 2007, 7:56 am

    Hi all,

    How to monitor bind named server?
    Is it possible to monitor this kind of server.

    TQ

  • Artem Russakovskii December 11, 2007, 2:53 am

    Very useful article. I was up and running on SUSE in only a few minutes.

    I completely forgot about monit and was looking for a solution to restart daemons just to find your article and realize I already know about it.

    Monit is no enterprise grade solution though, I think the latest nagios is quite nice and allows daemon restarts, custom scripts, etc. Another one I’ve seen around is zabbix. Maybe someone will find these useful.

  • hw* December 27, 2007, 6:51 am

    what advantages does the monit packag have compared to adding a 2 minute crontab entry that checks and restarts the daemon and sends the email alert

  • nixCraft December 27, 2007, 7:21 am

    Reliability
    Advanced configuration option
    Manage process, files, directory
    md5 security and checksum
    Read monit docs for complete feature list. I don’t think so shell script offers all these features.

  • Tim Elam January 4, 2008, 10:34 pm

    Thanks for a great article. Just what I was looking for. FYI, Monit 4.10.1 is out now.

  • paul March 26, 2008, 7:11 am

    I have installed monit on my server. But here a problem happened, such if I wants to run a process,such as “/root/appname -m ‘pram1′”, it wokrs, but if I write it in a executable Shell file, it cannot be executed. Why?

  • paul March 28, 2008, 4:07 am

    It’s me, the last problem is resolved, it has something to do with working path. But here other question.: why I cann’t excute a shell files, the content like this:
    1 cd /root/test
    2 ./appname -m “param1″

  • Marcus Herou April 30, 2008, 8:34 am

    Hi.

    My lighty instance only return 400 to monit. It is really strange since my app server with almost the same monit setup work flawlessly.

    Any ideas ?

    lighttpd-1.4.19 (compiled)
    monit-4.8.1 (ubuntu-hardy)

  • yara June 13, 2008, 8:00 am

    Hi,
    I have debian with apache2 compiled and mysql 5, i don’t have /var/run/xxxx.pid , how can chacke my process :

    check process httpd with pidfile .....

  • nixCraft June 13, 2008, 10:06 am

    Under Debian it is /var/run/apache2.pid file.

  • drewish June 15, 2008, 8:49 pm

    For Debian Linux see monitrc config file.

  • woodsb02 August 31, 2008, 7:06 pm

    I had the same problem as Marcus Herou. My lighttpd is showing as “Connection failed” and when I look in my access.log I see that lighttpd keeps returning 400. I have no idea why.

  • Cjso December 12, 2008, 6:33 pm

    What are the reasons monit can not execute /etc/init.d/httpd service?

    httpd is part of ‘root’ group.

    Thanks,

  • whatvn January 1, 2009, 4:43 pm

    you miss “s” in command:
    “tail -f /var/log/message”, it should be “tail -f /var/log/messages”

    Thank so much for awesome post.

  • nixCraft January 1, 2009, 5:01 pm

    Thanks for the heads up.

  • unix newbie January 7, 2009, 12:52 am

    Hi,

    I am a newbie and am having problems getting monit to work. I have gotten all the way to the following command but I get stuck after this:

    mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc

    I inserted this command into the inittab file but after that I can not simply copy and paste “init -q” into the shell terminal. I get an —INSERT— prompt and then anything I type after doesnt matter.

    I know very little about Unix commands other than copying and pasting and what I can dig up on a quick google search. Im figuring that I am doing something or dont know something very elementary and would just like to get this to work.

    Thank you guys/gals in advance

  • desis January 21, 2009, 10:36 pm

    I have installed on redhat Centos 5..I have changed the inittab file… but after that when I used # tail -f /var/log/messages command …I dont get the output …as describe here…actually I dont get any output related with monit … any help is appreciated … I need to install it badly… is there something I am missing..

  • Alexander D January 22, 2009, 11:06 pm

    Hello
    first of all thanks for great manual. THe problem is, that i did as i descrbed here, everything was ok … until this

    Now start monit:
    # init -qOR
    # telinit -q

    Goting such output
    [root@server etc]# init -q
    init: invalid option -- q
    Usage: init 0123456SsQqAaBbCcUu
    [root@server etc]# telinit -q
    telinit: invalid option -- q
    Usage: telinit 0123456SsQqAaBbCcUu

    and of course all steps after this

    f.e. for this command

    # tail -f /var/log/messages
    server stop for a minute, … i stop it manually …
    later check that my messages file become ~16mb

    so, for now Monit don’t monitor, don’t send any alert … and don’t know what to do ?

    try to find any solution, or may be unistall it

  • Alexander D January 22, 2009, 11:08 pm

    Forgot to mention that after this

    got such output

    Jan 22 13:32:54 server sshd[10743]: subsystem request for sftp
    Jan 22 13:46:45 server proftpd[11262]: server.My-Server.org (87.241.187.91[87.241.187.91]) - FTP session opened.
    Jan 22 15:02:44 server sshd[13648]: Accepted password for root from ::ffff:87.241.177.184 port 14969 ssh2
    Jan 22 15:16:32 server sshd[18913]: Accepted password for root from ::ffff:87.241.177.184 port 15068 ssh2
    Jan 22 15:16:33 server sshd[18913]: subsystem request for sftp
    Jan 22 15:19:03 server sshd[18985]: Accepted password for root from ::ffff:87.241.177.184 port 15069 ssh2
    Jan 22 15:19:04 server sshd[18985]: subsystem request for sftp
    Jan 22 15:23:33 server sshd[19137]: Accepted password for root from ::ffff:87.241.177.184 port 15070 ssh2
    Jan 22 15:23:33 server sshd[19137]: subsystem request for sftp
    Jan 22 15:44:14 server sshd[19743]: Accepted password for root from ::ffff:87.241.177.184 port 15308 ssh2

    btw 87.241.177.184 isn’t my iP :-(

  • Mfawa Alfred Onen May 29, 2009, 7:05 pm

    Great Post nixcraft. Just Check on Some of your Typos. Keep it Up!

  • Martin June 20, 2009, 4:09 pm

    Gracias por la Guia, me ha sido de gran ayuda :)

  • Mohd Anas October 26, 2009, 10:41 am

    Hi,

    Can someone tell me how to monitor vsftpd.
    I’m not getting its pid file. I’m using CentOS 5.3.

    Thanks

  • Yogesh February 19, 2010, 1:24 pm

    @ Mohd. Anas –> VSFTP can’t be monitored using MONIT. Reason is as what you already know… it doesn’t create PID file ;)
    And MONIT uses PID file to learn about running process…. :)

    @ Vivek –> I am a big NixCraft fan. Sorry to advice you but please update the document for monitoring and managing services from a web interface.

    # vi /etc/monitrc
    set httpd port 2812 # To bind monit on port 2812 of apache
    allow admin:monit # require user ‘admin’ with password ‘monit’

    Restart monit to apply changes then open http://:2812 in web-browser.

  • 3dex May 6, 2010, 1:22 am

    Searched the net for days until I’ve found this thread and understand I must replace the localhost with the server’s IP for Apache monitoring to work… but, better late then never :)
    Thanks for the intel!

  • Gobnd Singh August 17, 2010, 3:00 pm

    Hello All,
    I have configured as told by above manual , however i don’t know how to check that weather its working or not.
    can somebody guide me. how can i test that monitor is working or nit.

  • Chris Walker August 29, 2010, 1:42 am

    It appears the new version 5.1.1 has a bit of a different structure than explainewd here.
    I get some errors with my configuration like mo repeated to fast disabling for 5 minutes, and an unmonitor error.

    It does however, appear to be functioning, as I can watch what users login, in real time, and various SPI activity. However, I don;t know if it is monitoring the service properly to restart it, when needed.

  • Rocky November 25, 2010, 12:10 pm

    Hi,

    I want to configuer monit to only generate alert mail for service monitoring and want to monitor only service.
    How to configure it……..?
    Because when i see monit status it showas service and local system statistics.

    Thanks
    Rocky

  • screenmates March 11, 2012, 10:04 pm

    What if the service died but the pid file never got deleted?

  • anacondaweb April 27, 2012, 2:57 pm

    Excelente guía.
    Gracias por compartirla.