Monitor and restart Apache or lighttpd webserver when daemon is killed

by on November 21, 2006 · 47 comments· LAST UPDATED January 1, 2009

in , ,

When you cannot monitor your server for service availability, it is better to take help of automated monitor and restart utility. Last 4 days I was away from my server as I was enjoying my vacation. During this time due to load my lighttpd webserver died but it was restarted automatically within 2 minutes. I had utility configured for monitoring services on a Linux system called monit. It offers all features you ever needed for system monitoring and perform error recovery for UNIX like system.

Before monit I had my own shell and perl script for monitoring service. If service failed script will try to restart service and send an automated email to me. However monit is a superior solution.

monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. For example, monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.

Monit logo

You may also use monit to monitor files, directories and devices on localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons you can monitor the md5 checksum of files that should not change.

Personally, I always install and configure monit on all boxes which are under my control.

Install monit under Debian or Ubuntu Linux

Use apt-get command to install monit
# apt-get install monitOR$ sudo apt-get install monit

Install monit under Red Hat enterprise Linux / CentOS Linux (source code installation)

Many distributions include monit. However monit is not included in official Red hat enterprise Linux. Just download monit source code from official web site using wget command:
# cd /opt
# wget http://www.tildeslash.com/monit/dist/monit-4.8.2.tar.gz
Untar monit
# tar -zxvf monit-4.8.2.tar.gz
# cd monit-4.8.2

Configure and compile monit:

# ./configure
# make

Install monit

# make install

Copy monit configuration file:

# cp monitrc /etc/monitrc

By default monit is located at /usr/local/bin/monit

How do I Configure monit?

monitrc is name of monit configuration file and it is by default located at /etc/monitrc location. However each distribution places file in different location: .
=> Source code installation : /etc/monitrc
=> Debian / Unentu Linux installation : /etc/monit/monitrc

Open monit configuration file and setup values as follows:
# vi /etc/monitrc

a) Run it as daemon and check the services (such as web, mysql, sshd) at 2-minute
intervals.
set daemon 120

b) Set syslog logging with the 'daemon' facility:
set logfile syslog facility log_daemon

c) Set mail server name to send email alert
set mailserver mail.cyberciti.biz
Set email format such as from email
set mail-format { from: alert@nixcraft.in
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
}

d) Now most important part, restart lighttpd or apache web server if failed or killed by Linux kernel due to any causes:
check process lighttpd with pidfile /var/run/lighttpd.pid
group lighttpd
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if failed host 75.126.43.232 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Where,

  • check process lighttpd with pidfile /var/run/lighttpd.pid : You are specifying lighttpd pid file and daemon name
  • group lighttpd: Specify group name, which is allowed or used to start/restart lighttpd
  • start program = "/etc/init.d/lighttpd start" : Command to start lighttpd server
  • stop program = "/etc/init.d/lighttpd stop" : Command to stop lighttpd server
  • if failed host 127.0.0.1 port 80 : Server IP address and port number (80)
  • protocol http then restart : If above IP and port failed restart the webserver
  • if 5 restarts within 5 cycles then timeout : Try to restart 5 times; if monit cannot restart webserver 5 times; just time out to avoid race condition.

Here is my mysql server restart configuration directives:
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout

Here is my sshd server configuration directives:
check process sshd with pidfile /var/run/sshd.pid
start program "/etc/init.d/sshd start"
stop program "/etc/init.d/sshd stop"
if failed host 127.0.0.1 port 22 protocol ssh then restart
if 5 restarts within 5 cycles then timeout

Here is my Apache serverrestart configuration directives:
check process httpd with pidfile /var/run/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Replace IP address 127.0.0.1 with your actual IP address. If you are using Debian just start monit:
# /etc/init.d/monit start

If you are using Red Hat Enterprise Linux, start monit from /etc/inittab file:
Open /etc/inittab file:
# vi /etc/inittab
Append following line:
mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc

Now start monit:
# init -qOR
# telinit -q

You can verify that monit is started from /var/log/messages log file:
# tail -f /var/log/messagesOutput:

Nov 21 04:39:21 server monit[8759]: Starting monit daemon
Nov 21 04:39:21 server monit[8759]: Monit started

If lighttpd died, you will see something as follows in log file:

Nov 21 04:45:13 server monit[8759]: 'lighttpd' process is not running
Nov 21 04:45:13 server monit[8759]: 'lighttpd' trying to restart
Nov 21 04:45:13 server monit[8759]: 'lighttpd' start: /etc/init.d/lighttpd

You may use monit to monitor daemon processes or similar programs running on localhost or started from /etc/init.d/ location such as
=> Apache Web Server
=> SSH Server
=> Postfix/Sendmail MTA
=> MySQL etc

Further readings

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 47 comments… read them below or add one }

1 hoba November 29, 2006 at 1:28 pm

never took the time to read through the documentation. But after reading this article I say it sounds easy enough.

I will give it a try and probably abandon my custom setup. Thanks! :-)

Reply

2 Matthew Musgrove November 29, 2006 at 6:18 pm

Your untar command is uncorrect. It should be:
tar -zxvf monit-4.8.2.tar.gz

Reply

3 nixCraft November 29, 2006 at 7:10 pm

Matthew, Thanks for heads up!

It was a typo…

Reply

4 Jeff December 1, 2006 at 5:48 pm

I think “inittab -q” should actually be “telinit q”.

Thanks.

Reply

5 nixCraft December 1, 2006 at 6:28 pm

Jeff,

Yup, it was typo. You need to use init -q or telinit -q

Appreciate your post.

Reply

6 Brian December 28, 2006 at 2:18 am

Does it work with Suse linux?

Reply

7 nixCraft December 28, 2006 at 5:45 am

Brian,

Yes it does works with Sues Linux and all other UNIX like oses.

Reply

8 rick January 26, 2007 at 3:08 am

how do you configure monit to monitor apache’s load such that if reaches 5.00 and above it will automatically restart apache?

thanks in advance

Reply

9 apache February 26, 2007 at 2:41 pm

Another option, which should be a bit simpler, is to use a cron script to check the health of Apache and MySQL.

Reply

10 thecoolone March 4, 2007 at 7:24 am

Thanx for this tutorial. was very helpful and easy to understand.
But i noticed one flaw/bug with monit:
Assume that you have monit setup to monitor apache and mysql. And the section of apache comes first in the monit.conf file followed by the mysql section.
If only apache down, monit then ONLY restarts apache (since its first in the monit.conf file). But if ONLY mysql is down(apache still running), then monit first shut’s down and restart apache and the starts mysql. So in this way i cannot make monit ONLY restart the daemon that is down.

Any work around for this ??

Reply

11 nixCraft March 5, 2007 at 2:57 pm

thecoolone,

I’ve not noticed such a bug. Try to use latest version.

Reply

12 Skullz March 6, 2007 at 9:50 pm

any body tell me how can i get site domin name and password please brothers please am comming newly in hacking so please brothers help me am your brother please help me

Reply

13 Ira June 8, 2007 at 5:25 pm

This document is simple and it is easy to set up. One question though.
Can monit be set up in such way: if it can not restart a service, then send an alert to another email address, which is a pager? The line I am looking at is:

if 5 restarts within 5 cycles then timeout

Thanks for help in advance.
Ira

Reply

14 nixCraft June 8, 2007 at 5:48 pm

This is default. Monit will raise an email alert when event is timeout.

You can also try as follows:
check process httpd with pidfile /var/run/httpd.pid
alert pager@service.airtel.in only on { timeout, nonexist }

monit man page has all the example, please refer the same.

Reply

15 Ira June 8, 2007 at 8:36 pm

Thanks so much. Read through man page more.

Ira

Reply

16 jon ivar July 2, 2007 at 8:40 pm

I have a ” something.x86 ” file that I’m running as a executable

But I can’t find any “something.pid” anywhere

How do I check if ” something.x86 ” is running and how do I set it up to restart if it’s not

Best regards
jon ivar

Reply

17 nixCraft July 2, 2007 at 10:43 pm

us ps -aux command:
ps -aux | grep -i 'something'

2nd column shows PID. Generally /var/run/something stores the PID. If it is your own custom-made program, consider writing sys v init script; so that you can integrate your program with monit.

HTH

Reply

18 jon ivar July 3, 2007 at 8:42 pm

thanks vivek

Reply

19 Zubir July 4, 2007 at 7:56 am

Hi all,

How to monitor bind named server?
Is it possible to monitor this kind of server.

TQ

Reply

20 Artem Russakovskii December 11, 2007 at 2:53 am

Very useful article. I was up and running on SUSE in only a few minutes.

I completely forgot about monit and was looking for a solution to restart daemons just to find your article and realize I already know about it.

Monit is no enterprise grade solution though, I think the latest nagios is quite nice and allows daemon restarts, custom scripts, etc. Another one I’ve seen around is zabbix. Maybe someone will find these useful.

Reply

21 hw* December 27, 2007 at 6:51 am

what advantages does the monit packag have compared to adding a 2 minute crontab entry that checks and restarts the daemon and sends the email alert

Reply

22 nixCraft December 27, 2007 at 7:21 am

Reliability
Advanced configuration option
Manage process, files, directory
md5 security and checksum
Read monit docs for complete feature list. I don’t think so shell script offers all these features.

Reply

23 Tim Elam January 4, 2008 at 10:34 pm

Thanks for a great article. Just what I was looking for. FYI, Monit 4.10.1 is out now.

Reply

24 paul March 26, 2008 at 7:11 am

I have installed monit on my server. But here a problem happened, such if I wants to run a process,such as “/root/appname -m ‘pram1′”, it wokrs, but if I write it in a executable Shell file, it cannot be executed. Why?

Reply

25 paul March 28, 2008 at 4:07 am

It’s me, the last problem is resolved, it has something to do with working path. But here other question.: why I cann’t excute a shell files, the content like this:
1 cd /root/test
2 ./appname -m “param1″

Reply

26 Marcus Herou April 30, 2008 at 8:34 am

Hi.

My lighty instance only return 400 to monit. It is really strange since my app server with almost the same monit setup work flawlessly.

Any ideas ?

lighttpd-1.4.19 (compiled)
monit-4.8.1 (ubuntu-hardy)

Reply

27 yara June 13, 2008 at 8:00 am

Hi,
I have debian with apache2 compiled and mysql 5, i don’t have /var/run/xxxx.pid , how can chacke my process :

check process httpd with pidfile .....

Reply

28 nixCraft June 13, 2008 at 10:06 am

Under Debian it is /var/run/apache2.pid file.

Reply

29 drewish June 15, 2008 at 8:49 pm

For Debian Linux see monitrc config file.

Reply

30 woodsb02 August 31, 2008 at 7:06 pm

I had the same problem as Marcus Herou. My lighttpd is showing as “Connection failed” and when I look in my access.log I see that lighttpd keeps returning 400. I have no idea why.

Reply

31 Cjso December 12, 2008 at 6:33 pm

What are the reasons monit can not execute /etc/init.d/httpd service?

httpd is part of ‘root’ group.

Thanks,

Reply

32 whatvn January 1, 2009 at 4:43 pm

you miss “s” in command:
“tail -f /var/log/message”, it should be “tail -f /var/log/messages”

Thank so much for awesome post.

Reply

33 nixCraft January 1, 2009 at 5:01 pm

Thanks for the heads up.

Reply

34 unix newbie January 7, 2009 at 12:52 am

Hi,

I am a newbie and am having problems getting monit to work. I have gotten all the way to the following command but I get stuck after this:

mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc

I inserted this command into the inittab file but after that I can not simply copy and paste “init -q” into the shell terminal. I get an —INSERT— prompt and then anything I type after doesnt matter.

I know very little about Unix commands other than copying and pasting and what I can dig up on a quick google search. Im figuring that I am doing something or dont know something very elementary and would just like to get this to work.

Thank you guys/gals in advance

Reply

35 desis January 21, 2009 at 10:36 pm

I have installed on redhat Centos 5..I have changed the inittab file… but after that when I used # tail -f /var/log/messages command …I dont get the output …as describe here…actually I dont get any output related with monit … any help is appreciated … I need to install it badly… is there something I am missing..

Reply

36 Alexander D January 22, 2009 at 11:06 pm

Hello
first of all thanks for great manual. THe problem is, that i did as i descrbed here, everything was ok … until this

Now start monit:
# init -qOR
# telinit -q

Goting such output
[root@server etc]# init -q
init: invalid option -- q
Usage: init 0123456SsQqAaBbCcUu
[root@server etc]# telinit -q
telinit: invalid option -- q
Usage: telinit 0123456SsQqAaBbCcUu

and of course all steps after this

f.e. for this command

# tail -f /var/log/messages
server stop for a minute, … i stop it manually …
later check that my messages file become ~16mb

so, for now Monit don’t monitor, don’t send any alert … and don’t know what to do ?

try to find any solution, or may be unistall it

Reply

37 Alexander D January 22, 2009 at 11:08 pm

Forgot to mention that after this

got such output

Jan 22 13:32:54 server sshd[10743]: subsystem request for sftp
Jan 22 13:46:45 server proftpd[11262]: server.My-Server.org (87.241.187.91[87.241.187.91]) - FTP session opened.
Jan 22 15:02:44 server sshd[13648]: Accepted password for root from ::ffff:87.241.177.184 port 14969 ssh2
Jan 22 15:16:32 server sshd[18913]: Accepted password for root from ::ffff:87.241.177.184 port 15068 ssh2
Jan 22 15:16:33 server sshd[18913]: subsystem request for sftp
Jan 22 15:19:03 server sshd[18985]: Accepted password for root from ::ffff:87.241.177.184 port 15069 ssh2
Jan 22 15:19:04 server sshd[18985]: subsystem request for sftp
Jan 22 15:23:33 server sshd[19137]: Accepted password for root from ::ffff:87.241.177.184 port 15070 ssh2
Jan 22 15:23:33 server sshd[19137]: subsystem request for sftp
Jan 22 15:44:14 server sshd[19743]: Accepted password for root from ::ffff:87.241.177.184 port 15308 ssh2

btw 87.241.177.184 isn’t my iP :-(

Reply

38 Mfawa Alfred Onen May 29, 2009 at 7:05 pm

Great Post nixcraft. Just Check on Some of your Typos. Keep it Up!

Reply

39 Martin June 20, 2009 at 4:09 pm

Gracias por la Guia, me ha sido de gran ayuda :)

Reply

40 Mohd Anas October 26, 2009 at 10:41 am

Hi,

Can someone tell me how to monitor vsftpd.
I’m not getting its pid file. I’m using CentOS 5.3.

Thanks

Reply

41 Yogesh February 19, 2010 at 1:24 pm

@ Mohd. Anas –> VSFTP can’t be monitored using MONIT. Reason is as what you already know… it doesn’t create PID file ;)
And MONIT uses PID file to learn about running process…. :)

@ Vivek –> I am a big NixCraft fan. Sorry to advice you but please update the document for monitoring and managing services from a web interface.

# vi /etc/monitrc
set httpd port 2812 # To bind monit on port 2812 of apache
allow admin:monit # require user ‘admin’ with password ‘monit’

Restart monit to apply changes then open http://:2812 in web-browser.

Reply

42 3dex May 6, 2010 at 1:22 am

Searched the net for days until I’ve found this thread and understand I must replace the localhost with the server’s IP for Apache monitoring to work… but, better late then never :)
Thanks for the intel!

Reply

43 Gobnd Singh August 17, 2010 at 3:00 pm

Hello All,
I have configured as told by above manual , however i don’t know how to check that weather its working or not.
can somebody guide me. how can i test that monitor is working or nit.

Reply

44 Chris Walker August 29, 2010 at 1:42 am

It appears the new version 5.1.1 has a bit of a different structure than explainewd here.
I get some errors with my configuration like mo repeated to fast disabling for 5 minutes, and an unmonitor error.

It does however, appear to be functioning, as I can watch what users login, in real time, and various SPI activity. However, I don;t know if it is monitoring the service properly to restart it, when needed.

Reply

45 Rocky November 25, 2010 at 12:10 pm

Hi,

I want to configuer monit to only generate alert mail for service monitoring and want to monitor only service.
How to configure it……..?
Because when i see monit status it showas service and local system statistics.

Thanks
Rocky

Reply

46 screenmates March 11, 2012 at 10:04 pm

What if the service died but the pid file never got deleted?

Reply

47 anacondaweb April 27, 2012 at 2:57 pm

Excelente guía.
Gracias por compartirla.

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , , , , ,

Previous post:

Next post: