Monitor and restart Apache or lighttpd webserver when daemon is killed

When you cannot monitor your server for service availability, it is better to take help of automated monitor and restart utility. Last 4 days I was away from my server as I was enjoying my vacation. During this time due to load my lighttpd webserver died but it was restarted automatically within 2 minutes. I had utility configured for monitoring services on a Linux system called monit. It offers all features you ever needed for system monitoring and perform error recovery for UNIX like system.

Advertisement

Before monit I had my own shell and perl script for monitoring service. If service failed script will try to restart service and send an automated email to me. However monit is a superior solution.

monit is a utility for managing and monitoring processes, files, directories and devices on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. For example, monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You may use monit to monitor files, directories and devices for changes, such as timestamps changes, checksum changes or size changes.

Monit logo

You may also use monit to monitor files, directories and devices on localhost. Monit can monitor these items for changes, such as timestamps changes, checksum changes or size changes. This is also useful for security reasons you can monitor the md5 checksum of files that should not change.

Personally, I always install and configure monit on all boxes which are under my control.

Install monit under Debian or Ubuntu Linux

Use apt-get command to install monit
# apt-get install monitOR$ sudo apt-get install monit

Install monit under Red Hat enterprise Linux / CentOS Linux (source code installation)

Many distributions include monit. However monit is not included in official Red hat enterprise Linux. Just download monit source code from official web site using wget command:
# cd /opt
# wget http://www.tildeslash.com/monit/dist/monit-4.8.2.tar.gz
Untar monit
# tar -zxvf monit-4.8.2.tar.gz
# cd monit-4.8.2

Configure and compile monit:

# ./configure
# make

Install monit

# make install

Copy monit configuration file:

# cp monitrc /etc/monitrc

By default monit is located at /usr/local/bin/monit

How do I Configure monit?

monitrc is name of monit configuration file and it is by default located at /etc/monitrc location. However each distribution places file in different location: .
=> Source code installation : /etc/monitrc
=> Debian / Unentu Linux installation : /etc/monit/monitrc

Open monit configuration file and setup values as follows:
# vi /etc/monitrc

a) Run it as daemon and check the services (such as web, mysql, sshd) at 2-minute
intervals.
set daemon 120

b) Set syslog logging with the ‘daemon’ facility:
set logfile syslog facility log_daemon

c) Set mail server name to send email alert
set mailserver mail.cyberciti.biz
Set email format such as from email
set mail-format { from: alert@nixcraft.in
subject: $SERVICE $EVENT at $DATE
message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.
}

d) Now most important part, restart lighttpd or apache web server if failed or killed by Linux kernel due to any causes:
check process lighttpd with pidfile /var/run/lighttpd.pid
group lighttpd
start program = "/etc/init.d/lighttpd start"
stop program = "/etc/init.d/lighttpd stop"
if failed host 75.126.43.232 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Where,

  • check process lighttpd with pidfile /var/run/lighttpd.pid : You are specifying lighttpd pid file and daemon name
  • group lighttpd: Specify group name, which is allowed or used to start/restart lighttpd
  • start program = “/etc/init.d/lighttpd start” : Command to start lighttpd server
  • stop program = “/etc/init.d/lighttpd stop” : Command to stop lighttpd server
  • if failed host 127.0.0.1 port 80 : Server IP address and port number (80)
  • protocol http then restart : If above IP and port failed restart the webserver
  • if 5 restarts within 5 cycles then timeout : Try to restart 5 times; if monit cannot restart webserver 5 times; just time out to avoid race condition.

Here is my mysql server restart configuration directives:
check process mysqld with pidfile /var/run/mysqld/mysqld.pid
group database
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"
if failed host 127.0.0.1 port 3306 then restart
if 5 restarts within 5 cycles then timeout

Here is my sshd server configuration directives:
check process sshd with pidfile /var/run/sshd.pid
start program "/etc/init.d/sshd start"
stop program "/etc/init.d/sshd stop"
if failed host 127.0.0.1 port 22 protocol ssh then restart
if 5 restarts within 5 cycles then timeout

Here is my Apache serverrestart configuration directives:
check process httpd with pidfile /var/run/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout

Replace IP address 127.0.0.1 with your actual IP address. If you are using Debian just start monit:
# /etc/init.d/monit start

If you are using Red Hat Enterprise Linux, start monit from /etc/inittab file:
Open /etc/inittab file:
# vi /etc/inittab
Append following line:
mo:2345:respawn:/usr/local/bin/monit -Ic /etc/monitrc

Now start monit:
# init -qOR
# telinit -q

You can verify that monit is started from /var/log/messages log file:
# tail -f /var/log/messagesOutput:

Nov 21 04:39:21 server monit[8759]: Starting monit daemon
Nov 21 04:39:21 server monit[8759]: Monit started

If lighttpd died, you will see something as follows in log file:

Nov 21 04:45:13 server monit[8759]: 'lighttpd' process is not running
Nov 21 04:45:13 server monit[8759]: 'lighttpd' trying to restart
Nov 21 04:45:13 server monit[8759]: 'lighttpd' start: /etc/init.d/lighttpd

You may use monit to monitor daemon processes or similar programs running on localhost or started from /etc/init.d/ location such as
=> Apache Web Server
=> SSH Server
=> Postfix/Sendmail MTA
=> MySQL etc

Further readings

🥺 Was this helpful? Please add a comment to show your appreciation or feedback.

nixCrat Tux Pixel Penguin
Hi! 🤠
I'm Vivek Gite, and I write about Linux, macOS, Unix, IT, programming, infosec, and open source. Subscribe to my RSS feed or email newsletter for updates.

49 comments… add one
  • Yogesh Feb 19, 2010 @ 13:24

    @ Mohd. Anas –> VSFTP can’t be monitored using MONIT. Reason is as what you already know… it doesn’t create PID file 😉
    And MONIT uses PID file to learn about running process…. 🙂

    @ Vivek –> I am a big NixCraft fan. Sorry to advice you but please update the document for monitoring and managing services from a web interface.

    # vi /etc/monitrc
    set httpd port 2812 # To bind monit on port 2812 of apache
    allow admin:monit # require user ‘admin’ with password ‘monit’

    Restart monit to apply changes then open http://:2812 in web-browser.

  • 3dex May 6, 2010 @ 1:22

    Searched the net for days until I’ve found this thread and understand I must replace the localhost with the server’s IP for Apache monitoring to work… but, better late then never 🙂
    Thanks for the intel!

  • Gobnd Singh Aug 17, 2010 @ 15:00

    Hello All,
    I have configured as told by above manual , however i don’t know how to check that weather its working or not.
    can somebody guide me. how can i test that monitor is working or nit.

  • Chris Walker Aug 29, 2010 @ 1:42

    It appears the new version 5.1.1 has a bit of a different structure than explainewd here.
    I get some errors with my configuration like mo repeated to fast disabling for 5 minutes, and an unmonitor error.

    It does however, appear to be functioning, as I can watch what users login, in real time, and various SPI activity. However, I don;t know if it is monitoring the service properly to restart it, when needed.

  • Rocky Nov 25, 2010 @ 12:10

    Hi,

    I want to configuer monit to only generate alert mail for service monitoring and want to monitor only service.
    How to configure it……..?
    Because when i see monit status it showas service and local system statistics.

    Thanks
    Rocky

  • screenmates Mar 11, 2012 @ 22:04

    What if the service died but the pid file never got deleted?

  • anacondaweb Apr 27, 2012 @ 14:57

    Excelente guía.
    Gracias por compartirla.

  • Carlos Baez Apr 21, 2016 @ 13:34

    Can this be used with Asterisk?

  • Christopher Apr 28, 2016 @ 15:42

    Just because the port is open and the pid exists does not mean that a service is ‘running’ the true test for responsiveness and usability is as close to using the thing as possible… though I appreciate the tool I’ll stick with curl to check the actual site.

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre> for code samples. Your comment will appear only after approval by the site admin.