≡ Menu


I've already written about Linux process accounting under Linux ( see how to keep a detailed audit trail of what's being done on your Linux systems). You can easily setup process accounting under FreeBSD.

FreeBSD Process Accounting

FreeBSD process accounting is a security method in which you can keep track of system resources used, their allocation among users, provide for system monitoring, and minimally track a user's commands.
[click to continue…]

Nagios: System and Network Monitoring Book

Nagios is a popular open source computer system and network monitoring application software. You can easily monitor all your hosts, network equipment and services. It can send alert when things go wrong and again when they get better.

The convenience and reliability that monitoring programs offer system administrators is astounding. Whether at home, commuting, or on vacation, admins can continuously monitor their networks, learning of issues long before they become catastrophes.

Nagios, the most popular open source solution for system and network monitoring, is extremely robust, but it's also intensely complex. This eagerly anticipated revision of the highly acclaimed Nagios: System and Network Monitoring, has been updated to address Nagios 3.0 and will help readers take full advantage of the many powerful features of the new version. Ethan Galstad, the main developer of Nagios, called the first edition of Nagios "incredibly detailed." He went on to say, "I don't think I could have gone into that much detail if I wrote a book myself."

Nagios, which runs on Linux and most *nix variants, can be configured to continuously monitor network services such as SMTP, POP3, HTTP, NNTP, SSH, and FTP. It can also supervise host resources (processor load, disk and memory usage, running processes, log files, and so on) and environmental factors, such as temperature and humidity. Readers of Nagios learn how to:

  • Install and configure the Nagios core, all standard plugins, and selected third-party plugins
  • Configure the notification system
  • Program event handlers to take automatic action when trouble occurs
  • Write Perl plugins to customize Nagios for unique system needs
  • Quickly understand Nagios data using graphing and visualization tools
  • Monitor Windows servers, SAP systems, and databases

This dense, all-inclusive guide to Nagios also contains a chapter that highlights the differences between Nagios versions 2 and 3 and gives practical migration and compatibility tips. Nagios, 2nd Edition is a key resource for any system and network administrator and will ease the pain of network monitoring migraines in no time.

Wolfgang Barth has written several books for professional network administrators, including The Firewall Book (Suse Press), Network Analysis (Suse Press), and Backup Solutions with Linux (Open Source Press). He is a professional system administrator with considerable experience using Nagios.

Book Info

  • Title: Nagios: System and Network Monitoring, 2nd Edition
  • Author: Wolfgang Barth
  • Pub Date: October 2008, 720 pp
  • ISBN 9781593271794, $59.95 USD
  • Download free chapter 18: "NagVis" (PDF)
  • Order info: order@oreilly.com // 1-800-998-9938 // 1-707-827-7000
  • Support nixCraft: Order Nagios: System and Network Monitoring from Amazon.

This is a user contributed tutorial.

Nagios is free, open source host, service and network monitoring services. Nagios provides an extensible framework, that can monitor pretty much anything using plugins. Some of the items that can be monitored using Nagios plugins are listed below.

=> Disk space usage of remote Linux and Windows server
=> CPU Usage
=> Memory usage
=> Hardware Temperature
=> VPN tunnels
=> Router and Switches
=> Databases
=> Network services (DHCP, DNS, LDAP, SMTP etc.)

Nagios Configurations are very granular and managed using following three different category of configuration files:

  • Nagios server and web console configuration files can be used to configure the Nagios server itself. For e.g. Use the nagios.cfg and cgi.cfg
  • Resource files can be used to store user defined macros and sensitive configuration informations such as passwords.
  • Object definition configuration files are used to store information about the hosts, services, commands, contacts, notification period etc.

Nagios has a web front end to display the status. Apart from getting the notification about the hosts and service status through email, SMS etc., you can also see the hosts, services, status through nagios web front end. You can project is on the NOC (Network Operation Center) to view the current status of your whole data center. You can also perform few actions on the web console such as disable and enable notification for a specific service. If you have defined the relationship between your hosts properly in the nagios configuration files, you can use the 3D display view to see a graphical representation of the whole data center visually. This also provides reporting feature where you can view the historic data such as availability of a particular service on a specific host over a period of time.

(Fig. 01 – Nagios web UI displaying status of various services on a Linux host)

Notification process on the Nagios is defined at a very granular level that it covers a wide range of possible scenarios on the notification including escalation process where a specific contact group can be notified if an issues has not been fixed after certain number of initial notifications. This is very helpful to automatically notify the management team about a critical service that was not fixed immediately.

Nagios can also be configured in a distributed setup, where datacenters from different parts of the world can be monitored using local nagios server that can report the status back to a central nagios server. This is achieved by NSCA (Nagios Service Check Acceptor) sending monitoring results from the local nagios server to the central server.

Following articles from The Geek Stuff blog, explains about everything that is required to get a jumpstart on the Nagios installation, configuration on Linux. This also explains about how to monitor Linux and Windows host.

wtop is really cool application for web server log analysis and to see server stats at a glance. It also has powerful log grepping capability. It is just like 'top' for your webserver.

It can find out number of searches or signups per seconds. It can also create histogram of response time. There is also another tool called logrep a powerful command-line program for ad-hoc analysis and filtering for log files. You can dig up lots of information using wtop tools.

You need Python version 2.5 to run wtop.

Download wtop

Type the following command:
$ cd /tmp
$ wget http://wtop.googlecode.com/files/wtop-0.5.6.tar.gz
$ tar -zxvf wtop-0.5.6.tar.gz
$ cd wtop-0.5.6
# python setup.py install

Configuring wtop

Once installed you can start using the tool immediately. You need to edit /etc/wtop.cfg file to setup parameters, Apache log files and other directives
# vi /etc/wtop.cfg
Sample configuration file:

# This must match your webserver log format. You MUST have at least %h, %r and %D
LOG_FORMAT=%h %l %u %t "%r" %>s %B "%{Referer}i" "%{User-Agent}i" %D
# max time before a request is logged in the "slow" column
# minimum requests/second before a URL class appears in top mode
# you can extend these to make any classes you wish
# the generic pattern is applied if a line does not match any
# of the named classes. By default it uses the top-level directory.
# incomplete list of known web robots
robots = r'(?:nutch|MSRBOT|translate.google.com|Feedster|Nutch|Gaisbot|Snapbot|VisBot|libwww|CazoodleBot|polybot|VadixBot|Sogou|SBider|BecomeBot|Yandex|Pagebull|chudo|Pockey|nicebot|entireweb|FeedwhipBOT|ConveraCrawler|NG/2.0|WebImages|Factbot|information-online|gsa-crawler|Jyxobot|SentinelCrawler|BlogPulseLive|YahooFeedSeeker|GurujiBot|wwwster|Y\!J-SRD|Findexa|SurveyBot|yetibot|discoveryengine|fastsearch|noxtrum|Googlebot|Snapbot|OGSearchSpider|heritrix|nutch-agent|Slurp|msnbot|cuill|Mediapartners|YahooSeeker|GrabPERF|keywen|ia_archiver|crawler.archive.org|Baiduspider|larbin|shopwiki)'

Now simply type wtop at a shell prompt:
$ wtop$
See all human traffic, enter:
$ logrep -m top -h access.log
See response times for all MSNBot homepage hits:
$ logrep -m grep -g MSNBot -i home -o status,msec,url access.log
Display the current log for traffic to pages about wordpress or themes sent from google.com
$ logrep -m tail --f 'url~wordpress|themes,ref~google.com' access.log

Further readings:

Ubuntu / Debian Linux Find Weak OpenSSL keys

This bug really was a bad one. I've client with over 200 Debian Linux server. Updating all systems wasn't the problem. With the help of Cfengine I was able to push updates but managing all workstation ssh keys (over 1000+ Windows and Linux/BSD workstations) and testing everything took so much time. Debian shouldn't have modified the package in first place. I also had to upgrade over 30 SSL certificates and a whole new CA for OpenVPN. Luckily VeriSign is providing revocation and replacement of SSL certificates (generally it is not provided free of charge) till 30-June-2008.

How do I find out all weak keys?

You can check all your weak keys with following commands:
# wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
# wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz.asc
# gpg --keyserver subkeys.pgp.net --recv-keys 02D524BE
# gpg --verify dowkd.pl.gz.asc
# gunzip dowkd.pl.gz
# perl dowkd.pl host localhost

You should see 0 weak keys. If you run Debian or Ubuntu Linux upgrade your OpenSSL and fix all the affected softwares. There is also wiki page that will address all your concerns. Overall it lasted for few days for large clients. How many hours did you spend updating Debian systems?

So how do you list the network open ports on your Linux server and the process that owns them? The answer is simple. Use the following command (must be run as the root user):

sudo lsof -i
sudo netstat -lptu
sudo netstat -tulpn

Sample outputs (see video demo):

Under Linux you can use strace or valgrind tool for reporting and finding a bug. However, under *BSD / Mac OS X you need to use ktrace as replacement for strace tool.

kreace runs on the following platforms:
=> FreeBSD
=> OpenBSD
=> Mac OS X
=> NetBSD

The ktrace utility enables kernel trace logging for the specified processes. Kernel trace data is logged to the file ktrace.out. The kernel operations that are traced include system calls, namei translations, sig nal processing, and I/O. Once tracing is enabled on a process, trace data will be logged until either the process exits or the trace point is cleared. A traced process can generate enormous amounts of log data quickly; It is strongly suggested that users memorize how to disable tracing before attempting to trace a process.

To trace all kernel operations for process id # 2546, enter:
$ ktrace -p 2546
To disable all tracing of process # 2546, enter:
$ ktrace -cp 2546
To disable tracing on all user-owned processes, and, if executed by root, all processes in the system:
# ktrace -C
Attach to process id # 123 and log trace records to myapp.dbg.log instead of ktrace.out.
$ ktrace -p 123 -f myapp.dbg.log
To enable tracing of I/O on process # 123
$ ktrace -ti -p 123
The -t option is very useful to trace various kernel trace points, one per letter. The following table equates the letters with the trace points:

  • c : trace system calls
  • n : trace namei translations
  • i : trace I/O
  • s : trace signal processing
  • u : userland traces
  • w : context switches
  • + : trace the default set of trace points - c, n, i, s, u

Run the command called myapp and track only system calls, enter:
$ ktrace -tc ./myapp
Please note that the output of ktrace is not as informative as strace, but it does help to solve many problems.

truss: trace system calls

FreeBSD has another tool called truss. It traces the system calls called by the specified process or program. Output is to the specified output file, or standard error by default.
Attach to an already-running process # 123, enter
$ truss -p 123
Follow the system calls used myapp
$ truss ./myapp -d /tmp -f 120
Same as above, but put the output into a file called /tmp/myapp.dbg
$ truss -o /tmp/truss.out ./myapp -d /tmp -f 120

strace under FreeBSD

You can install strace under FreeBSD and other *BSD like oses.

Further readings:

  • ktrace man page
  • truss man page