Apache Web Server: Log Analysis and Server Status Monitoring Tool
wtop is really cool application for web server log analysis and to see server stats at a glance. It also has powerful log grepping capability. It is just like 'top' for your webserver.
It can find out number of searches or signups per seconds. It can also create histogram of response time. There is also another tool called logrep a powerful command-line program for ad-hoc analysis and filtering for log files. You can dig up lots of information using wtop tools.
You need Python version 2.5 to run wtop.
Download wtop
Type the following command:
$ cd /tmp
$ wget http://wtop.googlecode.com/files/wtop-0.5.6.tar.gz
$ tar -zxvf wtop-0.5.6.tar.gz
$ cd wtop-0.5.6
# python setup.py install
Configuring wtop
Once installed you can start using the tool immediately. You need to edit /etc/wtop.cfg file to setup parameters, Apache log files and other directives
# vi /etc/wtop.cfg
Sample configuration file:
[main]
LOG_ROOT=/var/log/lighttpd/cyberciti.biz/
LOG_FILE=access.log
DEFAULT_OUTPUT_FIELDS=ts,class,ipcnt,ip,msec,uas,url
# This must match your webserver log format. You MUST have at least %h, %r and %D
LOG_FORMAT=%h %l %u %t "%r" %>s %B "%{Referer}i" "%{User-Agent}i" %D
[wtop]
# max time before a request is logged in the "slow" column
MAX_REQUEST_TIME=5000
# minimum requests/second before a URL class appears in top mode
MIN_RPS=0.2
[classes]
# you can extend these to make any classes you wish
home=^/(?:\?.*)?$
xml=\.xml(?:\?.*)?$
js=\.js(?:\?.*)?$
css=\.css(?:\?.*)?$
img=\.(?:png|gif|jpe?g|cur|ico|bmp)(?:\?.*)?$
[patterns]
# the generic pattern is applied if a line does not match any
# of the named classes. By default it uses the top-level directory.
generic=^/([^/\?]+)
# incomplete list of known web robots
robots = r'(?:nutch|MSRBOT|translate.google.com|Feedster|Nutch|Gaisbot|Snapbot|VisBot|libwww|CazoodleBot|polybot|VadixBot|Sogou|SBider|BecomeBot|Yandex|Pagebull|chudo|Pockey|nicebot|entireweb|FeedwhipBOT|ConveraCrawler|NG/2.0|WebImages|Factbot|information-online|gsa-crawler|Jyxobot|SentinelCrawler|BlogPulseLive|YahooFeedSeeker|GurujiBot|wwwster|Y\!J-SRD|Findexa|SurveyBot|yetibot|discoveryengine|fastsearch|noxtrum|Googlebot|Snapbot|OGSearchSpider|heritrix|nutch-agent|Slurp|msnbot|cuill|Mediapartners|YahooSeeker|GrabPERF|keywen|ia_archiver|crawler.archive.org|Baiduspider|larbin|shopwiki)'
Now simply type wtop at a shell prompt:
$ wtop$
See all human traffic, enter:
$ logrep -m top -h access.log
See response times for all MSNBot homepage hits:
$ logrep -m grep -g MSNBot -i home -o status,msec,url access.log
Display the current log for traffic to pages about wordpress or themes sent from google.com
$ logrep -m tail --f 'url~wordpress|themes,ref~google.com' access.log
Further readings:
E-mail this to a Friend
Printable Version
You may also be interested in other helpful articles:
- Download of the day: AutoScan Network Monitoring - Management Tool
- How To Track Changes in Your Linux Filesystem
- Suse Linux: Nagios Basic Installation and Configuration
- Find and Fix Linux Latency Problem with LatencyTOP Software
- Linux and UNIX interactive, process and users monitoring tool
Leave a Reply
We encourage your comments, and suggestions. But please stay on topic, be polite, and avoid spam. Thank you very much for stopping by our site!
Tags: apache log files, converacrawler, default output, generic pattern, google, python version, request time, snapbot, surveybot, web robots, wtop
Recent Comments
Today ~ 2 Comments
Today ~ 179 Comments
01/02/2008 10:46 pm ~ 1 Comment
Yesterday ~ 4 Comments
08/24/2006 01:56 pm ~ 11 Comments