≡ Menu

python version

wtop is really cool application for web server log analysis and to see server stats at a glance. It also has powerful log grepping capability. It is just like 'top' for your webserver.

It can find out number of searches or signups per seconds. It can also create histogram of response time. There is also another tool called logrep a powerful command-line program for ad-hoc analysis and filtering for log files. You can dig up lots of information using wtop tools.

You need Python version 2.5 to run wtop.

Download wtop

Type the following command:
$ cd /tmp
$ wget http://wtop.googlecode.com/files/wtop-0.5.6.tar.gz
$ tar -zxvf wtop-0.5.6.tar.gz
$ cd wtop-0.5.6
# python setup.py install

Configuring wtop

Once installed you can start using the tool immediately. You need to edit /etc/wtop.cfg file to setup parameters, Apache log files and other directives
# vi /etc/wtop.cfg
Sample configuration file:

[main]
LOG_ROOT=/var/log/lighttpd/cyberciti.biz/
LOG_FILE=access.log
DEFAULT_OUTPUT_FIELDS=ts,class,ipcnt,ip,msec,uas,url
# This must match your webserver log format. You MUST have at least %h, %r and %D
LOG_FORMAT=%h %l %u %t "%r" %>s %B "%{Referer}i" "%{User-Agent}i" %D
[wtop]
# max time before a request is logged in the "slow" column
MAX_REQUEST_TIME=5000
# minimum requests/second before a URL class appears in top mode
MIN_RPS=0.2
[classes]
# you can extend these to make any classes you wish
home=^/(?:\?.*)?$
xml=\.xml(?:\?.*)?$
js=\.js(?:\?.*)?$
css=\.css(?:\?.*)?$
img=\.(?:png|gif|jpe?g|cur|ico|bmp)(?:\?.*)?$
[patterns]
# the generic pattern is applied if a line does not match any
# of the named classes. By default it uses the top-level directory.
generic=^/([^/\?]+)
# incomplete list of known web robots
robots = r'(?:nutch|MSRBOT|translate.google.com|Feedster|Nutch|Gaisbot|Snapbot|VisBot|libwww|CazoodleBot|polybot|VadixBot|Sogou|SBider|BecomeBot|Yandex|Pagebull|chudo|Pockey|nicebot|entireweb|FeedwhipBOT|ConveraCrawler|NG/2.0|WebImages|Factbot|information-online|gsa-crawler|Jyxobot|SentinelCrawler|BlogPulseLive|YahooFeedSeeker|GurujiBot|wwwster|Y\!J-SRD|Findexa|SurveyBot|yetibot|discoveryengine|fastsearch|noxtrum|Googlebot|Snapbot|OGSearchSpider|heritrix|nutch-agent|Slurp|msnbot|cuill|Mediapartners|YahooSeeker|GrabPERF|keywen|ia_archiver|crawler.archive.org|Baiduspider|larbin|shopwiki)'

Now simply type wtop at a shell prompt:
$ wtop$
See all human traffic, enter:
$ logrep -m top -h access.log
See response times for all MSNBot homepage hits:
$ logrep -m grep -g MSNBot -i home -o status,msec,url access.log
Display the current log for traffic to pages about wordpress or themes sent from google.com
$ logrep -m tail --f 'url~wordpress|themes,ref~google.com' access.log

Further readings:

Linux > Command line BitTorrent client

BitTorrent is the name of a peer-to-peer (P2P) file distribution protocol, and of a free software implementation of that protocol.

The BitTorrent client is a computer program developed by BitTorrent, Inc. used to download and upload files via the BitTorrent protocol. You can download python based command line BitTorrent client download for Linux/BSD/Mac OS X/UNIX link oses. This is quite useful, if you would like to download file remotely over SSH session. bittorrent.com offers bittorrent client in .deb (Debian Linux) or rpm (RedHat/Fedora Core/CentOS Linux) file formats.

To use the bittorrent in command line one you need to use command bittorrent-console or bittorrent-curses (see below for examples).

Step # 1: Find Python version

Make sure you download file according to your python version. Type following command to find out python version:
$ python -V
Output:

Python 2.4.2

Step # 2: Download bittorrent client

Use wget to download bittorrent client

Download bittorrent client for Debian Linux:

$ wget http://download.bittorrent.com/dl/bittorrent_5.0.3_python2.4.deb

Download bittorrent clientFor Red Hat / Fedora Core / Cent OS Linux

$ wget http://download.bittorrent.com/dl/BitTorrent-5.0.3-1-Python2.4.noarch.rpm

Step # 3: Linux Install BitTorrent client

First login as a root user (use su or sudo command). If you are using Debian Linux, use dpkg command install bittorrent client
# dpkg -i bittorrent_5.0.3_python2.4.deb
If you are using Red Hat / Fedora Core /Cent OS Linux, use rpm command install bittorrent client:
# rpm -ivh BitTorrent-5.0.3-1-Python2.4.noarch.rpm

Step # 4: Start using or downloading files

Use bittorrent client as follows:
$ /usr/bin/bittorrent-curses 'http://www.some.org/DesktopBSD.iso.torrent'
OR
$ bittorrent-curses 'http://www.some.org/DesktopBSD.iso.torrent'
OR
$ /usr/bin/bittorrent-curses '/path/to/file.torrent'
You can try out pure console based client bittorrent-console instead of curses based client:
$ bittorrent-console '/path/to/file.torrent'
$ /usr/bin/bittorrent-console 'http://www.some.org/DesktopBSD.iso.torrent'

Tips about using and Troubleshooting BitTorrent client

(A) If you get [Errno 2] No such file or directory, you need to use wget or lynx command to download .torrent file to local hard drive first. For example
$ bittorrent-curses 'http://www.mininova.org/get/some_file[222].torrent'
First download download .torrent file using any one of the following method:
$ lynx 'http://www.mininova.org/get/some_file[222].torrent'
When prompted save .torrent file to a hard-disk. Another option is use wget to download .torrent file:
$ wget 'http://www.mininova.org/get/some_file[222].torrent'
Now start the downloading as follows:
$ bittorrent-curses 'some_file[222].torrent'

(B) If you get error 'This seems to be an old Python version which does not support detecting the filesystem encoding. Assuming 'ascii', then upgrade your python version to 2.4.xx:

Debian user upgrade python version using apt-get command:
# apt-get update
# apt-get install python2.4

Red Hat Linux user try out:
# up2date python2.4
Fedora Linux user try out:
# yum install python2.4

Remember if you have both versions (python v2.3 and v2.4) installed, you need to run bittorent client as follows:
$ python2.4 /usr/bin/bittorrent-curses '/path/to/file.torrent'
Command line BitTorrent client should work with other UNIX like operating systems such as FreeBSD/OpenBSD/Solaris etc (as long as you have Python it should work).

The current client enables a range of features including multiple parallel downloads. It also intermediates peering between itself, source file servers (read as trackers) and other clients, thereby yielding great distribution efficiencies. The client also enables users to create and share torrent files. See help files for more information.

See also: