Monitoring

wtop is really cool application for web server log analysis and to see server stats at a glance. It also has powerful log grepping capability. It is just like ‘top’ for your webserver.

It can find out number of searches or signups per seconds. It can also create histogram of response time. There is also another tool called logrep a powerful command-line program for ad-hoc analysis and filtering for log files. You can dig up lots of information using wtop tools.

You need Python version 2.5 to run wtop.

Download wtop

Type the following command:
$ cd /tmp
$ wget http://wtop.googlecode.com/files/wtop-0.5.6.tar.gz
$ tar -zxvf wtop-0.5.6.tar.gz
$ cd wtop-0.5.6
# python setup.py install

Configuring wtop

Once installed you can start using the tool immediately. You need to edit /etc/wtop.cfg file to setup parameters, Apache log files and other directives
# vi /etc/wtop.cfg
Sample configuration file:

[main]
LOG_ROOT=/var/log/lighttpd/cyberciti.biz/
LOG_FILE=access.log
DEFAULT_OUTPUT_FIELDS=ts,class,ipcnt,ip,msec,uas,url

# This must match your webserver log format. You MUST have at least %h, %r and %D
LOG_FORMAT=%h %l %u %t "%r" %>s %B "%{Referer}i" "%{User-Agent}i" %D

[wtop]
# max time before a request is logged in the "slow" column
MAX_REQUEST_TIME=5000
# minimum requests/second before a URL class appears in top mode
MIN_RPS=0.2

[classes]
# you can extend these to make any classes you wish
home=^/(?:\?.*)?$
xml=\.xml(?:\?.*)?$
js=\.js(?:\?.*)?$
css=\.css(?:\?.*)?$
img=\.(?:png|gif|jpe?g|cur|ico|bmp)(?:\?.*)?$

[patterns]
# the generic pattern is applied if a line does not match any
# of the named classes. By default it uses the top-level directory.
generic=^/([^/\?]+)

# incomplete list of known web robots
robots = r'(?:nutch|MSRBOT|translate.google.com|Feedster|Nutch|Gaisbot|Snapbot|VisBot|libwww|CazoodleBot|polybot|VadixBot|Sogou|SBider|BecomeBot|Yandex|Pagebull|chudo|Pockey|nicebot|entireweb|FeedwhipBOT|ConveraCrawler|NG/2.0|WebImages|Factbot|information-online|gsa-crawler|Jyxobot|SentinelCrawler|BlogPulseLive|YahooFeedSeeker|GurujiBot|wwwster|Y\!J-SRD|Findexa|SurveyBot|yetibot|discoveryengine|fastsearch|noxtrum|Googlebot|Snapbot|OGSearchSpider|heritrix|nutch-agent|Slurp|msnbot|cuill|Mediapartners|YahooSeeker|GrabPERF|keywen|ia_archiver|crawler.archive.org|Baiduspider|larbin|shopwiki)'

Now simply type wtop at a shell prompt:
$ wtop$
See all human traffic, enter:
$ logrep -m top -h access.log
See response times for all MSNBot homepage hits:
$ logrep -m grep -g MSNBot -i home -o status,msec,url access.log
Display the current log for traffic to pages about wordpress or themes sent from google.com
$ logrep -m tail --f 'url~wordpress|themes,ref~google.com' access.log

Further readings:

This bug really was a bad one. I’ve client with over 200 Debian Linux server. Updating all systems wasn’t the problem. With the help of Cfengine I was able to push updates but managing all workstation ssh keys (over 1000+ Windows and Linux/BSD workstations) and testing everything took so much time. Debian shouldn’t have modified the package in first place. I also had to upgrade over 30 SSL certificates and a whole new CA for OpenVPN. Luckily VeriSign is providing revocation and replacement of SSL certificates (generally it is not provided free of charge) till 30-June-2008.

How do I find out all weak keys?

You can check all your weak keys with following commands:
# wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz
# wget http://security.debian.org/project/extra/dowkd/dowkd.pl.gz.asc
# gpg --keyserver subkeys.pgp.net --recv-keys 02D524BE
# gpg --verify dowkd.pl.gz.asc
# gunzip dowkd.pl.gz
# perl dowkd.pl host localhost

You should see 0 weak keys. If you run Debian or Ubuntu Linux upgrade your OpenSSL and fix all the affected softwares. There is also wiki page that will address all your concerns. Overall it lasted for few days for large clients. How many hours did you spend updating Debian systems?

So how do you list the network open ports on your Linux server and the process that owns them? The answer is simple. Use the following command (must be run as the root user):


sudo lsof -i
sudo netstat -lptu
sudo netstat -tulpn

Sample outputs (see video demo):

Under Linux you can use strace or valgrind tool for reporting and finding a bug. However, under *BSD / Mac OS X you need to use ktrace as replacement for strace tool.

kreace runs on the following platforms:
=> FreeBSD
=> OpenBSD
=> Mac OS X
=> NetBSD

The ktrace utility enables kernel trace logging for the specified processes. Kernel trace data is logged to the file ktrace.out. The kernel operations that are traced include system calls, namei translations, sig nal processing, and I/O. Once tracing is enabled on a process, trace data will be logged until either the process exits or the trace point is cleared. A traced process can generate enormous amounts of log data quickly; It is strongly suggested that users memorize how to disable tracing before attempting to trace a process.

To trace all kernel operations for process id # 2546, enter:
$ ktrace -p 2546
To disable all tracing of process # 2546, enter:
$ ktrace -cp 2546
To disable tracing on all user-owned processes, and, if executed by root, all processes in the system:
# ktrace -C
Attach to process id # 123 and log trace records to myapp.dbg.log instead of ktrace.out.
$ ktrace -p 123 -f myapp.dbg.log
To enable tracing of I/O on process # 123
$ ktrace -ti -p 123
The -t option is very useful to trace various kernel trace points, one per letter. The following table equates the letters with the trace points:

  • c : trace system calls
  • n : trace namei translations
  • i : trace I/O
  • s : trace signal processing
  • u : userland traces
  • w : context switches
  • + : trace the default set of trace points – c, n, i, s, u

Run the command called myapp and track only system calls, enter:
$ ktrace -tc ./myapp
Please note that the output of ktrace is not as informative as strace, but it does help to solve many problems.

truss: trace system calls

FreeBSD has another tool called truss. It traces the system calls called by the specified process or program. Output is to the specified output file, or standard error by default.
Attach to an already-running process # 123, enter
$ truss -p 123
Follow the system calls used myapp
$ truss ./myapp -d /tmp -f 120
Same as above, but put the output into a file called /tmp/myapp.dbg
$ truss -o /tmp/truss.out ./myapp -d /tmp -f 120

strace under FreeBSD

You can install strace under FreeBSD and other *BSD like oses.

Further readings:

  • ktrace man page
  • truss man page

I’ve worked with a various load balancing systems (LBS). They are complex pieces of hardware and software. In this post I will highlight some of the open source load balancing software. But what is load balancing?
It is nothing but a technique used to share (spared) load / services between two or more servers. For example, busy e-commerce or bank website uses load balancer to increase reliability, throughput, uptime, response time and better resource utilization. You can use following softwares as an advanced load balancing solution for web, cache, dns, mail, ftp, auth servers, VoIP services etc.

Linux Virtual Server (LVS)

LVS is ultimate open source Linux load sharing and balancing software. You can easily build a high-performance and highly available server for Linux using this software. From the project page:

Virtual server is a highly scalable and highly available server built on a cluster of real servers. The architecture of server cluster is fully transparent to end users, and the users interact with the cluster system as if it were only a single high-performance virtual server.

=> Project Web Site

Red Hat Cluster Suite

It is a high availability cluster software implementation from Linux leader Red Hat. It provide two type services:

  1. Application / Service Failover – Create n-node server clusters for failover of key applications and services
  2. IP Load Balancing – Load balance incoming IP network requests across a farm of servers

=> Product web page

The High Availability Linux Project

Linux-HA provides sophisticated high-availability (failover) capabilities on a wide range of platforms, supporting several tens of thousands of mission critical sites.

=> Project web site

Ultra Monkey

Ultra Monkey is a project to create load balanced and highly available network services. For example a cluster of web servers that appear as a single web server to end-users. The service may be for end-users across the world connected via the internet, or for enterprise users connected via an intranet.

Ultra Monkey makes use of the Linux operating system to provide a flexible solution that can be tailored to a wide range of needs. From small clusters of only two nodes to large systems serving thousands of connections per second.

=> Project web site

Personally, I’ve worked with both LVS and Red Hat Cluster Suite and I highly recommend these softwares.

netstat command and shell pipe feature can be used to dig out more information about particular IP address connection. You can find out total established connections, closing connection, SYN and FIN bits and much more. You can also display summary statistics for each protocol using netstat.

This is useful to find out if your server is under attack or not. You can also list abusive IP address using this method.
# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n
Output:

      1 CLOSE_WAIT
      1 established)
      1 Foreign
      3 FIN_WAIT1
      3 LAST_ACK
     13 ESTABLISHED
     17 LISTEN
    154 FIN_WAIT2
    327 TIME_WAIT

Dig out more information about a specific ip address:
# netstat -nat |grep {IP-address} | awk '{print $6}' | sort | uniq -c | sort -n

      2 LAST_ACK
      2 LISTEN
      4 FIN_WAIT1
     14 ESTABLISHED
     91 TIME_WAIT
    130 FIN_WAIT2

Busy server can give out more information:
# netstat -nat |grep 202.54.1.10 | awk '{print $6}' | sort | uniq -c | sort -n
Output:

  15 CLOSE_WAIT
  37 LAST_ACK
  64 FIN_WAIT_1
  65 FIN_WAIT_2
1251 TIME_WAIT
3597 SYN_SENT
5124 ESTABLISHED

Get List Of All Unique IP Address

To print list of all unique IP address connected to server, enter:
# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq
To print total of all unique IP address, enter:
# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq | wc -l
Output:

449

Find Out If Box is Under DoS Attack or Not

If you think your Linux box is under attack, print out a list of open connections on your box and sorts them by according to IP address, enter:
# netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n
Output:

    1 10.0.77.52
      2 10.1.11.3
      4 12.109.42.21
      6 12.191.136.3
.....
...
....
    13 202.155.209.202
     18 208.67.222.222
     28 0.0.0.0
    233 127.0.0.1

You can simply block all abusive IPs using iptables or just null route them.

Get Live View of TCP Connections

You can use tcptrack command to display the status of TCP connections that it sees on a given network interface. tcptrack monitors their state and displays information such as state, source/destination addresses and bandwidth usage in a sorted, updated list very much like the top command.

Display Summary Statistics for Each Protocol

Simply use netstat -s:
# netstat -s | less
# netstat -t -s | less
# netstat -u -s | less
# netstat -w -s | less
# netstat -s

Output:

Ip:
    88354557 total packets received
    0 forwarded
    0 incoming packets discarded
    88104061 incoming packets delivered
    96037391 requests sent out
    13 outgoing packets dropped
    66 fragments dropped after timeout
    295 reassemblies required
    106 packets reassembled ok
    66 packet reassembles failed
    34 fragments failed
Icmp:
    18108 ICMP messages received
    58 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 7173
        timeout in transit: 472
        redirects: 353
        echo requests: 10096
    28977 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 18881
        echo replies: 10096
Tcp:
    1202226 active connections openings
    2706802 passive connection openings
    7394 failed connection attempts
    47018 connection resets received
    23 connections established
    87975383 segments received
    95235730 segments send out
    681174 segments retransmited
    2044 bad segments received.
    80805 resets sent
Udp:
    92689 packets received
    14611 packets to unknown port received.
    0 packet receive errors
    96755 packets sent
TcpExt:
    48452 invalid SYN cookies received
    7357 resets received for embryonic SYN_RECV sockets
    43 ICMP packets dropped because they were out-of-window
    5 ICMP packets dropped because socket was locked
    2672073 TCP sockets finished time wait in fast timer
    441 time wait sockets recycled by time stamp
    368562 delayed acks sent
    430 delayed acks further delayed because of locked socket
    Quick ack mode was activated 36127 times
    32318597 packets directly queued to recvmsg prequeue.
    741479256 packets directly received from backlog
    1502338990 packets directly received from prequeue
    18343750 packets header predicted
    10220683 packets header predicted and directly queued to user
    17516622 acknowledgments not containing data received
    36549771 predicted acknowledgments
    102672 times recovered from packet loss due to fast retransmit
    Detected reordering 1596 times using reno fast retransmit
    Detected reordering 1 times using time stamp
    8 congestion windows fully recovered
    32 congestion windows partially recovered using Hoe heuristic
    19 congestion windows recovered after partial ack
    0 TCP data loss events
    39951 timeouts after reno fast retransmit
    29653 timeouts in loss state
    197005 fast retransmits
    186937 retransmits in slow start
    131433 other TCP timeouts
    TCPRenoRecoveryFail: 20217
    147 times receiver scheduled too late for direct processing
    29010 connections reset due to unexpected data
    365 connections reset due to early user close
    6979 connections aborted due to timeout

Display Interface Table

You can easily display dropped and total transmitted packets with netstat for eth0:
# netstat --interfaces eth0
Output:

Kernel Interface table
Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0       1500   0  2040929      0      0      0  3850539      0      0      0 BMRU

Other netstat related articles / tips:

  1. Get Information about All Running Services Remotely
  2. Linux / UNIX Find Out What Program / Service is Listening on a Specific TCP Port

Read following man pages for the details:
$ man netstat
$ man cut
$ man awk
$ man sed
$ man grep

Updated for accuracy.

You can use traditional netstat / lsof command to lists open Internet or UNIX domain sockets on FreeBSD. FreeBSD comes with a simple and easy to use command called sockstat.
The -4 option only displays IPv4 sockets.

The -6 option only displays IPv6 sockets.

The -c option only displays connected sockets.

The -l option only displays listening sockets (open port).

For example, display IPv4 related open ports, enter:
# sockstat -4 -l
Output:

USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sendmail   653   3  tcp4   127.0.0.1:25          *:*
root     sshd       647   3  tcp4   10.20.110.2:22        *:*
root     ntpd       616   4  udp4   *:123                 *:*

Here the equivalent of netstat:
$ netstat -nat | grep LISTEN
For information read sockstat command man page:
$ man sockstat

A TOP-like tool for monitoring system latency and its causes for Linux system.

The Intel Open Source Technology Center is pleased to announce the release of version 0.1 of LatencyTOP, a tool for developers to visualize system latencies. Skipping audio, slower servers, everyone knows the symptoms of latency. But to know what’s going on in the system, what’s causing the latency, how to fix it… that’s a hard question without good answers right now.

LatencyTOP is a Linux tool for software developers (both kernel and userspace), aimed at identifying where in the system latency is happening, and what kind of operation/action is causing the latency to happen so that the code can be changed to avoid the worst latency hiccups.

(Fig. 01: LatencyTOP in Action [ Image Credit: Intel Corp. ])

Download LatencyTOP

=> Visit official project site to download LatencyTOP software. Please note that you also need to patch Linux kernel.

You can test your IDS or IPS devices to monitor your scan traffic. Use this feature to avoid detection with nmap. You may not want to get caught performing a network scan. For example, using following technique you can test your own IDS / IPS / network security from remote location or home.

nmap Decoy option – Cloak a scan with decoys

nmap has -D option. It is called decoy scan. With -D option it appear to the remote host that the host(s) you specify as decoys are scanning the target network too. Thus their IDS might report 5-10 port scans from unique IP addresses, but they won’t know which IP was scanning them and which were innocent decoys. While this can be defeated through router path tracing, response-dropping, and other active mechanisms, it is generally an effective technique for hiding your IP address.

You can separate each decoy host with commas, and you can optionally use ME as one of the decoys to represent the position for your real IP address. If you put ME in the 6th position or later, some common port scan detectors (such as Solar Designer’s excellent scanlogd) are unlikely to show your IP address at all. If you don’t use ME, nmap will put you in a random position. Note that the hosts you use as decoys should be up or you might accidentally SYN flood your targets. Also it will be pretty easy to determine which host is scanning if only one is actually up on the network. You might want to use IP addresses instead of names (so the decoy networks don’t see you in their nameserver logs).

WARNING! These penetration testing (security testing) examples may be considered as Unauthorized Access or Illegal Behavior. Use examples on your own RISK and/or to secure your own network host / IPS /IDS.

Use the following syntax:
# nmap -n -Ddecoy-ip1,decoy-ip2,your-own-ip,decoy-ip3,decoy-ip4 remote-host-ip
# nmap -n -D192.168.1.5,10.5.1.2,172.1.2.4,3.4.2.1 192.168.1.5

Host 192.168.1.5 (or network IDS / IPS) will see 4 port scan and remote host / IDS has no way telling which one was real. Decoys are used both in the initial ping scan (using ICMP, SYN, ACK, or whatever) and during the actual port scanning phase. Decoys are also used during remote OS detection (-O). Decoys do not work with version detection or TCP connect scan. It is worth noting that using too many decoys may slow your scan and potentially even make it less accurate. Also, some ISPs will filter out your spoofed packets, but many do not restrict spoofed IP packets at all.

nmap ideal scan technique to hide your IP

Following example, uses an an idle scan technique. It uses port 1234 on 1.1.1.1 IP as as a zombie to scan host – 192.1.2.3:
# nmap -P0 -sI 1.1.1.1:1234 192.1.2.3

This technique only hides your source address but remote IPS / IDS always record and logs scan. Please refer to nmap man page for more information:
man nmap