Linux / Unix logtop: Realtime Log Line Rate Analyser

How can I analyze line rate taking log file as input on a Linux system? How do I find the IP flooding my Apache/Nginx/Lighttpd web-server on a Debian or Ubuntu Linux?

[donotprint]
Tutorial details
DifficultyEasy (rss)
Root privilegesYes
RequirementsNone
TimeN/A
[/donotprint]You need to use a tool called logtop. It is a system administrator tool to analyze line rate taking log file as input. It reads on stdin and print a constantly updated result displaying, in columns in the following format:

ADVERTISEMENTS

Line number, count, frequency, and the actual line

How do install logtop on a Debian or Ubuntu based system?

Simply type the following apt-get command:
$ sudo apt-get install logtop
Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  logtop
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 15.7 kB of archives.
After this operation, 81.9 kB of additional disk space will be used.
Get:1 http://mirrors.service.networklayer.com/ubuntu/ precise/universe logtop amd64 0.3-1 [15.7 kB]
Fetched 15.7 kB in 0s (0 B/s)
Selecting previously unselected package logtop.
(Reading database ... 114954 files and directories currently installed.)
Unpacking logtop (from .../logtop_0.3-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up logtop (0.3-1) ...

Syntax

The syntax is as follows:

logtop [OPTIONS] [FILE]
command | logtop
command1 | filter | logtop
command1 | filter | logtop [options] [file]

Examples

Here are some common examples of logtop.

Show the IP address flooding your LAMP server

Type the following command:

tail -f www.cyberciti.biz_access.log | cut -d' ' -f1 | logtop

Sample outputs:

Fig.01: logtop command in action

Fig.01: logtop command in action

See squid cache HIT and MISS log

tail -f cache.log | grep -o "HIT\|MISS" | logtop

To see realtime hit / miss ratio on some caching software log file, enter:
tail -f access.log | cut -d' ' -f1 | logtop -s 20000
The -s option set logtop to work with the maximum of K lines instead of 10000.

See also
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source/DevOps topics:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNCentOS 8 Debian 10 Firewall Ubuntu 20.04

ADVERTISEMENTS
6 comments… add one
  • Charles Jul 22, 2014 @ 17:24

    This looks really cool, possibly invaluable during a DoS attack.

    Any chance there’s instructions on how to install on FreeBSD? logtop is not in the ports collection :(

  • John Jul 22, 2014 @ 18:38

    Agreed. Good tool to find out during a DoS attack. Download it from git repo and may be compile from source? https://github.com/JulienPalard/logtop ?

  • Rabin Jul 23, 2014 @ 7:48

    how is this diffrent then just using `sort | uniq -c`

    • Yordan Georgiev Jul 27, 2014 @ 6:16

      *** how is this different than just using `sort | uniq -c`
      sudo cat /var/log/httpd/access_log | cut -d’ ‘ -f1 | logtop
      2453 lines, 2453.00 lines/s
      RANK CNT LINE/S LINE
      1 1213 1213.00 88.195.123.26
      2 201 201.00 66.249.74.226
      3 157 157.00 ::1
      4 105 105.00 91.156.142.150
      5 83 83.00 194.86.38.32
      6 77 77.00 85.76.96.172
      7 40 40.00 85.76.42.107
      8 37 37.00 69.164.101.198
      9 37 37.00 84.250.125.210
      10 27 27.00 66.249.61.81

  • Marco Jul 23, 2014 @ 8:18

    Here is my guide to install it on CentOS 7 (should also apply to RHEL 7)
    http://korovamilky.tumblr.com/post/92616874227

  • mauron85 Aug 11, 2014 @ 20:58

    I believe that difference is that it works incrementaly. With uniq sort combo you have to proces whole log again. With this only inctlrements thanks to tail.

Leave a Reply

Your email address will not be published.

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.