≡ Menu

grep command

While administrating a box, you may wanted to find out what a processes is doing and find out how many file descriptors (fd) are being used. You will surprised to find out that process does open all sort of files:
=> Actual log file

=> /dev files

=> UNIX Sockets

=> Network sockets

=> Library files /lib /lib64

=> Executables and other programs etc

In this quick post, I will explain how to to count how many file descriptors are currently in use on your Linux server system.
[click to continue…]

Domain Expiration Check Shell Script

I've already written about a shell script to check / monitor domain renew / expiration date here. Now I’ve modified matt’s domain-check script to support additional C/TLDs .in, .biz, .org and .info domains. I've also added 5 seconds delay to avoid whois server rejecting query. This script checks to see if a domain has expired. It can be run in interactive and batch mode, and provides facilities to alarm if a domain is about to expire in advance.

Sample usage

Display expiration date and registrar for theos.in domain:
domain-check-2 -d {domain-name}

$ domain-check-2 -d theos.in
Output:

Domain                              Registrar         Status   Expires     Days Left
----------------------------------- ----------------- -------- ----------- ---------
theos.in                            et4India (R7-AFIN Valid    28-Oct-2009   799  

You can also get an email if theos.in going to expire in 30 days
$ domain-check-2 -a -d theos.in -q -x 30 -e vivek@nixcraft.com

However most killer feature is you can read list of domain names from a file such as mydomains.txt (list each domain on a new line):
$ domain-check-2 -a -f mydomains.txt -q -x 30 -e vivek@nixcraft.com
OR
$ domain-check-2 -f mydomains.txt

Output:

Domain                              Registrar         Status   Expires     Days Left
----------------------------------- ----------------- -------- ----------- ---------
theos.in                            et4India (R7-AFIN Valid    28-Oct-2009   799
nixcraft.org                        oDaddy.com, Inc.  Valid    13-Aug-2009   723
vivekgite.com                       MONIKER ONLINE SE Valid    18-aug-2010   1093
cyberciti.biz                                         Valid    30-Jun-2009   679
nixcraft.info                       oDaddy.com Inc. ( Valid    26-Jun-2009   675
nixcraft.net                        GODADDY.COM, INC. Valid    11-dec-2009   843  

Download

=> Download modified domain-check-2 script here.

Quick installation

Use wget command to download and install domain-check script:
$ wget http://www.cyberciti.biz/files/scripts/domain-check-2.txt
$ mv domain-check-2.txt domain-check
$ chmod +x domain-check
$ ./domain-check -d vivekgite.com

Make sure your run domain-check script using a cron job.

As I said earlier you need to tweak few things to run this monitor with Intel 810 or 845 based chip-set under Ubuntu Linux.

In order to use higher resolution install updated Intel i8xx, i9xx display driver. It is provided by a package called xserver-xorg-video-intel. This package provides the driver for the Intel i8xx and i9xx family of chipsets, including i810, i815, i830, i845, i855, i865, i915, and i945 series chips.

Replace driver by running the apt-get command. Open your terminal and type command:
$ sudo apt-get install xserver-xorg-video-intel

Now run ddcprobe command to get monitor VertRefresh values and HorizSync rate:
$ sudo ddcprobe | grep monitorrange
Output:

monitorrange: 24-82, 50-75

24-82 is your HorizSync rates and the second pair is your VertRefresh (50-75) values.

Next reconfigure xserver (X.org) video and monitor using auto detect feature:
$ sudo dpkg-reconfigure xserver-xorg

Set monitor resolution 1440 x 900 and others as per your requirement. Save the changes and reboot the system.

After login you can adjust screen resolution by visiting System > Preferences > Screen Resolution > Select desired screen resolution such as 1440 x 900 and save the changes.

This is one of the key questions many new sys admin ask:

How do I audit file events such as read / write etc? How can I use audit to see who changed a file in Linux?

The answer is to use 2.6 kernel’s audit system. Modern Linux kernel (2.6.x) comes with auditd daemon. It’s responsible for writing audit records to the disk. During startup, the rules in /etc/audit.rules are read by this daemon. You can open /etc/audit.rules file and make changes such as setup audit file log location and other option. The default file is good enough to get started with auditd.

In order to use audit facility you need to use following utilities
=> auditctl - a command to assist controlling the kernel’s audit system. You can get status, and add or delete rules into kernel audit system. Setting a watch on a file is accomplished using this command:

=> ausearch - a command that can query the audit daemon logs based for events based on different search criteria.

=> aureport - a tool that produces summary reports of the audit system logs.

Note that following all instructions are tested on CentOS 4.x and Fedora Core and RHEL 4/5 Linux.

Task: install audit package

The audit package contains the user space utilities for storing and searching the audit records generate by the audit subsystem in the Linux 2.6 kernel. CentOS/Red Hat and Fedora core includes audit rpm package. Use yum or up2date command to install package
# yum install audit
or
# up2date install audit

Auto start auditd service on boot
# ntsysv
OR
# chkconfig auditd on
Now start service:
# /etc/init.d/auditd start

How do I set a watch on a file for auditing?

Let us say you would like to audit a /etc/passwd file. You need to type command as follows:
# auditctl -w /etc/passwd -p war -k password-file

Where,

  • -w /etc/passwd : Insert a watch for the file system object at given path i.e. watch file called /etc/passwd
  • -p war : Set permissions filter for a file system watch. It can be r for read, w for write, x for execute, a for append.
  • -k password-file : Set a filter key on a /etc/passwd file (watch). The password-file is a filterkey (string of text that can be up to 31 bytes long). It can uniquely identify the audit records produced by the watch. You need to use password-file string or phrase while searching audit logs.

In short you are monitoring (read as watching) a /etc/passwd file for anyone (including syscall) that may perform a write, append or read operation on a file.

Wait for some time or as a normal user run command as follows:
$ grep 'something' /etc/passwd
$ vi /etc/passwd

Following are more examples:

File System audit rules

Add a watch on "/etc/shadow" with the arbitrary filterkey "shadow-file" that generates records for "reads, writes, executes, and appends" on "shadow"
# auditctl -w /etc/shadow -k shadow-file -p rwxa

syscall audit rule

The next rule suppresses auditing for mount syscall exits
# auditctl -a exit,never -S mount

File system audit rule

Add a watch "tmp" with a NULL filterkey that generates records "executes" on "/tmp" (good for a webserver)
# auditctl -w /tmp -p e -k webserver-watch-tmp

syscall audit rule using pid

To see all syscalls made by a program called sshd (pid - 1005):
# auditctl -a entry,always -S all -F pid=1005

How do I find out who changed or accessed a file /etc/passwd?

Use ausearch command as follows:
# ausearch -f /etc/passwd
OR
# ausearch -f /etc/passwd | less
OR
# ausearch -f /etc/passwd -i | less
Where,

  • -f /etc/passwd : Only search for this file
  • -i : Interpret numeric entities into text. For example, uid is converted to account name.

Output:

----
type=PATH msg=audit(03/16/2007 14:52:59.985:55) : name=/etc/passwd flags=follow,open inode=23087346 dev=08:02 mode=file,644 ouid=root ogid=root rdev=00:00
type=CWD msg=audit(03/16/2007 14:52:59.985:55) :  cwd=/webroot/home/lighttpd
type=FS_INODE msg=audit(03/16/2007 14:52:59.985:55) : inode=23087346 inode_uid=root inode_gid=root inode_dev=08:02 inode_rdev=00:00
type=FS_WATCH msg=audit(03/16/2007 14:52:59.985:55) : watch_inode=23087346 watch=passwd filterkey=password-file perm=read,write,append perm_mask=read
type=SYSCALL msg=audit(03/16/2007 14:52:59.985:55) : arch=x86_64 syscall=open success=yes exit=3 a0=7fbffffcb4 a1=0 a2=2 a3=6171d0 items=1 pid=12551 auid=unknown(4294967295) uid=lighttpd gid=lighttpd euid=lighttpd suid=lighttpd fsuid=lighttpd egid=lighttpd sgid=lighttpd fsgid=lighttpd comm=grep exe=/bin/grep

Let us try to understand output

  • audit(03/16/2007 14:52:59.985:55) : Audit log time
  • uid=lighttpd gid=lighttpd : User ids in numerical format. By passing -i option to command you can convert most of numeric data to human readable format. In our example user is lighttpd used grep command to open a file
  • exe="/bin/grep" : Command grep used to access /etc/passwd file
  • perm_mask=read : File was open for read operation

So from log files you can clearly see who read file using grep or made changes to a file using vi/vim text editor. Log provides tons of other information. You need to read man pages and documentation to understand raw log format.

Other useful examples

Search for events with date and time stamps. if the date is omitted, today is assumed. If the time is omitted, now is assumed. Use 24 hour clock time rather than AM or PM to specify time. An example date is 10/24/05. An example of time is 18:00:00.
# ausearch -ts today -k password-file
# ausearch -ts 3/12/07 -k password-file

Search for an event matching the given executable name using -x option. For example find out who has accessed /etc/passwd using rm command:
# ausearch -ts today -k password-file -x rm
# ausearch -ts 3/12/07 -k password-file -x rm

Search for an event with the given user name (UID). For example find out if user vivek (uid 506) try to open /etc/passwd:
# ausearch -ts today -k password-file -x rm -ui 506
# ausearch -k password-file -ui 506

Other auditing related posts

Further readings

  • Read man pages - auditd, ausearch, auditctl

Updated for accuracy.

Update: Check out new improved domain-check script.

Forgetting to renew your domain name can happen to all of us. According to this post:
Reports are coming in from Germany that Google.de was down for many hours yesterday, and has now gone live again. We're trying to confirm the reason, but it appears to be because Google forgot to renew the Google.de domain name..

I am going to share my little (read as dirty) shell script. It monitors and lists domain expiration date.

whois command line client for the whois directory service. It provides domain whois information. To find out nixcraft.com domain information you need to type:
$ whois nixcraft.com

Find domain expiration date

To get expiration date use grep command:
$ whois nixcraft.com | egrep -i 'Expiration|Expires on'
Output:

 Expiration Date: 10-may-2009
NOTICE: The expiration date displayed in this record is the date the
currently set to expire. This date does not necessarily reflect the expiration
view the registrar's reported date of expiration for this registration.
      Expires on: 10-May-09

Here is my script:

#!/bin/bash
# Domain name list - add your domainname here
DOM="theos.in cricketnow.in nixcraft.com nixcraft.org nixcraft.biz nixcraft.net nixcraft.info cyberciti.biz cyberciti.org gite.in nixcraft.in"
for d in $DOM
do
  echo -n "$d - "
  whois $d | egrep -i 'Expiration|Expires on' | head -1
  # If you need list..
  # whois $d | egrep -i 'Expiration|Expires on' | head -1 >> /tmp/domain.date
  #
  echo ""
done
#
# [ -f /tmp/domain.date ] && mail -s 'Domain renew / expiration date' you@yahoo.com < /tmp/domain.date || :
#

Output:

theos.in - Expiration Date:28-Oct-2007 13:01:58 UTC
cricketnow.in - Expiration Date:29-Jul-2008 09:17:56 UTC
nixcraft.com -    Expiration Date: 10-may-2009
nixcraft.org - Expiration Date:13-Aug-2007 14:58:30 UTC
nixcraft.biz - Domain Expiration Date:                      Fri Jun 01 23:59:59 GMT 2007
nixcraft.net -    Expiration Date: 11-dec-2007
nixcraft.info - Expiration Date:26-Jun-2007 11:05:13 UTC
cyberciti.biz - Domain Expiration Date:                      Tue Jun 30 23:59:59 GMT 2009
cyberciti.org - Expiration Date:25-May-2007 11:20:40 UTC
gite.in - Expiration Date:14-Sep-2007 06:47:36 UTC
nixcraft.in - Expiration Date:02-Feb-2008 05:33:08 UTC

Install a script and run on weekly / monthly basis via Linux/UNIX Cron facility.

You can simply monitor your remote system hosted some remote IDC. There may be tons of causes for which system goes out the network.

Understanding ping command

ping is one of the basic and nifty command to test network connection. For example you can send 4 ping request to cyberciti.biz with following command:
$ ping -c 4 cyberciti.biz
$ ping -c 4 router

Output:

PING router (192.168.1.254) 56(84) bytes of data.
64 bytes from router (192.168.1.254): icmp_seq=1 ttl=64 time=1.02 ms
64 bytes from router (192.168.1.254): icmp_seq=2 ttl=64 time=0.824 ms
64 bytes from router (192.168.1.254): icmp_seq=3 ttl=64 time=0.826 ms
64 bytes from router (192.168.1.254): icmp_seq=4 ttl=64 time=0.843 ms
--- router ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3012ms
rtt min/avg/max/mdev = 0.824/0.880/1.027/0.085 ms

Just see output '--- router ping statistics ---', 4 received is key to verify that 4 packets send and received successfully. You can extract 4 received with the following command:
$ ping -c4 router | grep 'received'
Output:

4 packets transmitted, 4 received, 0% packet loss, time 3012ms

Next logical step is to just get number 4 received using awk command:
$ ping -c4 router | grep 'received' | awk -F',' '{ print $4}'
Output:

4 received 

Last step is to just get 4 number and remove received word:
$ ping -c4 router | grep 'received' | awk -F',' '{ print $2}' | awk '{ print $1}'
Output:

4 

Shell script to system monitoring with ping command

Now you know how to obtained received packets, it is time to automate entire process with simple script. Here is sample script (download link):

#!/bin/bash
HOSTS="cyberciti.biz theos.in router"
COUNT=4
for myHost in $HOSTS
do
  count=$(ping -c $COUNT $myHost | grep 'received' | awk -F',' '{ print $2 }' | awk '{ print $1 }')
  if [ $count -eq 0 ]; then
    # 100% failed 
    echo "Host : $myHost is down (ping failed) at $(date)"
  fi
done

You can download complete working script which sends an email alert here.

Setup Cron job

In order to run this script every 30 minutes (or as per your requirements), you need to install a script as cron job:
$ chmod +x /path/to/monitorHost
Install the monitorHost script as crontab using the editor:
$ crontab -e
Append the following cronjob entry:
# Monitor remote host every 30 minutes using monitorHost
30 * * * * /home/vivek/bin/monitorHost

Save and close the file.

Shell script to watch the disk space

df displays the amount of disk space available on the file system containing each file name argument. If no file name is given, the space available on all currently mounted file systems is shown. Read man page of df if you are new to df command.

Steps

=> Find disk space using df

=> Filter out filesystem and find out the percentage of space using grep

=> Write a shell script

Step # 1: First get disk space:

$ df -H

Output:

Filesystem             Size   Used  Avail Use% Mounted on
/dev/hdb1               20G    14G   5.5G  71% /
tmpfs                  394M   4.1k   394M   1% /dev/shm
/dev/hdb5               29G    27G   654M  98% /nas/www

Step # 2: Next filter out filesystem and find out the percentage of space

$ df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }'

Output:

71% /dev/hdb1
98% /dev/hdb5

Step # 3: Write a shell script

Above command displays field 5 and 1 of df command. Now all you need to do is write a script to see if the percentage of space is >= 90% (download script):

#!/bin/sh
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 90 ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)" |
     mail -s "Alert: Almost out of disk space $usep%" you@somewhere.com
  fi
done

Setup Cron job

Save and install script as cronjob. Copy script to /etc/cron.daily/ (script downolad link)
# cp diskAlert /etc/cron.daily/
# chmod +x /etc/cron.daily/diskAlert

OR install as cronjob:
crontab -e

Write cronjob as per your requirement
10 0 * * * /path/to/diskAlert

Updated script version

Tony contributed and updated my script - You can exclude selected filesystem in case you don't want monitor all filesystems.

#!/bin/sh
# set -x
# Shell script to monitor or watch the disk space
# It will send an email to $ADMIN, if the (free available) percentage of space is >= 90%.
# -------------------------------------------------------------------------
# Set admin email so that you can get email.
ADMIN="root"
# set alert level 90% is default
ALERT=90
# Exclude list of unwanted monitoring, if several partions then use "|" to separate the partitions.
# An example: EXCLUDE_LIST="/dev/hdd1|/dev/hdc5"
EXCLUDE_LIST="/auto/ripper"
#
#::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
#
function main_prog() {
while read output;
do
#echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
  partition=$(echo $output | awk '{print $2}')
  if [ $usep -ge $ALERT ] ; then
     echo "Running out of space \"$partition ($usep%)\" on server $(hostname), $(date)" | \
     mail -s "Alert: Almost out of disk space $usep%" $ADMIN
  fi
done
}
if [ "$EXCLUDE_LIST" != "" ] ; then
  df -H | grep -vE "^Filesystem|tmpfs|cdrom|${EXCLUDE_LIST}" | awk '{print $5 " " $6}' | main_prog
else
  df -H | grep -vE "^Filesystem|tmpfs|cdrom" | awk '{print $5 " " $6}' | main_prog
fi