Linux audit files to see who made changes to a file

Posted on in Categories File system, GNU/Open source, Howto, Linux, Monitoring, RedHat/Fedora Linux, Security, Sys admin, Tips last updated March 19, 2007

This is one of the key questions many new sys admin ask:

How do I audit file events such as read / write etc? How can I use audit to see who changed a file in Linux?

The answer is to use 2.6 kernel’s audit system. Modern Linux kernel (2.6.x) comes with auditd daemon. It’s responsible for writing audit records to the disk. During startup, the rules in /etc/audit.rules are read by this daemon. You can open /etc/audit.rules file and make changes such as setup audit file log location and other option. The default file is good enough to get started with auditd.

In order to use audit facility you need to use following utilities
=> auditctl – a command to assist controlling the kernel’s audit system. You can get status, and add or delete rules into kernel audit system. Setting a watch on a file is accomplished using this command:

=> ausearch – a command that can query the audit daemon logs based for events based on different search criteria.

=> aureport – a tool that produces summary reports of the audit system logs.

Note that following all instructions are tested on CentOS 4.x and Fedora Core and RHEL 4/5 Linux.

Task: install audit package

The audit package contains the user space utilities for storing and searching the audit records generate by the audit subsystem in the Linux 2.6 kernel. CentOS/Red Hat and Fedora core includes audit rpm package. Use yum or up2date command to install package
# yum install audit
or
# up2date install audit

Auto start auditd service on boot
# ntsysv
OR
# chkconfig auditd on
Now start service:
# /etc/init.d/auditd start

How do I set a watch on a file for auditing?

Let us say you would like to audit a /etc/passwd file. You need to type command as follows:
# auditctl -w /etc/passwd -p war -k password-file

Where,

  • -w /etc/passwd : Insert a watch for the file system object at given path i.e. watch file called /etc/passwd
  • -p war : Set permissions filter for a file system watch. It can be r for read, w for write, x for execute, a for append.
  • -k password-file : Set a filter key on a /etc/passwd file (watch). The password-file is a filterkey (string of text that can be up to 31 bytes long). It can uniquely identify the audit records produced by the watch. You need to use password-file string or phrase while searching audit logs.

In short you are monitoring (read as watching) a /etc/passwd file for anyone (including syscall) that may perform a write, append or read operation on a file.

Wait for some time or as a normal user run command as follows:
$ grep 'something' /etc/passwd
$ vi /etc/passwd

Following are more examples:

File System audit rules

Add a watch on “/etc/shadow” with the arbitrary filterkey “shadow-file” that generates records for “reads, writes, executes, and appends” on “shadow”
# auditctl -w /etc/shadow -k shadow-file -p rwxa

syscall audit rule

The next rule suppresses auditing for mount syscall exits
# auditctl -a exit,never -S mount

File system audit rule

Add a watch “tmp” with a NULL filterkey that generates records “executes” on “/tmp” (good for a webserver)
# auditctl -w /tmp -p e -k webserver-watch-tmp

syscall audit rule using pid

To see all syscalls made by a program called sshd (pid – 1005):
# auditctl -a entry,always -S all -F pid=1005

How do I find out who changed or accessed a file /etc/passwd?

Use ausearch command as follows:
# ausearch -f /etc/passwd
OR
# ausearch -f /etc/passwd | less
OR
# ausearch -f /etc/passwd -i | less
Where,

  • -f /etc/passwd : Only search for this file
  • -i : Interpret numeric entities into text. For example, uid is converted to account name.

Output:

----
type=PATH msg=audit(03/16/2007 14:52:59.985:55) : name=/etc/passwd flags=follow,open inode=23087346 dev=08:02 mode=file,644 ouid=root ogid=root rdev=00:00
type=CWD msg=audit(03/16/2007 14:52:59.985:55) :  cwd=/webroot/home/lighttpd
type=FS_INODE msg=audit(03/16/2007 14:52:59.985:55) : inode=23087346 inode_uid=root inode_gid=root inode_dev=08:02 inode_rdev=00:00
type=FS_WATCH msg=audit(03/16/2007 14:52:59.985:55) : watch_inode=23087346 watch=passwd filterkey=password-file perm=read,write,append perm_mask=read
type=SYSCALL msg=audit(03/16/2007 14:52:59.985:55) : arch=x86_64 syscall=open success=yes exit=3 a0=7fbffffcb4 a1=0 a2=2 a3=6171d0 items=1 pid=12551 auid=unknown(4294967295) uid=lighttpd gid=lighttpd euid=lighttpd suid=lighttpd fsuid=lighttpd egid=lighttpd sgid=lighttpd fsgid=lighttpd comm=grep exe=/bin/grep

Let us try to understand output

  • audit(03/16/2007 14:52:59.985:55) : Audit log time
  • uid=lighttpd gid=lighttpd : User ids in numerical format. By passing -i option to command you can convert most of numeric data to human readable format. In our example user is lighttpd used grep command to open a file
  • exe=”/bin/grep” : Command grep used to access /etc/passwd file
  • perm_mask=read : File was open for read operation

So from log files you can clearly see who read file using grep or made changes to a file using vi/vim text editor. Log provides tons of other information. You need to read man pages and documentation to understand raw log format.

Other useful examples

Search for events with date and time stamps. if the date is omitted, today is assumed. If the time is omitted, now is assumed. Use 24 hour clock time rather than AM or PM to specify time. An example date is 10/24/05. An example of time is 18:00:00.
# ausearch -ts today -k password-file
# ausearch -ts 3/12/07 -k password-file

Search for an event matching the given executable name using -x option. For example find out who has accessed /etc/passwd using rm command:
# ausearch -ts today -k password-file -x rm
# ausearch -ts 3/12/07 -k password-file -x rm

Search for an event with the given user name (UID). For example find out if user vivek (uid 506) try to open /etc/passwd:
# ausearch -ts today -k password-file -x rm -ui 506
# ausearch -k password-file -ui 506

Other auditing related posts

Further readings

  • Read man pages – auditd, ausearch, auditctl

Updated for accuracy.

Shell script to check / monitor domain renew / expiration date

Posted on in Categories Linux, Shell scripting, Suse Linux, Sys admin, Tips, Ubuntu Linux, UNIX last updated January 24, 2007

Update: Check out new improved domain-check script.

Forgetting to renew your domain name can happen to all of us. According to this post:
Reports are coming in from Germany that Google.de was down for many hours yesterday, and has now gone live again. We’re trying to confirm the reason, but it appears to be because Google forgot to renew the Google.de domain name..

I am going to share my little (read as dirty) shell script. It monitors and lists domain expiration date.

whois command line client for the whois directory service. It provides domain whois information. To find out nixcraft.com domain information you need to type:
$ whois nixcraft.com

Find domain expiration date

To get expiration date use grep command:
$ whois nixcraft.com | egrep -i 'Expiration|Expires on'
Output:

 Expiration Date: 10-may-2009
NOTICE: The expiration date displayed in this record is the date the
currently set to expire. This date does not necessarily reflect the expiration
view the registrar's reported date of expiration for this registration.
      Expires on: 10-May-09

Here is my script:

#!/bin/bash
# Domain name list - add your domainname here
DOM="theos.in cricketnow.in nixcraft.com nixcraft.org nixcraft.biz nixcraft.net nixcraft.info cyberciti.biz cyberciti.org gite.in nixcraft.in"
for d in $DOM
do
  echo -n "$d - "
  whois $d | egrep -i 'Expiration|Expires on' | head -1
  # If you need list..
  # whois $d | egrep -i 'Expiration|Expires on' | head -1 >> /tmp/domain.date
  #
  echo ""
done
#
# [ -f /tmp/domain.date ] && mail -s 'Domain renew / expiration date' [email protected] < /tmp/domain.date || :
#

Output:

theos.in - Expiration Date:28-Oct-2007 13:01:58 UTC

cricketnow.in - Expiration Date:29-Jul-2008 09:17:56 UTC

nixcraft.com -    Expiration Date: 10-may-2009

nixcraft.org - Expiration Date:13-Aug-2007 14:58:30 UTC

nixcraft.biz - Domain Expiration Date:                      Fri Jun 01 23:59:59 GMT 2007

nixcraft.net -    Expiration Date: 11-dec-2007

nixcraft.info - Expiration Date:26-Jun-2007 11:05:13 UTC

cyberciti.biz - Domain Expiration Date:                      Tue Jun 30 23:59:59 GMT 2009

cyberciti.org - Expiration Date:25-May-2007 11:20:40 UTC

gite.in - Expiration Date:14-Sep-2007 06:47:36 UTC

nixcraft.in - Expiration Date:02-Feb-2008 05:33:08 UTC

Install a script and run on weekly / monthly basis via Linux/UNIX Cron facility.

Simple Linux and UNIX Shell Script Based System Monitoring With ping Command

Posted on in Categories Monitoring, Networking, Shell scripting last updated July 21, 2006

You can simply monitor your remote system hosted in some remote IDC. There may be many reasons for which system may out of the network. This simple script is useful to monitor your own small network at home or work.
Continue reading “Simple Linux and UNIX Shell Script Based System Monitoring With ping Command”

Shell script to watch the disk space

Posted on in Categories Howto, Linux, Monitoring, Shell scripting last updated July 13, 2006

df displays the amount of disk space available on the file system containing each file name argument. If no file name is given, the space available on all currently mounted file systems is shown. Read man page of df if you are new to df command.

Steps

=> Find disk space using df

=> Filter out filesystem and find out the percentage of space using grep

=> Write a shell script

Step # 1: First get disk space:

$ df -H

Output:

Filesystem             Size   Used  Avail Use% Mounted on
/dev/hdb1               20G    14G   5.5G  71% /
tmpfs                  394M   4.1k   394M   1% /dev/shm
/dev/hdb5               29G    27G   654M  98% /nas/www

Step # 2: Next filter out filesystem and find out the percentage of space

$ df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }'

Output:

71% /dev/hdb1
98% /dev/hdb5

Step # 3: Write a shell script

Above command displays field 5 and 1 of df command. Now all you need to do is write a script to see if the percentage of space is >= 90% (download script):

#!/bin/sh
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 90 ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)" |
     mail -s "Alert: Almost out of disk space $usep%" [email protected]
  fi
done

Setup Cron job

Save and install script as cronjob. Copy script to /etc/cron.daily/ (script downolad link)
# cp diskAlert /etc/cron.daily/
# chmod +x /etc/cron.daily/diskAlert

OR install as cronjob:
crontab -e

Write cronjob as per your requirement
10 0 * * * /path/to/diskAlert

Updated script version

Tony contributed and updated my script – You can exclude selected filesystem in case you don’t want monitor all filesystems.

#!/bin/sh
# set -x
# Shell script to monitor or watch the disk space
# It will send an email to $ADMIN, if the (free available) percentage of space is >= 90%.
# -------------------------------------------------------------------------
# Set admin email so that you can get email.
ADMIN="root"
# set alert level 90% is default
ALERT=90
# Exclude list of unwanted monitoring, if several partions then use "|" to separate the partitions.
# An example: EXCLUDE_LIST="/dev/hdd1|/dev/hdc5"
EXCLUDE_LIST="/auto/ripper"
#
#::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
#
function main_prog() {
while read output;
do
#echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
  partition=$(echo $output | awk '{print $2}')
  if [ $usep -ge $ALERT ] ; then
     echo "Running out of space \"$partition ($usep%)\" on server $(hostname), $(date)" | \
     mail -s "Alert: Almost out of disk space $usep%" $ADMIN
  fi
done
}

if [ "$EXCLUDE_LIST" != "" ] ; then
  df -H | grep -vE "^Filesystem|tmpfs|cdrom|${EXCLUDE_LIST}" | awk '{print $5 " " $6}' | main_prog
else
  df -H | grep -vE "^Filesystem|tmpfs|cdrom" | awk '{print $5 " " $6}' | main_prog
fi

Debian Linux How to find out if installed package is from stable or testing environment

Posted on in Categories Debian Linux, Howto, Linux, Linux distribution, Tips, Ubuntu Linux last updated March 1, 2006

I install many packages for Debian / Ubuntu Linux from both stable/testing environment but some time I need to find out installed package is from stable or testing environment. I can use dpkg status file to get this information. But Debian comes with perl script called apt-show-versions which lists available package versions with distribution.

apt-show-versions parses the dpkg status file and the APT lists for the installed and available package versions and distribution and shows upgrade options within the specific distribution of the selected package. apt-show-versions uses caching for the status information of installed and available packages. If you run apt-show-versions as root the cache is updated as needed. If you run as non-root uses the newest available information, but can’t update the cache. If you run as root with the option -i the cache is initialized or updated only.

This is really useful if you have a mixed stable/testing environment and want to list all packages which are from testing and can be upgraded in testing.

Install apt-show-versions

Type the following command at shell prompt:
$ sudo apt-get install apt-show-versions
Just type command apt-show-versions:
$ apt-show-versions
Output:

java-common/testing uptodate 0.23
libperl5.8/testing upgradeable from 5.8.7-10 to 5.8.8-2
sysutils/testing upgradeable from 2.0.0-1 to 2.0.1
autoconf/unstable uptodate 2.59a-8

To upgrade all packages in testing you can type command:

# apt-get install $(apt-show-versions -u -b | fgrep testing)

Find out a list of all available versions of postgresql database server:

$ apt-show-versions -a -p postgresql

Read UNIX / Linux System IP Address In a Shell Script

Posted on in Categories Linux, Shell scripting last updated January 16, 2006

Reading an IP address in shell script required many time. However, different Linux distribution stores IP address in different files. If you are looking to run script under different UNIX like OSes such as Solaris or FreeBSD then you need to use the ifconfig command. The ifconfig command is not just used to configure a network interface, but it can be use to obtained information such as network IP, netmask and much more.

Uninstall files installed from a source code tar ball on Linux or Unix

Posted on in Categories Debian Linux, File system, FreeBSD, GNU/Open source, Howto, Linux, RedHat/Fedora Linux, Solaris, Suse Linux, Tips, Ubuntu Linux, UNIX last updated December 24, 2005

Installing software from a source code is common practice in Linux and Unix world. Some time this is preferred method because it gives all power and flexibility you need to optimize your software such as MySQL, PHP, Apache and more. However, uninstalling files installed from a source code tar ball can be a big headache.

Two methods can be used to uninstall files:
Continue reading “Uninstall files installed from a source code tar ball on Linux or Unix”

Linux commands to help you navigate

Posted on in Categories CentOS, File system, Gentoo Linux, Howto, Linux, Linux desktop, Sys admin, Tips last updated October 13, 2005

As a Linux system administrator, you will need to find files in directories all over the file system. Especially those coming from a Windows background, often lost themselves while navigating file system.

Linux and other UNIX (BSD) OS offers an excellent collection of utilities, which can be use to finding the files and executables, remember you cannot memorize all the commands and files 😉
Commands to help you navigate:

  • file: Determines file types
  • which: Locates an executable in your PATH
  • whereis: Locates binaries and man page
  • find: Find the file
  • grep: Search for text/string in the named file name
  • strings: Find text string in a binary file

The which command

It is useful to locate a command. Some opertating system such as Solaris/HP-UX (even linux) have multiple homes. So you wanna find out which version you are going to use by order of the directories in your PATH variable. Try out following commands:
$ which ls
$ which vi
$ which vim

The file command

You would like to find out if a command is a shell script or a binary file or simply cannot recognize file by its extension then use file command to determine file type.
$ file /usr/sbin/useradd
Output:

/usr/sbin/useradd: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.0, dynamically linked (uses shared libs), stripped

Let us try another example:
# file /etc/shadow
Output:

/etc/shadow: ASCII text

But wait sec, you don’t have to type full command path:
$ file $(which adduser)
Output:

/usr/sbin/adduser: perl script text executable

The whereis command

It locates binaries and man pages. When you get message command not found then use whereis command to locate binary file. For example ifconfig command:
$ ifconfig
Output:

bash: ifconfig: command not found

Now locate ifconfig binary, enter:
$ whereis -b ifconfig
Output:

ifconfig: /sbin/ifconfig

So let us try the full path, enter:
$ /sbin/ifconfig

The grep command

The grep command can search for text or strings such as IP address, domain names and lots of other stuff inside a text file. Often new Linux sys admin forgets to configuration file names. However, you can use grep to find out those configuration file name. For example, find out the file containing IP address 192.168.1.1
# grep -R "192.168.1.1" /etc/* | less

Find out kernel driver module bttv configuration file name, so that you can remove the driver:
# grep -R "bttv" /etc/* | less<

The strings Commands

The grep command is useful to search a text file, if you would like to find text string in a binary file then use strings command.
# strings /usr/bin/users

You might think this is stupid idea to search inside binary file for text string. Well, no it is not a stupid idea. For example, you would like to quickly find out if internet service supports tcpd access control facility via /etc/hosts.allow and /etc/hosts.deny files (read as tcp wrappers) or not. Let us find out if sshd server support tcpd or not:
# strings $(which sshd)| grep libwrap

libwrap.so.0
libwrap refuse returns

The find Command

Use find command to find the files. Find all files belonging to the user charvi:
# find / -user charvi

Remove all core dump files
# find / -name core -exec rm -i{}\;

Please see more find command examples here and here. For more info please read the man pages of find, grep, file, which.