Configure lighttpd alias (mod_alias)

Posted on in Categories Howto, lighttpd, Linux, UNIX last updated July 25, 2006

This lighttpd module provides for mapping different parts of the host filesystem in the document tree. You can use it for mapping various directories. For example cgi-bin directory mapped to /var/lib/cgi-bin. The alias module is used to specify a special document-root for a given url-subset.

Configuration

Open your lighttpd configuration file:
vi /etc/lighttpd/lighttpd.conf

Append/add mod_ alias to list of server modules:
server.modules += ( "mod_alias" )

Examples

Add cgi-bin alias for doamin theos.in
alias.url = ( "/cgi-bin/" => "/home/lighttpd/theos.in/cgi-bin/" )

Browse all documents installed at /usr/share/doc/ directory with following alias:
alias.url = ( "/docs/" => "/usr/share/doc/" )
alias.url += ( "/stats/" => "/home/theos.in/http/webalizer/" )

Open a browser and type url http://theos.in/docs/ or http://your-domain.com/docs/

Shell script to watch the disk space

Posted on in Categories Howto, Linux, Monitoring, Shell scripting last updated July 13, 2006

df displays the amount of disk space available on the file system containing each file name argument. If no file name is given, the space available on all currently mounted file systems is shown. Read man page of df if you are new to df command.

Steps

=> Find disk space using df

=> Filter out filesystem and find out the percentage of space using grep

=> Write a shell script

Step # 1: First get disk space:

$ df -H

Output:

Filesystem             Size   Used  Avail Use% Mounted on
/dev/hdb1               20G    14G   5.5G  71% /
tmpfs                  394M   4.1k   394M   1% /dev/shm
/dev/hdb5               29G    27G   654M  98% /nas/www

Step # 2: Next filter out filesystem and find out the percentage of space

$ df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }'

Output:

71% /dev/hdb1
98% /dev/hdb5

Step # 3: Write a shell script

Above command displays field 5 and 1 of df command. Now all you need to do is write a script to see if the percentage of space is >= 90% (download script):

#!/bin/sh
df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 90 ]; then
    echo "Running out of space \"$partition ($usep%)\" on $(hostname) as on $(date)" |
     mail -s "Alert: Almost out of disk space $usep%" [email protected]
  fi
done

Setup Cron job

Save and install script as cronjob. Copy script to /etc/cron.daily/ (script downolad link)
# cp diskAlert /etc/cron.daily/
# chmod +x /etc/cron.daily/diskAlert

OR install as cronjob:
crontab -e

Write cronjob as per your requirement
10 0 * * * /path/to/diskAlert

Updated script version

Tony contributed and updated my script – You can exclude selected filesystem in case you don’t want monitor all filesystems.

#!/bin/sh
# set -x
# Shell script to monitor or watch the disk space
# It will send an email to $ADMIN, if the (free available) percentage of space is >= 90%.
# -------------------------------------------------------------------------
# Set admin email so that you can get email.
ADMIN="root"
# set alert level 90% is default
ALERT=90
# Exclude list of unwanted monitoring, if several partions then use "|" to separate the partitions.
# An example: EXCLUDE_LIST="/dev/hdd1|/dev/hdc5"
EXCLUDE_LIST="/auto/ripper"
#
#::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
#
function main_prog() {
while read output;
do
#echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1)
  partition=$(echo $output | awk '{print $2}')
  if [ $usep -ge $ALERT ] ; then
     echo "Running out of space \"$partition ($usep%)\" on server $(hostname), $(date)" | \
     mail -s "Alert: Almost out of disk space $usep%" $ADMIN
  fi
done
}

if [ "$EXCLUDE_LIST" != "" ] ; then
  df -H | grep -vE "^Filesystem|tmpfs|cdrom|${EXCLUDE_LIST}" | awk '{print $5 " " $6}' | main_prog
else
  df -H | grep -vE "^Filesystem|tmpfs|cdrom" | awk '{print $5 " " $6}' | main_prog
fi

Repairing ReiserFS file system with reiserfsck

Posted on in Categories CentOS, Data recovery, Debian Linux, File system, Linux, Troubleshooting last updated July 13, 2006
Repairing ReiserFS file system with reiserfsck

The idea and commands in this article submitted by Jacques Wagener via email. In his own words, “After nuking my partition by accident (and through my stupidity) I was really disappointed in myself, especially in losing my bookmarks and rss-feeds“. The following article is based upon our email communication. I am just putting them as an article.

We have already written about ext2/ext3 file repair using fsck and other utilities. Linux comes with different filesystems and different repair utilities. To repair a ReiserFS filesystem you need to run reiserfsck command, which is a checking tool for the ReiserFS filesystem (just like fsck command for ext2/ext3 file system).

Reiserfsck searches for a Reiserfs filesystem on a device, replays any necessary transactions, and either checks or repairs the file system. ReiserFS saves data or log in a special file for pending disk updates and later on it commit updates to disk resulting into very good filesystem consistency.

Step # 1: Install reiserfsck

You need to install reiserfsprogs package, which includes reiserfsck user level tools for ReiserFS filesystems.
# apt-get install reiserfsprogs

Or if you are using RedHat/Fedor Core Linux:
# yum install reiserfsprogs

Step # 2: Backup partition/disk

Take system down to runlevel 1.
# init 1

Unmount /dev/sda1 (if it is still mounted as read/write only):
# umount /dev/sda1; umount /dev/sda2

Before using any one of the following command you are strongly advised to make a backup copy of the whole partition using dd or ddrescue (recommended) command.
# ddrescue /dev/sda /dev/sdb

OR use dd command:
# dd if=/dev/sda of=/dev/sdb conv=noerror

Step # 3: Check filesystem consistency

Above command makes a backup of your drive. Next check filesystem consistency which will reports problem to you with the following command (assuming that /dev/sda1 is your partition):
# reiserfsck --check /dev/sda1

If you get an error Bad root block 0If you get following message:
Running with –rebuild-tree is required

Then you need to run following command rebuilds the entire filesystem tree using leaf nodes found on the device (this is nothing but indication of corrupted :
# reiserfsck --scan-whole-partition --rebuild-tree /dev/sda1

Where,

  • –scan-whole-partition: This option causes –rebuild-tree to scan the whole partition but not only the used space on the partition. You should always use this option with –rebuild-tree.
  • –check: Checks filesystem consistency and reports, but does not repair any corruption that it finds.
  • –rebuild-tree: This option rebuilds the entire filesystem tree using leaf nodes found on the device. Once you passed this option, do not stop or interrupt building operating.

Recovering corrupted superblock

Each file system has a superblock, which contains information about file system such as:

  • File system type
  • Size
  • Status
  • Information about other metadata structures

If this information lost, you are in trouble (data loss) so Linux maintains multiple redundant copies of the superblock in every file system.

During check (reiserfsck –check /dev/sda1) if you get an error superblock was missing, use following command to fix superblock:
# reiserfsck --rebuild-sb /dev/sda1

Where,

  • –rebuild-sb: This option recovers the superblock on a Reiserfs partition. Normally you only need this option if mount reports “read_super_block: can’t find a reiserfs file system”.

Caution: Do not run above command twice on same drive. You will damage your partition (data).

Final note

Next logical step is mount your partition /dev/sda1 and check for your data:
# mkdir -p /mnt/data
# mount /dev/sda1 /mnt/data
# cd /mnt/data
# ls
# ls lost+found/ -l

lost+found is a special directory where recovered files are kept by Linux/reiserfsck. You can examine these files and restore the data.

Better backup entire partition using tar or ssh session:
# tar cvf /dev/nst0 /mnt/data

OR use scp to dump data to remote system:
# scp -r /mnt/data [email protected]:/backup

See also:

Re-read The Partition Table Without Rebooting Linux System

Posted on in Categories File system, Linux, RedHat/Fedora Linux, Sys admin, Tips, Troubleshooting, Ubuntu Linux last updated May 8, 2006

If you are using hot swappable hard disk and created new partition using fdisk then you need to reboot Linux based system to get partition recognized. Without reboot you will NOT able to create filesystem on your newly created or modified partitions with the mke2fs command.

However with partprobe command you should able to create a new file system without rebooting the box. It is a program that informs the operating system kernel of partition table changes, by requesting that the operating system re-read the partition table.

Linux command to gather hardware and software snapshot for troubleshooting

Posted on in Categories CentOS, Debian Linux, Gentoo Linux, Hardware, Howto, Linux, RedHat/Fedora Linux, Tips last updated May 1, 2006

When you work in tech support department and deal with inexperienced clients debugging problems turns into a nightmare. As a sysadmin, you won’t become too paranoid if less experienced people have root-access. As a consultant, you won’t feel isolated if you don’t have remote access to your systems. As a support engineer, you won’t become frustrated if a customer has fiddled around with some important config file and you have to find which. As a performance tuner, you can capture the state of the system configuration in between performance tests/benchmarks.

Luckily, some nifty tools can create a system’s hardware and software configuration snapshot. This kind of information is valuable asset while troubleshooting problems.

dconf (System config collector) is one of such tool. It allows to take your system configuration with you on the road, compare identical systems (like nodes in a cluster) to troubleshoot HW or SW problems, indeed a lifesaver.

Dconf is also useful in projects where you have to manage changes as a team. Dconf can run periodically and send out system changes to a list of email addresses so that they can be revised and discussed in group.

You can customize your dconf configuration for specific needs, like making a profile of your web server’s hardware or copy specific software configuration files to send out or compare with other systems.

As a sysadmin, you will not become too paranoid if less experienced people have root-access. As a consultant, you will not feel isolated if you do not have remote access to your systems. As a support engineer, you will not become frustrated if a customer has fiddled around with some important config file and you have to find which. As a performance tuner, you can capture the state of the system configuration in between performance tests/benchmarks.

Install dconf

If you are using Debian / Ubuntu Linux then type the command:
# apt-get install dconf
You can download Dconf for RedHat or Suse Linux here

Create a system’s hardware and software configuration snapshot

Once installed you can simply create a snapshot using dconf command:
# dconf
It will write snapshot in /var/log/dconf/ directory. To view current snapshot info, enter:
# zcat /var/log/dconf/dconf-$HOSTNAME-latest.log.gz
To check the latest changes against the previous snapshot:
# zdiff -u /var/log/dconf/dconf-$HOSTNAME-previous.log.gz /var/log/dconf/dconf-$HOSTNAME-latest.log.gz

See also:

Why command df and du reports different output?

Posted on in Categories Debian Linux, File system, FreeBSD, RedHat/Fedora Linux, Solaris, Suse Linux, Troubleshooting, UNIX last updated January 31, 2006

You will never notice something like this on FreeBSD or Linux Desktop home system or your personal UNIX or Linux workstation. However, sometime on a production UNIX server you will notice that both df (display free disk space) and du (display disk usage statistics) reporting different output. Usually df will output a bigger disk usage than du.

If Linux or UNIX inode is deallocated you will see this problem. If you are using clustered system (file system such as GFS) you may see this scenario commonly.

Note following examples are FreeBSD and GNU/Linux specific.

Following is normal output of df and du for /tmp filesystem:
# df -h /tmp
Output:

Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1e    496M     22M    434M     5%    /tmp

Now type du command:
# du -d 0 -h /tmp/
Output:

22M    /tmp/

Why is there a mismatch between df and du outputs?

However, some time it reports different output (a bigger disk usage), for example:
# df -h /tmp/
Output:

Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1e    496M     39M    417M     9%    /tmp

Now type du command:
# du -d 0 -h /tmp/
Output:

 22M    /tmp/

As you see, both df and du reporting different output. Many new UNIX admin get confused with output (39M vs 22M).

Open file descriptor is main causes of such wrong information. For example if file called /tmp/application.log is open by third party application OR by a user and same file is deleted, both df and du reports different output. You can use lsof command to verify this:
# lsof | grep tmp
Output:

bash   594  root  cwd   VDIR  0,86      512      2 /tmp
bash   634  root  cwd   VDIR  0,86      512      2 /tmp
pwebd  635  root  cwd   VDIR  0,86      512      2 /tmp
pwebd  635  root  3rW   VREG  0,86 17993324     68 /tmp (/dev/ad0s1e)
pwebd  635  root   5u   VREG  0,86        0     69 /tmp (/dev/ad0s1e)
lsof   693  root  cwd   VDIR  0,86      512      2 /tmp
grep   694  root  cwd   VDIR  0,86      512      2 /tmp

You can see 17993324K file is open on /tmp by pwebd (our in house software) but deleted accidentally by me. You can recreate above scenario in your Linux, FreeBSD or Unixish system as follows:

First, note down /home file system output:
# df -h /home
# du -d 0 -h /home

If you are using Linux then use du as follows:
# du -s -h /tmp

Now create a big file:
# cd /home/user
# cat /bin/* >> demo.txt
# cat /sbin/* >> demo.txt

Login on other console and open file demo.txt using vi text editor:
# vi /home/user/demo.txt

Do not exit from vi (keep it running).

Go back to another console and remove file demo.txt
# rm demo.txt
Now run both du and df to see the difference.
# df -h /home
# du -d 0 -h /home

If you are using Linux then use du as follows:
# du -s -h /tmp

Login to another terminal and close vi.

Now close the vi and the root cause of the problem should be resoled, the du and df outputs should be correct.

Linux / UNIX: Finding and locating files with find command part # 2

Posted on in Categories CentOS, File system, FreeBSD, GNU/Open source, Howto, Linux, Linux desktop, OpenBSD, RedHat/Fedora Linux last updated July 23, 2005

In the first part we talked about find command basic usage.

Now let us see how to use find command
(a) To gain lots of useful information about users and their files

(b) Monitor and enhance the security of system using find command

Finding all set user id files

setuid (“suid”) and setgid are access right flags that can be assigned to files and directories on a Unix based operating system. They are mostly used to allow users on a computer system to execute binary executables with temporarily elevated privileges in order to perform a specific task.
# find / -perm +u=s
OR
# find / -perm +4000

See also, shell script to find all programs and scripts with setuid set on.

Finding all set group id files

# find / -perm +g=s
OR
# find / -perm +2000

See also, shell script to find all programs and scripts with setgid bit set on.

Finding all large directories

To find all directories taking 50k (kilobytes) blocks of space. This is useful to find out which directories on system taking lot of space.
# find / -type d -size +50k
Output:

/var/lib/dpkg/info
/var/log/ksymoops
/usr/share/doc/HOWTO/en-html
/usr/share/man/man3

Finding all large files on a Linux / UNIX

# find / -type f -size +20000k
Output:

var/log/kern.log
/sys/devices/pci0000:00/0000:00:02.0/resource0
/sys/devices/pci0000:00/0000:00:00.0/resource0
/opt/03Jun05/firefox-1.0.4-source.tar.bz2

However my favorite hack to above command is as follows:
# find / -type f -size +20000k -exec ls -lh {} \; | awk '{ print $8 ": " $5 }'

/var/log/kern.log: 22M
/sys/devices/pci0000:00/0000:00:02.0/resource0: 128M
/sys/devices/pci0000:00/0000:00:00.0/resource0: 256M
/opt/03Jun05/firefox-1.0.4-source.tar.bz2: 32M

Above command will find all files block size greater than 20000k and print filename followed by the file size. Output is more informative as compare to normal find command output 😀

Linux: Finding and Locating files with find command part # 1

Posted on in Categories CentOS, File system, Linux, Linux desktop, RedHat/Fedora Linux last updated July 22, 2005

Many newcomers find it difficult use the find command at shell prompt under Linux / *BSD or Solairs UNIX oses. Find is nifty tool on remote server where UNIX admin can find out lot of information too. Desktop users may find handy GNOME Search tool as a utility for finding files on system. Find command can perform a search based on a variety of search constraints. It searches through one or more directory tree(s) of a filesystem, locating files based on some user-specified criteria. By default, find returns all files below the current working directory. Further, find allows the user to specify an action to be taken on each matched file. Thus, it is an extremely powerful program for applying actions to many files. It also supports regexp matching.

GNOME Search Tool GUI Program

GNOME Search Tool is a utility for finding files on your system. To perform a basic search, you can type a filename or a partial filename, with or without wildcards. You can start this program from menus or by typing following command at shell prompt:
$ gnome-search-tool &
Internally GNOME Search Tool uses the find, grep, and locate UNIX commands. The case sensitivity of the search depends on your operating system. For example, on Linux, the find, grep, and locate commands support the -i option, so all searches are case-insensitive.

the GNOME Linux / UNIX GUI Search Tool(click to enlarge)

Find command syntax

find {search-path} {file-names-to-search} {action-to-take}

Where,

  • search-path : Define search path (default current directory). For example search in /home directory.
  • file-names-to-search : Name of the file you wish to find. For example all c files (*.c)
  • action-to-take : Action can be print file name, delete files etc. Default action is print file names.

Find command examples

Let us try out some examples.

Finding files and printing their full name

You wish to find out all *.c (all c source code) files located under /home directory, enter:
$ find /home -name "*.c"

You would like to find httpd.conf file location:
$ find / -name httpd.conf

Finding all files owned by a user

Find out all files owned by user vivek:
# find / -user vivek

Find out all *.sh owned by user vivek:
# find / -user vivek -name "*.sh"

Finding files according to date and time

Files not accessed in a time period – It is useful to find out files that have or have not been accessed within a specified number of days. Following command prints all files not accessed in the last 7 days:
# find /home -atime +7

  • -atime +7: All files that were last accessed more than 7 days ago
  • -atime 7: All files that were last accessed exactly 7 days ago
  • -atime -7: All files that were last accessed less than7 days ago

Finding files modified within a specified time – Display list of all files in /home directory that were not last modified less than then days ago.
# find /home -mtime -7

Finding newer (more recently) modified files

Use -newer option to find out if file was modified more recently than given file.
# find /etc/apache-perl -newer /etc/apache-perl/httpd.conf

Finding the most recent version of file

It is common practice before modifying the file is copied to somewhere in system. For example whenever I modify web server httpd.conf file I first make backup. Now I don’t remember whether I had modified the /backup.conf/httpd.conf or /etc/apache-perl/httpd.conf. You can use the find command as follows (tip you can also use ls -l command):
find / -name httpd.conf -newer /etc/apache-perl/httpd.conf

Locate command

The locate command is often the simplest and quickest way to find the locations of files and directories on Linux and other Unix-like operating systems.

For example, the following command uses the star wildcard to display all files on the system that have the .c filename extension:
# locate "*.c"

Further readings

  • Read find and locate command man page for more information.