Humor: Writing Web Service and Pages using Bourne Shell

Posted on in Categories Howto, Humor, Shell scripting, Tips last updated November 22, 2007

Why should shell languages be the only ones left out of the ${YOUR_LANGUAGE_HERE} Server Pages fad? This document chronicles foray into the not-so-fascinating world of ${YOUR_LANGUAGE_HERE} Server Pages technology. All of the code developed is available for free download, so you can use Bourne Shell Server Pages to build your very own killer Web application.

=> Bourne Shell Server Pages [hyperrealm.com]

Testing Connectivity with fping and send mail if any hosts are unreachable

Posted on in Categories CentOS, Debian Linux, Gentoo Linux, Hardware, Howto, Linux, Networking, Shell scripting, Sys admin, Tips, Ubuntu Linux, UNIX last updated November 21, 2007

fping is one my favorite network profiling / scripting tool. It uses the Internet Control Message Protocol (ICMP) echo request to determine if a target host is responding or not.

Unlike ping , fping is meant to be used in scripts, so its output is designed to be easy to parse.

You can easily write perl / shell script to check a list of hosts and send mail if any are unreachable.

fping command example

Just type the following command to see if we can reach to router:
$ fping router
Output:

router is alive

You can read list of targets (hosts / servers) from a file. The -f option can only be used by the root user. Regular users should pipe in the file via
I/O redirectors (stdin). For example read all host names from ~/.ping.conf file
$ fping < ~/.ping.conf

You can also netmask to ping the entire network i.e generate a target list from a supplied IP netmask. For example, ping the class C 192.168.1.x:
$ fping -g 192.168.1.0/24
or
$ fping -g 192.168.1.0 192.168.1.255

Sample shell script to send email if host is down

#!/bin/bash
HOSTS="router sun printer laptop sony-laptop xbox backup-server"
DLIST=""
for h in $HOSTS
do
  fping -u $h >& /dev/null
  if [ $? -ne 0 ]; then
          echo ${h} host is down send email
          # mail -s "Host ${h} down" [email protected] </dev/null
  fi
done

Another good example is when you want to perform an action only on hosts that are currently reachable.

#!/usr/bin/perl
$myHosts = ‘cat /etc/hosts.backup | fping -a‘;
foreach $host (split(/\n/,$myHosts)) {
        # take action or call other function
}

Bash Shell Completing File, User and Host Names Automatically

Posted on in Categories CentOS, Debian Linux, File system, RedHat/Fedora Linux, Shell scripting, Suse Linux, Tips, Ubuntu Linux, UNIX last updated November 14, 2007

Bash can auto complete your filenames and command name. It can also auto complete lots of other stuff such as:
=> Usernames

=> Hostname

=> Variable names

=> Fine tuning files and names with ESC keys

Match variable

If the text begins with $, bash will look for a variable. For example, open terminal and type echo $T and press TAB key, shell will complete that to echo $TERM:
echo $T {hit-tab-key}

Match Username

If the test begins with ~ (tilde), bash will look for a user name. For example, open terminal and type cat ~g and press TAB key, shell will complete that to cat ~guess/file.txt:
cat ~g {hit-tab-key}

Match hostname

If the test begins with @, bash will look for a host name. For example, open terminal and type scp file.txt [email protected] and press TAB key, shell will complete that to scp file.txt [email protected]:
scp file.txt [email protected] {hit-tab-key}
Please note that you need proper host resoultion configured to work with this hack via NIS or hosts file. Also after shell completes the command name / username or filename hit the [ENTER] key.

Fine tunning Shell Completing stuff with ESC key

Bash allows you to fine tune file completion using ESC key combinations. People get amazed when I use ESC combination in front of them. For example, to inserts all possible completions into your command use ESC+*. Let us see how to backup all /etc/*.conf files, type the command:
tar -zcvf /dev/rt0 /etc/*.conf {hit ESC followed by *}
As soon as you hit Esc+*, shell replaces the /etc/*.conf part with names of all matching wild card patterns
tar -zcvf /dev/rt0 /etc/aatv.conf /etc/adduser.conf /etc/apg.conf /etc/brltty.conf /etc/ca-certificates.conf /etc/cvs-cron.conf /etc/cvs-pserver.conf /etc/debconf.conf ....

To displays all possible completions of command or filenames or username type ESC+?, to display all username start with the word le, type
cat ~le {hit ESC followed by ?}

complete command

There is also in built command called complete. It is used to specify how arguments are to be completed for a command. For example, when you type passwd (or any other user admin command such as su / usermod etc) and hit tab key, bash will show you a list of all available users i.e. all user admin related commands will see only user names:
complete -u su usermod userdel passwd chage write chfn groups slay w
Now type passwd and hit tab key to see all username:
passwd {hit tab key}
Output:

avahi          bin            dhcp           gdm            haldaemon      klog           mail           news           root           sys            uucp
avahi-autoipd  cupsys         dnsmasq        gnats          hplip          list           man            nobody         sshd           syslog         vivek
backup         daemon         games          guest          irc            lp             messagebus     proxy          sync           telnetd        www-data
[email protected]:/tmp$ passwd 

Cool, huh? There is a nice file included with almost all distro to complete lots of stuff using complete command. Just add following line to your bash startup file:
source /etc/bash_completion

Further reading:

How to: Check the bash shell script is being run by root or not

Posted on in Categories CentOS, Debian Linux, FreeBSD, Linux, RedHat/Fedora Linux, Shell scripting last updated November 12, 2007

Sometime it is necessary to find out if a shell script is being run as root user or not.

When user account created a user ID is assigned to each user. BASH shell stores the user ID in $UID variable. Your effective user ID is stored in $EUID variable. You can

Old way…

You can easily add a simple check at the start of a script:

Check the script is being run by root user

#!/bin/bash
# Init
FILE="/tmp/out.$$"
GREP="/bin/grep"
#....
# Make sure only root can run our script
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi
# ...

New way: Using EUID

#!/bin/bash
# Init
FILE="/tmp/out.$$"
GREP="/bin/grep"
#....
# Make sure only root can run our script
if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi
# ...

Mount /dev/sdb1 only if you are a root

#!/bin/bash
if [[ $EUID -ne 0 ]]; then
  echo "You must be a root user" 2>&1
  exit 1
else
  mount /dev/sdb1 /mnt/disk2
fi

Updated for accuracy and more examples.

Linux Save the output of a command in a logfile

Posted on in Categories CentOS, Gentoo Linux, Howto, Linux, RedHat/Fedora Linux, Shell scripting, Suse Linux, Sys admin, Tips, Ubuntu Linux, UNIX last updated November 11, 2007

You can use logsave command to save the output of a command in a logfile. General syntax is as follows:
logsave /path/to/logfile command-name argument(s)

The logsave program will execute command-name with the specified argument(s), and save a copy of its output to logfile. If the containing directory for logfile does not exist, logsave will accumulate the output in memory until it can be written out. A copy of the output will also be written to standard output. If command-name is a single hyphen (-), then instead of executing a program, logsave will take its input from standard input and save it in logfile

logsave is useful for saving the output of initial boot scripts until the /var partition is mounted, so the output can be written to /var/log. For example, save output of ls command to output.log file:
logsave /tmp/output.txt ls
For example run lftp and save remote server file list to /tmp/filelist.txt for other scripts:
logsave /tmp/filelist.txt 'lftp -u -e "cd pub;ls;quit;" admin,mypassword ftp.nixcraft.in '

Protect Your Network from spamming, scanning, harvesting and dDoS attacks with DROP List

Posted on in Categories Debian Linux, Howto, Iptables, Linux, Networking, RedHat/Fedora Linux, Security, Shell scripting, Suse Linux, Sys admin, Tips, UNIX last updated October 24, 2007

DROP (Don’t Route Or Peer) is an advisory “drop all traffic” list, consisting of stolen ‘zombie’ netblocks and netblocks controlled entirely by professional spammers. DROP is a tiny sub-set of the SBL designed for use by firewalls and routing equipment.

DROP is currently available as a simple text list, but will also be available shortly as BGP with routes of listed IPs announced via an AS# allowing networks to then null those routes as being IPs that they do not wish to route traffic for.

The DROP list will NEVER include any IP space “owned” by any legitimate network and reassigned – even if reassigned to the “spammers from hell”. It will ONLY include IP space totally controlled by spammers or 100% spam hosting operations. These are “direct allocations” from ARIN, RIPE, APNIC, LACNIC, and others to known spammers, and the troubling run of “hijacked zombie” IP blocks that have been snatched away from their original owners (which in most cases are long dead corporations) and are now controlled by spammers or netblock thieves who resell the space to spammers.

When implemented at a network or ISP’s ‘core routers’, DROP will protect all the network’s users from spamming, scanning, harvesting and dDoS attacks originating on rogue netblocks.

Shell script to apply DROP

Here is a shell script, you need to run on Linux based firewall / router / dedicated Linux web / mail server:

#!/bin/bash
FILE="/tmp/drop.lasso"
URL="http://www.spamhaus.org/drop/drop.lasso"
echo ""
echo -n "Applying DROP list to existing firewall..."
[ -f $FILE ] && /bin/rm -f $FILE || :
cd /tmp
wget $URL
blocks=$(cat $FILE  | egrep -v '^;' | awk '{ print $1}')
iptables -N droplist
for ipblock in $blocks
do
 iptables -A droplist -s $ipblock -j LOG --log-prefix "DROP List Block"
 iptables -A droplist -s $ipblock -j DROP
done
iptables -I INPUT -j droplist
iptables -I OUTPUT -j droplist
iptables -I FORWARD -j droplist
echo "...Done"
/bin/rm -f $FILE

Call above script from existing firewall script every 24 hrs to update and block list. Every time it’s run by crontab it will download the list and reapply the changes. You may need to modify above script to delete droplist chain before applying list. Please note that if you are using Cicso routers, use this script for the same purpose. You can also use CISCO ‘null route’ command:

ip route <network> <mask> null0

If you don’t want to play with iptables, null route all bad ips using following route command under Linux syntax:
# route add <IP> gw 127.0.0.1 lo
# route add -net <IP/mask> gw 127.0.0.1 lo

Try this and you will surprise to see how much spam and other bad stuff can be blocked.

How to: Debug SSL certificate problems from the shell prompt

Posted on in Categories Apache, Howto, Linux, Security, Shell scripting, Sys admin, Tips, Troubleshooting, UNIX last updated October 18, 2007

OpenSSL is a cryptography toolkit implementing the Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1) network protocols and related cryptography standards required by them.

It also includes the openssl command, which provides a rich variety of commands You can use the same command to debug problems with SSL certificates.

To test the secure connections to a server, type the following command at a shell prompt:
openssl s_client -connect ssl.servername.com:443
Where,

  • s_client : This implements a generic SSL/TLS client which can establish a transparent connection to a remote server speaking SSL/TLS. It’s intended for testing purposes only and provides only rudimentary interface functionality but internally uses mostly all functionality of the OpenSSL ssl library. You can also connect to secure mail server (such as POP3S ~ 995) / web server port (443) and issue commands.

For example connect to www.cyberciti.biz at port 443, enter:
openssl s_client -connect www.cyberciti.biz:443
Output:

CONNECTED(00000003)
depth=0 /C=IN/ST=Berkshire/L=Newbury/O=My Company [email protected]
verify error:num=18:self signed certificate
verify return:1
depth=0 /C=IN/ST=Berkshire/L=Newbury/O=My Company [email protected]
verify return:1
---
Certificate chain
 0 s:/C=IN/ST=Berkshire/L=Newbury/O=My Company [email protected]
   i:/C=IN/ST=Berkshire/L=Newbury/O=My Company [email protected]
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDhDCCAu2gAwIBAgIJAMgof8IIjdD9MA0GCSqGSIb3DQEBBQUAMIGJMQswCQYD
VQQGEwJJTjESMBAGA1UECBMJQmVya3NoaXJlMRAwDgYDVQQHEwdOZXdidXJ5MRcw
FQYDVQQKEw5NeSBDb21wYW55IEx0ZDEYMBYGA1UEAwwPKi5jeWJlcmNpdGkuYml6
MSEwHwYJKoZIhvcNAQkBFhJ2aXZla0BuaXhjcmFmdC5jb20wHhcNMDcwOTIwMTEw
MzExWhcNMDgwOTE5MTEwMzExWjCBiTELMAkGA1UEBhMCSU4xEjAQBgNVBAgTCUJl
cmtzaGlyZTEQMA4GA1UEBxMHTmV3YnVyeTEXMBUGA1UEChMOTXkgQ29tcGFueSBM
dGQxGDAWBgNVBAMMDyouY3liZXJjaXRpLmJpejEhMB8GCSqGSIb3DQEJARYSdml2
ZWtAbml4Y3JhZnQuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCzYIxz
2JGAgYUJhLnmDbtC5kc+S4AHJHGTZmFuxVZDFOacHPitS4ohwzDadruUONucVZJY
Gi1M9j1jPUBX7oZ7F/Y7pbEO/YMfEPPDGq6uEkkwHDTXRH1qgL6v7q9XtP9Dafck
n3+YeTO0eYk0Or9a6xBqJmuN6M+ajprfXmQ9cwIDAQABo4HxMIHuMB0GA1UdDgQW
BBQH94MQusbxTH8UxH83EpmMz5v5UjCBvgYDVR0jBIG2MIGzgBQH94MQusbxTH8U
xH83EpmMz5v5UqGBj6SBjDCBiTELMAkGA1UEBhMCSU4xEjAQBgNVBAgTCUJlcmtz
aGlyZTEQMA4GA1UEBxMHTmV3YnVyeTEXMBUGA1UEChMOTXkgQ29tcGFueSBMdGQx
GDAWBgNVBAMMDyouY3liZXJjaXRpLmJpejEhMB8GCSqGSIb3DQEJARYSdml2ZWtA
bml4Y3JhZnQuY29tggkAyCh/wgiN0P0wDAYDVR0TBAUwAwEB/zANBgkqhkiG9w0B
AQUFAAOBgQActMUY+8CbFCcxGWvmN95/LsVxZMWWqOGoiFOgqKI9t1T/nBN6TrW5
MYeMwcMbI4OoBo5vnp6mHzcZNoMPiK9DITgb8O/P0EUhjL+QdARJYZX6lLB3qJkP
ts65VY0rFxjIhndtixKP1fLC/K2ovzo+43pE1EQB6UhjhHlHV2v34w==
-----END CERTIFICATE-----
subject=/C=IN/ST=Berkshire/L=Newbury/O=My Company [email protected]
issuer=/C=IN/ST=Berkshire/L=Newbury/O=My Company [email protected]
---
No client certificate CA names sent
---
SSL handshake has read 1066 bytes and written 316 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: 989C62FBF87884C9F6904DD216A9A36189BE660059F419DAA16711AF2A7F42D4
    Session-ID-ctx:
    Master-Key: 9A01374F14D7300E8DD02BE2AA3C3567F26E1BB00267D5AB0156C6C11A10EB0D8424FBD06D3B15013B4FBA0F121EC99D
    Key-Arg   : None
    Start Time: 1192732059
    Timeout   : 300 (sec)
    Verify return code: 18 (self signed certificate)
---

Using grep you can see the SSL and TLS connection handshaking, security negotiate, public keys and transfer of digital certificates and key information to the client:
$ openssl s_client -state -nbio -connect www.cyberciti.biz:443 2>&1 | grep "^SSL"
Output:

SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:error in SSLv3 read finished A
SSL_connect:error in SSLv3 read finished A
SSL_connect:SSLv3 read finished A
SSL handshake has read 1066 bytes and written 316 bytes
SSL-Session:

Further readings:

=> OpenSSL man pages and documentation.

ddate: Converts Gregorian dates to Discordian dates

Posted on in Categories Humor, Linux, Shell scripting, UNIX last updated October 16, 2007

Most of you may be aware of date command. There is also a ddate command. It converts Gregorian dates to Discordian dates or Erisian calendar, which is an alternative calendar used by some adherents of Discordianism. It is specified on page 00034 of the Principia Discordia:

The Discordian calendar has five 73-day seasons: Chaos, Discord, Confusion, Bureaucracy, and The Aftermath. The Discordian year is aligned with the Gregorian calendar and begins on January 1, thus Chaos 1, 3173 YOLD is January 1, 2007 Gregorian.

Just type ddate:
$ ddate
Output:

Today is Setting Orange, the 71st day of Bureaucracy in the YOLD 3173

For fun put following alias in /etc/profile (try out on 1st April 😉 ):
alias date=ddate

If called with no arguments, ddate will get the current system date, convert this to the Discordian date format and print this on the standard output.. Alternatively, a Gregorian date may be specified on the command line, in the form of a numerical day, month and year.

Quick Shell Tip: Remove grep command while grepping something using ps command

Posted on in Categories Linux, Shell scripting, Tip of the day, UNIX last updated October 15, 2007

Generally you use ps command to find out all running process. You may also pipe out ps command output via grep command to pickup desired output.

Basically you don’t want display grep command as the process.

Let us run combination of ps and grep command to find out all perl processes:
$ ps aux | grep perl
Output:

vivek      4611  0.0  0.7  10044  6068 ?        Ss   02:40   0:00 /usr/bin/perl apps/monitor/gwl.pl
root      4853  0.0  0.7  10044  6068 ?        Ss   02:40   0:00 /usr/bin/perl /usr/share/webmin/miniserv.pl /etc/webmin/miniserv.conf
vivek      5166  0.0  0.0   2884   748 pts/0    R+   03:06   0:00 grep perl

In above example you are getting the grep process itself. To ignore grep process from output, type any one of the following:
$ ps aux | grep perl | grep -v grep
OR
$ ps aux | grep '[p]erl'

How to: Run All Shell / Perl / Python Scripts in a Directory

Posted on in Categories Linux, Shell scripting, Sys admin, Tips, UNIX last updated October 12, 2007

I’ve directory called /home/vivek/scripts/daily with over 25 perl / shell / python scripts for managing daily tasks. One day for some weird reason my crond died and I did not noticed the incident for 2 days.

Now crond is started and I’d like to run all those scripts. Here is a quick for loop running all scripts in a directory called ~/scripts/daily/:

for s in ~/scripts/daily/*;do [ -x $s ] && $s || : ;done

Above script will run each and every executable script it finds in a directory.

Update: As pointed out by jeff (see below), you can use run-parts shell script for the same purpose:
$ run-parts ~/scripts/daily/*