Processing the delimited files using cut and awk

by nixcraft · 4 comments

From wikipedia, "Delimited data uses specific characters (delimiters) to separate its values. Most database and spreadsheet programs are able to read or save data in a delimited format.".

So how do you process delimited files under Linux shell prompt?

Processing the delimited files using cut

cut command print selected parts of lines from each FILE (or variable) i.e. it remove sections from each line of files:

For example /etc/passwd file is separated using character : delimiters.

To print list of all users, type the following command at shell prompt:

$ cut -d: -f1 /etc/passwd

Output:

root
you
me
vivek
httpd

Where,

  • -d : Specifies to use character : as delimiter
  • -f1 : Print first field, if you want print second field use -f2 and so on...

Now consider variable service. Let us print out mail word using cut command:
$ service="http mail ssh"
$ echo $service | cut -d' ' -f2

mail

Note that a blank space is used as delimiter.

Processing the delimited files using awk

You can also use awk command for same purpose:
$ awk -F':' '{ print $1 }' /etc/passwd

Output:

root
you
me
vivek
httpd

Where,

  • -F: - Use : as fs (delimiter) for the input field separator
  • print $1 - Print first field, if you want print second field use $2 and so on

Processing the delimited files using cut or awk is an essential skill for sys admin. Let us say you want to find out if particular service is active or not using shell script (download link):

#!/bin/bash

ports="22 80 25"
service="SSH WEB MAIL"
c=1

echo "Running services status:"

/bin/netstat -tulpn | grep -vE '^Active|Proto' | while read LINE
do
 # get active port name and use : as delimiter
 t=$(echo $LINE | awk '{ print $4}' | cut -d: -f2)
 [ "$t" == "" ] && t=-1 || :
 # get service name from $services and : as delimiter
 sname=$(echo $service | cut -d' ' -f$c)
 sstatus="$sname: No"
 # now compare port
 for i in $ports
 do
  if [ $i -eq $t ]; then
   sstatus="$sname: Ok"
  fi
 done
 # display service status as OK or NO
 echo "$sstatus"
 #next service please
 c=$( expr $c + 1 )
 # break afer 3 services
 [ $c -ge 4 ] && break || :
done

Execute a script:
$ chmod +x script.sh
$ ./script.sh

Output:

SSH: Yes
WEB: No
MAIL: Yes

As you see above script use cut and awk to extract one or many contiguous or non-contiguous fields from shell variables and command output. Download complete working script that sends an email alert to admin user.

Featured Articles:

Want to read Linux tips and tricks, but don't have time to check our blog everyday? Subscribe to our daily email newsletter to make sure you don't miss a single tip/tricks. Subscribe to our weekly newsletter here!

{ 4 comments… read them below or add one }

1 James D. Keeline 08.09.06 at 10:07 pm

This article has some bad formatting in the source code. The single (‘) and double quotes (“) are presented as if they were smart quotes. However, if one tries to type in the back-tick (`) when the single quote (‘) was intended, Bash will reject this. The display should use a standard monospaced font with no smart quotes.

2 nixcraft 08.09.06 at 11:12 pm

James,

Thanks for suggestion :) I will update CSS code, which is responsible for displaying source code using code class.

Appreciate your post.

3 steve_w 04.04.09 at 8:52 am

Nixcraft …I think James meant to say very good post with very clear examples…you’ve always gotta be careful of quotes after they’ve been copied and pasted around a windows environment or in word docs etc anyhow…

4 Ahmet Alp 04.23.09 at 6:40 pm

Thank you some much.

Super script.

Leave a Comment

You can use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Previous post:

Next post: