How To Use awk In Bash Scripting

How do I use awk pattern scanning and processing language under bash scripts? Can you provide a few examples?

Awk is an excellent tool for building UNIX/Linux shell scripts. AWK is a programming language that is designed for processing text-based data, either in files or data streams, or using shell pipes. In other words you can combine awk with shell scripts or directly use at a shell prompt.

This pages shows how to use awk in your bash shell scripts.

Print a Text File

awk '{ print }' /etc/passwd
OR
awk '{ print $0 }' /etc/passwd

Print Specific Field

Use : as the input field separator and print first field only i.e. usernames (will print the the first field. all other fields are ignored):
awk -F':' '{ print $1 }' /etc/passwd
Send output to sort command using a shell pipe:
awk -F':' '{ print $1 }' /etc/passwd | sort

Pattern Matching

You can only print line of the file if pattern matched. For e.g. display all lines from Apache log file if HTTP error code is 500 (9th field logs status error code for each http request):
awk '$9 == 500 { print $0}' /var/log/httpd/access.log
The part outside the curly braces is called the “pattern”, and the part inside is the “action”. The comparison operators include the ones from C:

== !=  = ?:

If no pattern is given, then the action applies to all lines. If no action is given, then the entire line is printed. If “print” is used all by itself, the entire line is printed. Thus, the following are equivalent:
awk '$9 == 500 ' /var/log/httpd/access.log
awk '$9 == 500 {print} ' /var/log/httpd/access.log
awk '$9 == 500 {print $0} ' /var/log/httpd/access.log

Print Lines Containing tom, jerry AND vivek

Print pattern possibly on separate lines:
awk '/tom|jerry|vivek/' /etc/passwd

Print 1st Line From File

awk "NR==1{print;exit}" /etc/resolv.conf
awk "NR==$line{print;exit}" /etc/resolv.conf

Simply Arithmetic

You get the sum of all the numbers in a column:
awk '{total += $1} END {print total}' earnings.txt
Shell cannot calculate with floating point numbers, but awk can:
awk 'BEGIN {printf "%.3f\n", 2005.50 / 3}'

Call AWK From Shell Script

A shell script to list all IP addresses that accessing your website. This script use awk for processing log file and verification is done using shell script commands.

#!/bin/bash
d=$1
OUT=/tmp/spam.ip.$$
HTTPDLOG="/www/$d/var/log/httpd/access.log"
[ $# -eq 0 ] && { echo "Usage: $0 domain-name"; exit 999; }
if [ -f $HTTPDLOG ];
then
	awk '{print}' $HTTPDLOG >$OUT
	awk '{ print $1}' $OUT  |  sort -n | uniq -c | sort -n
else
	echo "$HTTPDLOG not found. Make sure domain exists and setup correctly."
fi
/bin/rm -f $OUT

AWK and Shell Functions

Here is another example. chrootCpSupportFiles() find out the shared libraries required by each program (such as perl / php-cgi) or shared library specified on the command line and copy them to destination. This code calls awk to print selected fields from the ldd output:

chrootCpSupportFiles() {
# Set CHROOT directory name
local BASE="$1"         # JAIL ROOT
local pFILE="$2"        # copy bin file libs
 
[ ! -d $BASE ] && mkdir -p $BASE || :
 
FILES="$(ldd $pFILE | awk '{ print $3 }' |egrep -v ^'\(')"
for i in $FILES
do
  dcc="$(dirname $i)"
  [ ! -d $BASE$dcc ] && mkdir -p $BASE$dcc || :
  /bin/cp $i $BASE$dcc
done
 
sldl="$(ldd $pFILE | grep 'ld-linux' | awk '{ print $1}')"
sldlsubdir="$(dirname $sldl)"
if [ ! -f $BASE$sldl ];
then
        /bin/cp $sldl $BASE$sldlsubdir
else
        :
fi
}

This function can be called as follows:
chrootCpSupportFiles /lighttpd-jail /usr/local/bin/php-cgi

AWK and Shell Pipes

List your top 10 favorite commands:
history | awk '{print $2}' | sort | uniq -c | sort -rn | head
Sample Output:

   172 ls
    144 cd
     69 vi
     62 grep
     41 dsu
     36 yum
     29 tail
     28 netstat
     21 mysql
     20 cat

Another example to find out domain expiry date:
$ whois cyberciti.com | awk '/Registry Expiry Date:/ { print $4 }'
Sample outputs:

2018-07-31T18:42:58Z

Awk Program File

You can put all awk commands in a file and call the same from a shell script using the following syntax:
awk -f mypgoram.awk input.txt

Awk in Shell Scripts – Passing Shell Variables TO Awk

You can pass shell variables to awk using the -v option:

n1=5
n2=10
echo | awk -v x=$n1 -v y=$n2 -f program.awk

Assign the value n1 to the variable x, before execution of the program begins. Such variable values are available to the BEGIN block of an AWK program:

BEGIN{ans=x+y}
{print ans}
END{}
🐧 If you liked this page, please support my work on Patreon or with a donation.
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source/DevOps topics:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
33 comments… add one
  • Brad G. Sep 9, 2011 @ 11:46

    Very nice site with good useful info. Thank you.

  • walnutpony123 Feb 3, 2012 @ 19:53

    Hi Can some one help me put together bash script that runs coulmns
    1 2 3 4 5 6
    and not 1
    2
    3
    4
    5
    6

    with column headers column1 = tj/art
    column2=tj/art
    column3=tj
    column4=art
    column5=tj
    column6=art

    and then redirect to a file >>

  • mini_csreddy Mar 6, 2012 @ 11:43

    Hi All,

    I have a shell script program which works in Unix platform and able to load data in staging and interface tables, but when it migrated to other instance of linux platform, its not working and not able to load data.

    Is there any change to do with awk and sed commands in Linux ?

    Please help in this regard…

    Please find is shell script program

    #print "Dollar 1 $1"
    org_id=$(print $1 | awk '{print $9}' | sed 's/"//g')
    print $org_id 
    LOGIN=$(print $1 | awk '{print $3}' | sed 's/"//g' | sed 's/FCP_LOGIN=//') 
    datafile=FORECAST'.'csv
    print $datafile
    print $GT_TOP
    forecast_file=$GT_TOP/bin/$org_id$datafile
    print $forecast_file
    
    
    #  sed 's/
    //' $forecast_file  > GT_TOP/bin/$org_id.dat
    #  mv  $GT_TOP/bin/$org_id.dat $forecast_file
    
    dos2ux $forecast_file> GT_TOP/bin/test.csv
    rm $forecast_file
    mv $GT_TOP/bin/test.csv $forecast_file
    chmod 777 $forecast_file
    
    
    sqlldr $LOGIN control=$GT_TOP/bin/GB_FORECAST_UPLOAD.ctl data=$forecast_file
    
    sqlplus -s $LOGIN << EOF
    BEGIN
    gb_forecast_prog_pkg.gb_forecast_pr;
    commit;
    END;
    /
    Exit;
    
    EOF
    
  • madhuri Oct 15, 2013 @ 9:16

    I have a file with the content as
    /a/b/sometext1
    /a/b/c/sometex2
    /a/b/d

    I want to separate out /a/b fields from all the lines.
    how can I use awk command.

    • Dinesh Oct 15, 2013 @ 13:07

      $ cat file
      /a/b/sometext1
      /a/b/c/sometex2
      /a/b/d

      $ awk -F/ ‘{print “/”$2″/”$3}’ file
      /a/b
      /a/b
      /a/b

      $ awk -F/ ‘{print $2,$3}’ file
      a b
      a b
      a b

  • Dinesh Oct 15, 2013 @ 13:07

    $ cat file
    /a/b/sometext1
    /a/b/c/sometex2
    /a/b/d

    $ awk -F/ ‘{print “/”$2″/”$3}’ file
    /a/b
    /a/b
    /a/b

    $ awk -F/ ‘{print $2,$3}’ file
    a b
    a b
    a b

  • lau May 11, 2015 @ 8:13

    can anyone help me with this

    .

    Write a bash shell script that prints out only the even numbered UID’s from /etc/passwd.

    1) Use a for loop with awk to extract the third field in /etc/passwd to a variable x.

    2) Set a variable xmod2=$(($x%2))

    3) Use an if condition to test if xmod2 is 0.

    4) If it is zero, then the UID is even, so echo $x.

  • Madhuri Jun 4, 2016 @ 12:05

    Hi ,
    Can anyone tell me ? I want to take input from different file into script ,it should take feild wise,my input file has different fields separated by ; ex: madhuri;beautiful;girl;innocent; like this I have input file where name it should take first field,at second input second word,how can I write for this can you explain me pls .

    • Anupam Kumar Aug 4, 2017 @ 6:19

      Madhuri,

      Please try this-
      cat filename.dat | cut -f1 -d’;’

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.