Awk is an excellent tool for building UNIX/Linux shell scripts. AWK is a programming language that is designed for processing text-based data, either in files or data streams, or using shell pipes. In other words you can combine awk with shell scripts or directly use at a shell prompt.
This pages shows how to use awk in your bash shell scripts.
Print a Text File
awk '{ print }' /etc/passwd
OR
awk '{ print $0 }' /etc/passwd
Print Specific Field
Use : as the input field separator and print first field only i.e. usernames (will print the the first field. all other fields are ignored):
awk -F':' '{ print $1 }' /etc/passwd
Send output to sort command using a shell pipe:
awk -F':' '{ print $1 }' /etc/passwd | sort
Pattern Matching
You can only print line of the file if pattern matched. For e.g. display all lines from Apache log file if HTTP error code is 500 (9th field logs status error code for each http request):
awk '$9 == 500 { print $0}' /var/log/httpd/access.log
The part outside the curly braces is called the “pattern”, and the part inside is the “action”. The comparison operators include the ones from C:
== != = ?:
If no pattern is given, then the action applies to all lines. If no action is given, then the entire line is printed. If “print” is used all by itself, the entire line is printed. Thus, the following are equivalent:
awk '$9 == 500 ' /var/log/httpd/access.log
awk '$9 == 500 {print} ' /var/log/httpd/access.log
awk '$9 == 500 {print $0} ' /var/log/httpd/access.log
Print Lines Containing tom, jerry AND vivek
Print pattern possibly on separate lines:
awk '/tom|jerry|vivek/' /etc/passwd
Print 1st Line From File
awk "NR==1{print;exit}" /etc/resolv.conf
awk "NR==$line{print;exit}" /etc/resolv.conf
Simply Arithmetic
You get the sum of all the numbers in a column:
awk '{total += $1} END {print total}' earnings.txt
Shell cannot calculate with floating point numbers, but awk can:
awk 'BEGIN {printf "%.3f\n", 2005.50 / 3}'
Call AWK From Shell Script
A shell script to list all IP addresses that accessing your website. This script use awk for processing log file and verification is done using shell script commands.
#!/bin/bash d=$1 OUT=/tmp/spam.ip.$$ HTTPDLOG="/www/$d/var/log/httpd/access.log" [ $# -eq 0 ] && { echo "Usage: $0 domain-name"; exit 999; } if [ -f $HTTPDLOG ]; then awk '{print}' $HTTPDLOG >$OUT awk '{ print $1}' $OUT | sort -n | uniq -c | sort -n else echo "$HTTPDLOG not found. Make sure domain exists and setup correctly." fi /bin/rm -f $OUT
AWK and Shell Functions
Here is another example. chrootCpSupportFiles() find out the shared libraries required by each program (such as perl / php-cgi) or shared library specified on the command line and copy them to destination. This code calls awk to print selected fields from the ldd output:
chrootCpSupportFiles() { # Set CHROOT directory name local BASE="$1" # JAIL ROOT local pFILE="$2" # copy bin file libs [ ! -d $BASE ] && mkdir -p $BASE || : FILES="$(ldd $pFILE | awk '{ print $3 }' |egrep -v ^'\(')" for i in $FILES do dcc="$(dirname $i)" [ ! -d $BASE$dcc ] && mkdir -p $BASE$dcc || : /bin/cp $i $BASE$dcc done sldl="$(ldd $pFILE | grep 'ld-linux' | awk '{ print $1}')" sldlsubdir="$(dirname $sldl)" if [ ! -f $BASE$sldl ]; then /bin/cp $sldl $BASE$sldlsubdir else : fi }
This function can be called as follows:
chrootCpSupportFiles /lighttpd-jail /usr/local/bin/php-cgi
AWK and Shell Pipes
List your top 10 favorite commands:
history | awk '{print $2}' | sort | uniq -c | sort -rn | head
Sample Output:
172 ls 144 cd 69 vi 62 grep 41 dsu 36 yum 29 tail 28 netstat 21 mysql 20 cat
Another example to find out domain expiry date:
$ whois cyberciti.com | awk '/Registry Expiry Date:/ { print $4 }'
Sample outputs:
2018-07-31T18:42:58Z
Awk Program File
You can put all awk commands in a file and call the same from a shell script using the following syntax:
awk -f mypgoram.awk input.txt
Awk in Shell Scripts – Passing Shell Variables TO Awk
You can pass shell variables to awk using the -v option:
n1=5 n2=10 echo | awk -v x=$n1 -v y=$n2 -f program.awk
Assign the value n1 to the variable x, before execution of the program begins. Such variable values are available to the BEGIN block of an AWK program:
BEGIN{ans=x+y} {print ans} END{}
🐧 33 comments so far... add one ↓
Category | List of Unix and Linux commands |
---|---|
File Management | cat |
Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
Network Utilities | dig • host • ip • nmap |
OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
Package Manager | apk • apt |
Processes Management | bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time |
Searching | grep • whereis • which |
User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
Thanks for this Helpful post, but is it possible to call bash user-defined function within
awk.
eg:
!#/bin/bash
function Print_NATE (){
echo $1
}
awk ‘{ Print_name “ALOK’ }’
I know the awk syntax is wrong, can you provide me small HOW TO on this.
Thanks
Use system()
WTF?
41 dsu
some of my one liner awk tricks:
— To convert squid log timestamps to readable, sortable format:
— To avoid having to cut/paste the above, I have in my .profile file:
— To use AWK to process comma separated data:
— To count complex pattern occurrence
@nerdoug, Thanks for sharing your awk one-liners.
@ nfree, dsu is my own custom made tool to perform certain operations.
An alternate way to call awk programs (or “scripts”), in either Linux or cygwin:
have awk script start with the line:
#!/usr/bin/gawk -f
have bash calling sequences like:
./script input-file
./script -v param1=”08/15/” -v param2=”MISS/5″ dated > output-file
Thanks Vivek Gite, but
the issue comes when we need to pass an argument to the user-define-functions.
Can you paste any example, which take any argument .
Thanks
Hope this illustrates passing arguments from bash to an awk script:
Thanks vivek n nerdoug …. it looks awsome tut for newbies
It is a very useful post.
Comments are also vey helpful.
Thanks
HI nerdoug
Thanks for the nice post ” Hope this illustrates passing arguments from bash to an awk script:”
But I am looking for just inverse case
“passing arguments form awk to bash user-define function.”
It will be a great help if you can come up with some example.
Thanks for your time and such a useful note.
–Alok
Yes, you need to keep everything in same shell. Awk calls sh whenever system() is used. To OP, you better use perl or python if you need really complicated stuff.
dear sir,
I would like to know how could I use awk with conditional statement.
for ex . –
if there is two condition (let 14 , 17 ( which are in 3rd column)) & another 2 conditions(let ab,cd (which are in 4th & 6th column respectively)).
& it will porceed if condition ”14de” or ”17de” are true .
(all the data’s are in ”xyz.txt” file)
I have made like :-
‘cat xyz.txt | awk -F”,” ‘{ if(((substr($3,1,2)==14) || (substr($3,1,2)==17))) && (substr($4,1,4)==”\”ab\””) && (substr($6,1,4)!=”\”cd\””)) print($4) }’ “| wc -l’
.
I have a comment on your bash, and a couple on your awk.
bash: doing cat infile | awk ‘{stuff}’ > outfile causes each line in the input file to be processed twice, once by cat, and once by awk.
if instead you do awk ‘{stuff}’ infile > outfile there’s only one pass through the file. If you’re processing big files, this can be significant.
awk: I’m not sure if your boolean logic will work, because I don’t understand whatt you meant by the goal of proceeding if condition ”14de” or ”17de” are true. You can do some of the filtering in a regular expression, and perhaps all of it depending on the formating of your data. for example, I think you can replace the first part of your if logic like this:
awk -F, /^1[47]/ ‘{if((substr($4,1,4)==”\”ab\””) && (substr($6,1,4)!=”\”cd\””)) {print $4 } }’ xyz.txt | wc -l
I don’t think the last single quote and the last double quote are needed in your awk.
hope this helps.
@Vivek.
Thanks for your topic.
BTW, reagrding passing shell variables to [awk],
that can be used in BEGIN block,
one can also use ENVIRON array:
Try:
export myawkvar=something; awk ‘BEGIN {print “BEGIN:” ENVIRON[“myawkvar”] “:”}’
Would you add this builtin function to your topic?
–P
Thank u Vivek ji.
hi,
i wanted to do use awk in a bash funciton like
#sub
funciton myfun(){
local myval=$(awk -F= ‘/$1/ {print $2}’ myconfig.file)
echo $myval
}
#main
mymain=$(myfun “string”)
->but the problem is i have to use single quotes for awk in myfun function, where i cannot reference the variable $1, which is passed on from my main script.
->is there a solution to this ??
I assume that $1 is a parameter passed to your shell function and $2 is the column of the file you want to report; then, to get this content into the awk,
local myval=$(awk -F= '/'$1'/ {print $2}' myconfig.file)
a directory contains few folders where my files are located( all files in all folders need to be processed).now,how to use awk recursively for all files. in grep we have
$grep -r string *.asc
wat abt awk..?
Put it in a for loop
No, Anantha, this won’t do.
Again, think of Unix philosophy: Each command should do only one thing, and do it well.
http://en.wikipedia.org/wiki/Unix_philosophy
Most of the time, this implies using pipes.
Precisely in this case, each time a program needs to traverse directories recursively, the right tool is [find]. Then pipe its result into [xargs], that will call [awk].
Search The Fantastic Web with these commands names (unix find xargs awk) and it’s up to you now => http://lmgtfy.com/?q=unix+find+xargs+awk
–P
Great Post and comment! Thanks for sharing your awknowledge!
Hi to all!
I have a question on using awk in bash scripts.
I am completely confused.
I have a file
voc.txt:
book
help
and a script generate1.sh:
voc=$1
awk ‘{print $1}’ $voc
When I run
>./generate1.sh voc.txt
I get
book
help
as expected.
But when I run the script
generate.sh:
voc=$1
awk ‘{print ” word” NR ” = (any * ‘” $1 “‘ space @increment” NR “) * ;”}’ $voc
I get
word1 = (any * voc1.txt space @increment1) * ;
word2 = (any * voc1.txt space @increment2) * ;
If I put a backslash before $, I get
word1 = (any * ‘$1’ space @increment1) * ;
word2 = (any * ‘$1’ space @increment2) * ;
How can I get
word1 = (any * ‘book’ space @increment1) * ;
word2 = (any * ‘help’ space @increment2) * ;
?
Very nice site with good useful info. Thank you.
Hi Can some one help me put together bash script that runs coulmns
1 2 3 4 5 6
and not 1
2
3
4
5
6
with column headers column1 = tj/art
column2=tj/art
column3=tj
column4=art
column5=tj
column6=art
and then redirect to a file >>
Hi All,
I have a shell script program which works in Unix platform and able to load data in staging and interface tables, but when it migrated to other instance of linux platform, its not working and not able to load data.
Is there any change to do with awk and sed commands in Linux ?
Please help in this regard…
Please find is shell script program
I have a file with the content as
/a/b/sometext1
/a/b/c/sometex2
/a/b/d
I want to separate out /a/b fields from all the lines.
how can I use awk command.
$ cat file
/a/b/sometext1
/a/b/c/sometex2
/a/b/d
$ awk -F/ ‘{print “/”$2″/”$3}’ file
/a/b
/a/b
/a/b
$ awk -F/ ‘{print $2,$3}’ file
a b
a b
a b
$ cat file
/a/b/sometext1
/a/b/c/sometex2
/a/b/d
$ awk -F/ ‘{print “/”$2″/”$3}’ file
/a/b
/a/b
/a/b
$ awk -F/ ‘{print $2,$3}’ file
a b
a b
a b
can anyone help me with this
.
Write a bash shell script that prints out only the even numbered UID’s from /etc/passwd.
1) Use a for loop with awk to extract the third field in /etc/passwd to a variable x.
2) Set a variable xmod2=$(($x%2))
3) Use an if condition to test if xmod2 is 0.
4) If it is zero, then the UID is even, so echo $x.
Hi ,
Can anyone tell me ? I want to take input from different file into script ,it should take feild wise,my input file has different fields separated by ; ex: madhuri;beautiful;girl;innocent; like this I have input file where name it should take first field,at second input second word,how can I write for this can you explain me pls .
Madhuri,
Please try this-
cat filename.dat | cut -f1 -d’;’