UNIX: Read a File Line By Line

by on January 26, 2011 · 15 comments· LAST UPDATED January 26, 2011

in

How do I read a file line by line under UNIX using KSH or BASH shell?

You can use the following syntax using ksh:

#!/bin/ksh
file="/home/vivek/data.txt"
while read line
do
        # display $line or do somthing with $line
	echo "$line"
done <"$file"

Same code should work with bash too:

#!/bin/bash
file="/home/vivek/data.txt"
while IFS= read -r line
do
        # display $line or do somthing with $line
	echo "$line"
done <"$file"

You can also read field wise:

#!/bin/bash
file="/etc/passwd"
while IFS=: read -r f1 f2 f3 f4 f5 f6 f7
do
        # display fields using f1, f2,..,f7
        echo "Username: $f1, Shell: $f7, Home Dir: $f6"
done <"$file"
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 15 comments… read them below or add one }

1 Chris F.A. Johnson February 8, 2011 at 6:42 pm

The bash example is correct for all shells. The ksh example is not correct; it will strip leading and trailing whitespace.

To demonstrate, use:

echo ":$line:"

on a file with leading and/or trailing whitespace on one or more .

Reply

2 Anthony Thyssen September 14, 2011 at 11:35 pm

Reading a file line by line with a single loop is fine in 90% of the cases.

But if things get more complicated then you may be better off opening the file as a file descriptor.

Situations such as:

* the read of the file is only a minor aspect and not the main task, but you want to avoid opening an closing the file multiple times. For example preserving where you have read up to or avoiding file closure (network, named pipes, co-processing)

* the file being read requires different methods of parsing, that is better done in simpler but different loops. Reading emails (header and body) is an example of this, but so is reading files with separate operational and data blocks.

* you are reading from multiple files at the same time, and switching between different files, in a pseudo-random way. That is you never quite know whch file you will need to read from next.

Reply

3 princess anu April 4, 2012 at 7:01 pm

I have a scenario like this. I have a file which has servername, port and ipaddress
svr1
port1
ip1
svr2
port2
ip2

I tried couple of ways reading these records and assigning them a field and print the output to a file like below but nothing is helpful

servername port ipaddress
svr1 port1 ip1
scr2 port2 ip2
and so on

can someone please help me.

Thanks

Reply

4 princess anu April 5, 2012 at 3:20 pm

I figured it out. May be not professional but it is working for me.

#!/bin/sh
#!/bin/bash
num=1
echo -n "server name    "
echo "port number"
echo "-----------------------------------------"
while read line
do
echo -n "$line        "
i=$(($num % 2 ))
if [ $i -eq 0 ]
then
echo "" #### print the new line ###
fi
num=`expr $num + 1`
done< serverlist.txt

Thanks

Reply

5 Robert Edward Johnson June 10, 2012 at 2:30 am

HOW DO YOU SPLIT INPUT INTO FIELDS FOR PROCESSING?

I have a question concerning how you’d actually process individual fields of input.
For example, let’s say you’re doing the usual

while read LINE; do
process the statements
done firstbyteinq and looking at firstbyteinq with a read, but 1). the cut doesn’t seem to be doing it just to the record being read, it does it to ALL the records all at one time; 2). the read of the firstbyteinq then advances the other file one record!!!

It seems to me there ought to be some way to “declare” the positions from a to b as fields and THEN use the while read, but I’m not sure how to do that.
I’ll try anything. Thanks VERY much in advance.

Reply

6 Chris F.A. Johnson June 10, 2012 at 4:16 pm

princess anu, there’s no need to use arithmetic; just read the file in groups of three lines:

while :
do
  read server
  read port
  read ipaddr
  : do whatever
done < "$file"

Reply

7 Chris F.A. Johnson June 10, 2012 at 4:28 pm

A correction; you need to check that the read succeeded; checking the first should be enough:

while :
do
  read server || break
  read port
  read ipaddr
  : do whatever
done < "$file"

Reply

8 MAYANK September 6, 2012 at 8:19 am

I have scenario :

One file which contain values like:
ABS
SVS
SGS
SGS

another file is template which has to use values from 1st file (One value at a time, line by line )

ARGS=ABS >> save in one file

New file:
ARGS=SVS >> save in another file

Can you please help me to achive this

Reply

9 bhupesh May 14, 2014 at 6:30 am

u can use paste command if the fields are same …..and then output the records to the other file and print another.

Reply

10 bhupesh May 14, 2014 at 6:56 am

rm 3 4 5
first=1
seco=2
cat 1 | while read l
do
echo “$l =” >> 3
done
paste 3 2 | while read line
do
echo “$line” >> 4
done
awk ‘{print $1,$2,$3}’ 4 > 5

now u can move the records line by line to another file

Reply

11 Name July 2, 2013 at 1:35 pm

echo “$line” # <— will fail if "$line" is "-n", it won't print anything at all, even an empty line.

You should use printf builtin to work around this behavior:
printf "%s\n" "$line"

Reply

12 Chris F.A. Johnson July 3, 2013 at 11:40 am

Some shells shells will output -n; that is what POSIX requires.

Reply

13 Steve January 28, 2014 at 12:55 am

My while/read is having an issue with the data not being printed for the last line read.
The info is read and summed but is not being printed to the report.
The script is reading a file with tab delimited fields, sorting on a field then reading each line while assigning the fields to var names.
If statements determine whether the numbers are summed or if a new category is found with the previous categories data being written to the report. All categories except the last are complete in the report. The last category is left out.
I added trouble shooting echo statements to check the workflow and they showed that the last category is being read and summed but it doesnt get printed.
It’s as if the read gets to the last line, does the work and then doesnt know what to do with the output.
The sorted data looks something like:

247   an-IP   Name1
37   another-IP   Name1
52    more-IPs   Name2
1562   etc   Name2
29   etc   name3
4   etc   name3
The report has:
284 Name1
1591 Name2

Code with troubleshooting echos and most comments deleted:

#control vars
GT=0
name-Name
cat  | sort -k3 | while read -r Count IP Name
do
# Handle the first line
   if [ "Name" != "$name" -a $GT -eq 0 ]
   then
      name=$Name
      GT=$Count
# The read name is different than the last line (new category)
   elif [ "$Name != $name" -a $GT -gt 0 ]
   then
# Write previous name and count to the report
      echo " ${name} - ${GT}" >>
# Assign new categories values to the vars
      GT=$Count
      name=$Name
# New line but same category found
   else [ "$Name" == "$name"  ]
# Increment counts of the category
      GT=`expr $GT + $Count`
   fi
done

Reply

14 name April 3, 2014 at 9:45 am

The read name is different than the last line (new category)

   elif [ "$Name != $name" -a $GT -gt 0 ]
   then
# Write previous name and count to the report
      echo " ${name} - ${GT}" >>
# Assign new categories values to the vars
      GT=$Count
      name=$Name
# New line but same category found
   else [ "$Name" == "$name"  ]
# Increment counts of the category
      GT=`expr $GT + $Count`
   fi
done
REPLY

Reply

15 Chris F.A. Johnson May 14, 2014 at 11:25 am

UUOC

cat | sort -k3 | while read -r Count IP Name

is the same as

sort -k3 | while read -r Count IP Name

Reply

Leave a Comment

Tagged as: , , , , , , , ,

Previous Faq:

Next Faq: