Linux/UNIX: Bash Read a File Line By Line

Syntax: Read file line by line on a Bash Unix & Linux shell:

1. The syntax is as follows for bash, ksh, zsh, and all other shells to read a file line by line
2. while read -r line; do COMMAND; done
3. The -r option passed to read command prevents backslash escapes from being interpreted.
4. Add IFS= option before read command to prevent leading/trailing whitespace from being trimmed -
5. while IFS= read -r line; do COMMAND_on $line; done

How to Read a File Line By Line in Bash

Here is more human readable syntax for you:

#!/bin/bash
input="/path/to/txt/file"
while IFS= read -r line
do
  echo "$line"
done < "$input"

The input file ($input) is the name of the file you need use by the read command. The read command reads the file line by line, assigning each line to the $line bash shell variable. Once all lines are read from the file the bash while loop will stop. The internal field separator (IFS) is set to the empty string to preserve whitespace issues. This is a fail-safe feature.

How to use command/process substitution to read a file line by line

Process or command substitution means nothing more but to run a shell command and store its output to a variable or pass it to another command. The syntax is:

while IFS= read -r line
do
   ## take some action on $line
  echo "$line"
done < < (command)
 
while IFS= read -r line
do
   ## take some action on $line
  echo "$line"
done < <(ps aux)

Using a here strings

Here strings is just like here documents:

while IFS= read -r line
do
  # take action on $line #
  echo "$line"
done <<< $(command)
 
while IFS= read -r line
do
  # take action on $line #
  echo "$line"
done <<< $(ps aux)
 
## shell script to purge urls from Cloudflare ##
t="10"
I="/home/vivek/.data/tags.deleted.410"
url=""
while IFS= read -r line
do
        url="$url $line"
done <<<"$(tail -${t} ${I})"
[ "$url" != "" ] && ~/bin/cloudflare.purge.urls.sh "$url"

How to file descriptor with read command

The syntax is simple:

input="/path/to/file"
while IFS= read -r -uN line
do
  # do something on $line
  echo "$line"
done N< $input

Here is a sample script:

while IFS= read -r -u13 line
do 
   echo "$line"
done 13<"${input}"

Examples that shows how to read file line by line

Here are some examples:

#!/bin/ksh
file="/home/vivek/data.txt"
while IFS= read line
do
        # display $line or do something with $line
	echo "$line"
done <"$file"

The same example using bash shell:

#!/bin/bash
file="/home/vivek/data.txt"
while IFS= read -r line
do
        # display $line or do somthing with $line
	printf '%s\n' "$line"
done <"$file"

You can also read field wise:

#!/bin/bash
file="/etc/passwd"
while IFS=: read -r f1 f2 f3 f4 f5 f6 f7
do
        # display fields using f1, f2,..,f7
        printf 'Username: %s, Shell: %s, Home Dir: %s\n' "$f1" "$f7" "$f6"
done <"$file"

Sample outputs:

Fig.01: Bash shell scripting- read file line by line demo outputs

Bash Scripting: Read text file line-by-line to create pdf files

My input file is as follows (faq.txt):

4|http://www.cyberciti.biz/faq/mysql-user-creation/|Mysql User Creation: Setting Up a New MySQL User Account
4096|http://www.cyberciti.biz/faq/ksh-korn-shell/|What is UNIX / Linux Korn Shell?
4101|http://www.cyberciti.biz/faq/what-is-posix-shell/|What Is POSIX Shell?
17267|http://www.cyberciti.biz/faq/linux-check-battery-status/|Linux: Check Battery Status Command
17245|http://www.cyberciti.biz/faq/restarting-ntp-service-on-linux/|Linux Restart NTPD Service Command
17183|http://www.cyberciti.biz/faq/ubuntu-linux-determine-your-ip-address/|Ubuntu Linux: Determine Your IP Address
17172|http://www.cyberciti.biz/faq/determine-ip-address-of-linux-server/|HowTo: Determine an IP Address My Linux Server
16510|http://www.cyberciti.biz/faq/unix-linux-restart-php-service-command/|Linux / Unix: Restart PHP Service Command
8292|http://www.cyberciti.biz/faq/mounting-harddisks-in-freebsd-with-mount-command/|FreeBSD: Mount Hard Drive / Disk Command
8190|http://www.cyberciti.biz/faq/rebooting-solaris-unix-server/|Reboot a Solaris UNIX System

My bash script:

#!/bin/bash
# Usage: Create pdf files from input (wrapper script)
# Author: Vivek Gite <Www.cyberciti.biz> under GPL v2.x+
#---------------------------------------------------------
 
#Input file
_db="/tmp/wordpress/faq.txt"
 
#Output location
o="/var/www/prviate/pdf/faq"
 
_writer="~/bin/py/pdfwriter.py"
 
# If file exists 
if [[ -f "$_db" ]]
then
    # read it
	while IFS='|' read -r pdfid pdfurl pdftitle
    do
    	local pdf="$o/$pdfid.pdf"
        echo "Creating $pdf file ..."
	#Genrate pdf file
        $_writer --quiet --footer-spacing 2 \
        --footer-left "nixCraft is GIT UL++++ W+++ C++++ M+ e+++ d-" \
        --footer-right "Page [page] of [toPage]" --footer-line \
        --footer-font-size 7 --print-media-type "$pdfurl" "$pdf"
    done <"$_db"
fi

Read from bash shell variable

Let us say you want a list of all installed php packages on a Debian or Ubuntu Linux, enter:

# My input source is the contents of a variable called $list #
list=$(dpkg --list php\* | awk '/ii/{print $2}')
printf '%s\n' "$list"

Sample outputs:

php-pear
php5-cli
php5-common
php5-fpm
php5-gd
php5-json
php5-memcache
php5-mysql
php5-readline
php5-suhosin-extension

You can now read from $list and install the package:

#!/bin/bash
# BASH can iterate over $list variable using a "here string" #
while IFS= read -r pkg
do
    printf 'Installing php package %s...\n' "$pkg"
    /usr/bin/apt-get -qq install $pkg
done <<< "$list"
printf '*** Do not forget to run php5enmod and restart the server (httpd or php5-fpm) ***\n'

Sample outputs:

Installing php package php-pear...
Installing php package php5-cli...
Installing php package php5-common...
Installing php package php5-fpm...
Installing php package php5-gd...
Installing php package php5-json...
Installing php package php5-memcache...
Installing php package php5-mysql...
Installing php package php5-readline...
Installing php package php5-suhosin-extension...
*** Do not forget to run php5enmod and restart the server (httpd or php5-fpm) ***

Conclusion

This page explained how to read file line by line in bash shell script.