You can use while..do..done bash loop to read file line by line on a Linux, OSX, *BSD, or Unix-like system.
Syntax: Read file line by line on a Bash Unix & Linux shell:
Tutorial details | |
---|---|
Difficulty | Easy (rss) |
Root privileges | No |
Requirements | Bash/ksh/zsh/tcsh |
Time | 10m |
- The syntax is as follows for bash, ksh, zsh, and all other shells to read a file line by line
- while read -r line; do COMMAND; done
- The -r option passed to read command prevents backslash escapes from being interpreted.
- Add IFS= option before read command to prevent leading/trailing whitespace from being trimmed -
- while IFS= read -r line; do COMMAND_on $line; done
How to Read a File Line By Line in Bash
Here is more human readable syntax for you:
#!/bin/bash input="/path/to/txt/file" while IFS= read -r line do echo "$line" done < "$input"
The input file ($input) is the name of the file you need use by the read command. The read command reads the file line by line, assigning each line to the $line bash shell variable. Once all lines are read from the file the bash while loop will stop. The internal field separator (IFS) is set to the empty string to preserve whitespace issues. This is a fail-safe feature.
How to use command/process substitution to read a file line by line
Process or command substitution means nothing more but to run a shell command and store its output to a variable or pass it to another command. The syntax is:
while IFS= read -r line do ## take some action on $line echo "$line" done < < (command) while IFS= read -r line do ## take some action on $line echo "$line" done < <(ps aux)
Using a here strings
Here strings is just like here documents:
while IFS= read -r line do # take action on $line # echo "$line" done <<< $(command) while IFS= read -r line do # take action on $line # echo "$line" done <<< $(ps aux) ## shell script to purge urls from Cloudflare ## t="10" I="/home/vivek/.data/tags.deleted.410" url="" while IFS= read -r line do url="$url $line" done <<<"$(tail -${t} ${I})" [ "$url" != "" ] && ~/bin/cloudflare.purge.urls.sh "$url"
How to file descriptor with read command
The syntax is simple:
input="/path/to/file" while IFS= read -r -uN line do # do something on $line echo "$line" done N< $input
Here is a sample script:
while IFS= read -r -u13 line do echo "$line" done 13<"${input}"
Examples that shows how to read file line by line
Here are some examples:
#!/bin/ksh file="/home/vivek/data.txt" while IFS= read line do # display $line or do something with $line echo "$line" done <"$file"
The same example using bash shell:
#!/bin/bash file="/home/vivek/data.txt" while IFS= read -r line do # display $line or do somthing with $line printf '%s\n' "$line" done <"$file"
You can also read field wise:
#!/bin/bash file="/etc/passwd" while IFS=: read -r f1 f2 f3 f4 f5 f6 f7 do # display fields using f1, f2,..,f7 printf 'Username: %s, Shell: %s, Home Dir: %s\n' "$f1" "$f7" "$f6" done <"$file"
Sample outputs:
Fig.01: Bash shell scripting- read file line by line demo outputs
Bash Scripting: Read text file line-by-line to create pdf files
My input file is as follows (faq.txt):
4|http://www.cyberciti.biz/faq/mysql-user-creation/|Mysql User Creation: Setting Up a New MySQL User Account 4096|http://www.cyberciti.biz/faq/ksh-korn-shell/|What is UNIX / Linux Korn Shell? 4101|http://www.cyberciti.biz/faq/what-is-posix-shell/|What Is POSIX Shell? 17267|http://www.cyberciti.biz/faq/linux-check-battery-status/|Linux: Check Battery Status Command 17245|http://www.cyberciti.biz/faq/restarting-ntp-service-on-linux/|Linux Restart NTPD Service Command 17183|http://www.cyberciti.biz/faq/ubuntu-linux-determine-your-ip-address/|Ubuntu Linux: Determine Your IP Address 17172|http://www.cyberciti.biz/faq/determine-ip-address-of-linux-server/|HowTo: Determine an IP Address My Linux Server 16510|http://www.cyberciti.biz/faq/unix-linux-restart-php-service-command/|Linux / Unix: Restart PHP Service Command 8292|http://www.cyberciti.biz/faq/mounting-harddisks-in-freebsd-with-mount-command/|FreeBSD: Mount Hard Drive / Disk Command 8190|http://www.cyberciti.biz/faq/rebooting-solaris-unix-server/|Reboot a Solaris UNIX System
My bash script:
#!/bin/bash # Usage: Create pdf files from input (wrapper script) # Author: Vivek Gite <Www.cyberciti.biz> under GPL v2.x+ #--------------------------------------------------------- #Input file _db="/tmp/wordpress/faq.txt" #Output location o="/var/www/prviate/pdf/faq" _writer="~/bin/py/pdfwriter.py" # If file exists if [[ -f "$_db" ]] then # read it while IFS='|' read -r pdfid pdfurl pdftitle do local pdf="$o/$pdfid.pdf" echo "Creating $pdf file ..." #Genrate pdf file $_writer --quiet --footer-spacing 2 \ --footer-left "nixCraft is GIT UL++++ W+++ C++++ M+ e+++ d-" \ --footer-right "Page [page] of [toPage]" --footer-line \ --footer-font-size 7 --print-media-type "$pdfurl" "$pdf" done <"$_db" fi
Read from bash shell variable
Let us say you want a list of all installed php packages on a Debian or Ubuntu Linux, enter:
# My input source is the contents of a variable called $list # list=$(dpkg --list php\* | awk '/ii/{print $2}') printf '%s\n' "$list"
Sample outputs:
php-pear php5-cli php5-common php5-fpm php5-gd php5-json php5-memcache php5-mysql php5-readline php5-suhosin-extension
You can now read from $list and install the package:
#!/bin/bash # BASH can iterate over $list variable using a "here string" # while IFS= read -r pkg do printf 'Installing php package %s...\n' "$pkg" /usr/bin/apt-get -qq install $pkg done <<< "$list" printf '*** Do not forget to run php5enmod and restart the server (httpd or php5-fpm) ***\n'
Sample outputs:
Installing php package php-pear... Installing php package php5-cli... Installing php package php5-common... Installing php package php5-fpm... Installing php package php5-gd... Installing php package php5-json... Installing php package php5-memcache... Installing php package php5-mysql... Installing php package php5-readline... Installing php package php5-suhosin-extension... *** Do not forget to run php5enmod and restart the server (httpd or php5-fpm) ***
Conclusion
This page explained how to read file line by line in bash shell script.
🐧 24 comments so far... add one ↓
Category | List of Unix and Linux commands |
---|---|
File Management | cat |
Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
Network Utilities | dig • host • ip • nmap |
OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
Package Manager | apk • apt |
Processes Management | bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time |
Searching | grep • whereis • which |
User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
The bash example is correct for all shells. The ksh example is not correct; it will strip leading and trailing whitespace.
To demonstrate, use:
on a file with leading and/or trailing whitespace on one or more .
The BASH password file parsing example does not work for my centos 7 box: all the colons are stripped from each line and the whole line without colons is stored in f1
my output reads:
Username: root/root/bin/bash, Home directory: , Shell:
Reading a file line by line with a single loop is fine in 90% of the cases.
But if things get more complicated then you may be better off opening the file as a file descriptor.
Situations such as:
* the read of the file is only a minor aspect and not the main task, but you want to avoid opening an closing the file multiple times. For example preserving where you have read up to or avoiding file closure (network, named pipes, co-processing)
* the file being read requires different methods of parsing, that is better done in simpler but different loops. Reading emails (header and body) is an example of this, but so is reading files with separate operational and data blocks.
* you are reading from multiple files at the same time, and switching between different files, in a pseudo-random way. That is you never quite know whch file you will need to read from next.
I have a scenario like this. I have a file which has servername, port and ipaddress
svr1
port1
ip1
svr2
port2
ip2
…
…
I tried couple of ways reading these records and assigning them a field and print the output to a file like below but nothing is helpful
servername port ipaddress
svr1 port1 ip1
scr2 port2 ip2
and so on
can someone please help me.
Thanks
I figured it out. May be not professional but it is working for me.
Thanks
if the file is 100% predictable in each host,ip,port coming in groups of 3 you can use “paste”
cat list.txt | paste – – –
HOW DO YOU SPLIT INPUT INTO FIELDS FOR PROCESSING?
I have a question concerning how you’d actually process individual fields of input.
For example, let’s say you’re doing the usual
while read LINE; do
process the statements
done firstbyteinq and looking at firstbyteinq with a read, but 1). the cut doesn’t seem to be doing it just to the record being read, it does it to ALL the records all at one time; 2). the read of the firstbyteinq then advances the other file one record!!!
It seems to me there ought to be some way to “declare” the positions from a to b as fields and THEN use the while read, but I’m not sure how to do that.
I’ll try anything. Thanks VERY much in advance.
princess anu, there’s no need to use arithmetic; just read the file in groups of three lines:
A correction; you need to check that the read succeeded; checking the first should be enough:
I have scenario :
One file which contain values like:
ABS
SVS
SGS
SGS
another file is template which has to use values from 1st file (One value at a time, line by line )
ARGS=ABS >> save in one file
New file:
ARGS=SVS >> save in another file
Can you please help me to achive this
u can use paste command if the fields are same …..and then output the records to the other file and print another.
rm 3 4 5
first=1
seco=2
cat 1 | while read l
do
echo “$l =” >> 3
done
paste 3 2 | while read line
do
echo “$line” >> 4
done
awk ‘{print $1,$2,$3}’ 4 > 5
now u can move the records line by line to another file
echo “$line” # <— will fail if "$line" is "-n", it won't print anything at all, even an empty line.
You should use printf builtin to work around this behavior:
printf "%s\n" "$line"
Some shells shells will output -n; that is what POSIX requires.
My while/read is having an issue with the data not being printed for the last line read.
The info is read and summed but is not being printed to the report.
The script is reading a file with tab delimited fields, sorting on a field then reading each line while assigning the fields to var names.
If statements determine whether the numbers are summed or if a new category is found with the previous categories data being written to the report. All categories except the last are complete in the report. The last category is left out.
I added trouble shooting echo statements to check the workflow and they showed that the last category is being read and summed but it doesnt get printed.
It’s as if the read gets to the last line, does the work and then doesnt know what to do with the output.
The sorted data looks something like:
Code with troubleshooting echos and most comments deleted:
The read name is different than the last line (new category)
UUOC
cat | sort -k3 | while read -r Count IP Name
is the same as
sort -k3 | while read -r Count IP Name
It is also possible to store the elements in a bash array using the -a option
The following example reads some comma separated values and print them in reverse order:
input.txt
Hi guys. I’m just getting started with scripting and I am needing some help reading a file into the script. This shell script below works and does what it needs to do but I have to manually input the file name for “read list”. How do I modify this script to automatically read the file. Inside the file is just a list of aix server names.
#!/usr/bin/sh
echo ” Please input the master list you know to use”
read list
for i in `cat $list`
do
ssh $i hostname
ssh $i df
done
Hi all,
I need to write a script to look for user in /etc/passwd file. The program takes user input from command line and check to see if the user that the person entered exists. If user doesn’t exist, the script sends error message. I use grep to print the user which exist. But not sure what command to use and look and if the user doesn’t exist, send error message?
Thanks,
Hi stone,
me trying to get the list of server have /mysqlshare file system in it but i am unable to get it..
for now trying get only 2 server details from txt file ie(text.txt) which have
server1_name
server2_name
when i try this script its only reading 1st server details and getting exit from script
Can someone help me out..
Thanks man!
can anyone help me to include the line number while read line by line ,
# cat file.txt
mike
vero
james
my desire output is
1. mike
2. vero
3. james
thank you before
cat file.txt | nl
I see