FirstName LastName,DOB,SSN,Telephone,Status
How can I parse a CSV file in Bash?
A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. A CSV file stores tabular data in plain text format. Each line of the file is a data record. You can use while shell loop to read comma-separated cvs file. IFS variable will set cvs separated to , (comma). The read command will read each line and store data into each field. Let us see how to parse a CSV file in Bash running under Linux, macOS, *BSD or Unix-like operating systems.
Bash Read Comma Separated CSV File
The syntax is as follows phrase a CSV file named input.csv:
while IFS=, read -r field1 field2 do echo "$field and $field2" done < input.csv
How to parse a CSV file in Bash
Create a file called test.sh using a text editor such as vim command/nano command:
vim test.sh
Append the following code:
#!/bin/bash # Purpose: Read Comma Separated CSV File # Author: Vivek Gite under GPL v2.0+ # ------------------------------------------ INPUT=data.cvs OLDIFS=$IFS IFS=',' [ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; } while read flname dob ssn tel status do echo "Name : $flname" echo "DOB : $dob" echo "SSN : $ssn" echo "Telephone : $tel" echo "Status : $status" done < $INPUT IFS=$OLDIFS
Run the test.sh file shell script as follows by setting up a execute permissions:
chmod +x test.sh
./test.sh
bash test.sh
sh test.sh
Parse a CSV file in Bash
CSV application support
The CSV file format is supported by spreadsheets and database management systems, including LibreOffice Calc, and Apache OpenOffice Calc.
Importing and Exporting CSV Files
How to open/import a text CSV file in calc
One can read comma separated CSV file using GUI app too.
- Start calc
- Choose File > Open.
- Locate the CSV file that you want to open.
- If the file has a *.csv extension, select the file.
- Click Open.
- The Text Import dialog opens.
- Specify the options to divide the text in the file into columns.
- Click OK.
Many Linux and Unix command line utility programs such as cut, paste, join, sort, uniq, awk, sed can split files on a comma delimiter, and can therefore process simple CSV files. For example:
awk -F',' '{ print $1 " " $2 }'
Conclusion
You learned how to read and parse comma-separated (CSV) file under a Linux or Unix-like system using bash while loop and read command.
🐧 14 comments so far... add one ↓
Category | List of Unix and Linux commands |
---|---|
File Management | cat |
Firewall | Alpine Awall • CentOS 8 • OpenSUSE • RHEL 8 • Ubuntu 16.04 • Ubuntu 18.04 • Ubuntu 20.04 |
Network Utilities | dig • host • ip • nmap |
OpenVPN | CentOS 7 • CentOS 8 • Debian 10 • Debian 8/9 • Ubuntu 18.04 • Ubuntu 20.04 |
Package Manager | apk • apt |
Processes Management | bg • chroot • cron • disown • fg • jobs • killall • kill • pidof • pstree • pwdx • time |
Searching | grep • whereis • which |
User Information | groups • id • lastcomm • last • lid/libuser-lid • logname • members • users • whoami • who • w |
WireGuard VPN | Alpine • CentOS 8 • Debian 10 • Firewall • Ubuntu 20.04 |
script that worked for me
How to skip commented/blank lines in the CSV file?
i have used the same code to read my csv file but i cant read the last row of my csv file using while loop. please help.
I’m having the same issue. The last row of my csv isn’t being read. Did you find a solution roop?
I am having the same issue. will try to figure out ans post it.
you can get the last line when you access the variables again outside the loop. using the example discussed in the post:
—————————–
while read flname dob ssn tel status
…..
done < $INPUT
echo $flname # <—– this will be the last line of the CSV
I have two Csv files one is for daily report purposed and another is for daily data purposed.
masterreport.csv file format is
CODE,1-May-12,2-May-12,3-May-12,4-May-12,5-May-12,
205,0,0,0,0,0,
23070,0,0,0,0,0,
Interface error response CRBT fail,0,1,0,0,0,
C2S exception,0,1,2,0,2,
Interface Customer Recharge Not Allowed For Receiver,2240,2078,2050,2007,2363,
—–Many more up to 45 Rows and 32 column are there
and every day we have to update the corresponding date part from below mention file.
dailyreport.csv
C2S ERROR EXCEPTION TAKING TIME TILL VALIDATION , 624
CHNL_ERROR_SNDR_AMT_NOTBETWEEN_MINMAX , 56
Card Group Slab Suspended , 7096
like how many error’s are coming on that day we have to update the main report , which are not match daily report we put the value 0 on main report.
i didn’t found any logic how to do this , can any body help this
15 years working with csv files in bash and I didn’t know this method!
Thanks a lot.
wow it’s really helpful.
Thx.
This way to get fields into a CSV is easy to use.
Exactly what i needed.
Thanx
This was exactly what I needed! I’m using it to import account data from a Zimbra server backup.
IE account names are stored in the CSV and the script runs the import command with the $flname variable in the appropriate spots.
I do have a question, How does it know to hit the next line and not just read the first line every time? My assumption is that is what the $IFS & $OLDIFS variables do. But I’m not sure.
This method is only for regular simplest version of CSV. very often fields are in quotation marks and it contains comma. In that situation for row
content1,”content,number2″,content3
read c1 c2 c3 assign
c1=’content1′
c2='”content”
c3=’number2″
then this method is not as universal as it should be.
last problem, very often last row in csv file is not ended with new line. in that situation read has some problem with fetching last row.
summarizing. this is the simplest way for reading the simplest cvs formatting.
Awk solution on github: https://github.com/benalt613/csv