BASH Shell: For Loop File Names With Spaces

by on December 9, 2008 · 59 comments· LAST UPDATED November 16, 2013

in , ,

BASH for loop works nicely under UNIX / Linux / Windows and OS X while working on set of files. However, if you try to process a for loop on file name with spaces in them you are going to have some problem. For loop uses $IFS variable to determine what the field separators are. By default $IFS is set to the space character. There are multiple solutions to this problem.

Set $IFS variable

Try it as follows:

#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for f in *
do
  echo "$f"
done
IFS=$SAVEIFS

OR

#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
# set me
FILES=/data/*
for f in $FILES
do
  echo "$f"
done
# restore $IFS
IFS=$SAVEIFS

More examples using $IFS and while loop

Now you know that if the field delimiters are not whitespace, you can set IFS. For example, while loop can be used to get all fields from /etc/passwd file:

....
while IFS=: read userName passWord userID groupID geCos homeDir userShell
do
      echo "$userName -> $homeDir"
done < /etc/passwd

Using old good find command to process file names

To process the output of find with a command, try as follows:

find . -print0 | while read -d $'\0' file
do
  echo -v "$file"
done

Try to copy files to /tmp with spaces in a filename using find command and shell pipes:

find . -print0 | while read -d $'\0' file; do cp -v "$file" /tmp; done

Processing filenames using an array

Sometimes you need read a file into an array as one array element per line. Following script will read file names into an array and you can process each file using for loop. This is useful for complex tasks:

#!/bin/bash
DIR="$1"
 
# failsafe - fall back to current directory
[ "$DIR" == "" ] && DIR="."
 
# save and change IFS 
OLDIFS=$IFS
IFS=$'\n'
 
# read all file name into an array
fileArray=($(find $DIR -type f))
 
# restore it 
IFS=$OLDIFS
 
# get length of an array
tLen=${#fileArray[@]}
 
# use for loop read all filenames
for (( i=0; i<${tLen}; i++ ));
do
  echo "${fileArray[$i]}"
done

Playing mp3s with spaces in file names

Place following code in your ~/.bashrc file:

mp3(){
	local o=$IFS
	IFS=$(echo -en "\n\b")
	/usr/bin/beep-media-player "$(cat  $@)" &
	IFS=o
}

Keep list of all mp3s in a text file such as follows (~/eng.mp3.txt):

/nas/english/Adriano Celentano - Susanna.mp3
/nas/english/Nick Cave & Kylie Minogue - Where The Wild Roses Grow.mp3
/nas/english/Roberta Flack - Kiling Me Softly With This Song.mp3
/nas/english/The Beatles - Girl.mp3
/nas/english/John Lennon - Stand By Me.mp3
/nas/english/The Seatbelts, Cowboy Bebop - 01-Tank.mp3

To play just type:
$ mp3 eng.mp3.txt

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 59 comments… read them below or add one }

1 Sergio December 9, 2008 at 9:44 pm

Hi, I don’t fully understand how this…

BASH for loop works nicely under UNIX / Linux / Windows and OS X

works nicely under Windows…

Oh! btw, thank you.

Reply

2 nixCraft December 9, 2008 at 10:00 pm

Install cygwin from http://www.cygwin.com/

Install bash and your are done!

Reply

3 Jeff Schroeder December 10, 2008 at 4:02 am

For the shell equivalent of python’s splitlines, I always did it like this:

oldifs="$IFS"
IFS="
"
for line in $(< line-with-spaces-between-fields.txt); do
   ...
done
IFS="$oldifs"

or while read ... instead of for when the list exceeds $(getconf ARG_MAX)

Your way works too, but is more typing

Reply

4 Amr Hamdy December 10, 2008 at 4:06 am

Thanks really so much,
It’s so useful and I think it’d save me a lot of time and help me do better shell coding..

Reply

5 Benjamin Schmidt December 10, 2008 at 8:03 am

This is a great blog!

Instead of using following:

find . -print0 | while read -d $'' file
do
  echo -v "$file"
done

I just use:

find . | while read file
do
  echo "$file"
done

Never had problems with it. A filename can impossible contain newline-chars, or am I wrong?

Best regards,
Benjamin Schmidt

Reply

6 Matthias July 26, 2010 at 6:31 pm

File names can contain newline characters.

Gnome for example displays them correctly. Couldn’t manage to create such a file in Gnome. I tested creation in C and Python iirc.

Reply

7 Ernesto August 25, 2012 at 9:09 am

In bash you can create a file with a newline in the name using, e.g:

touch $’weird\nfile’

Reply

8 nixCraft December 10, 2008 at 8:59 am

@Jeff/Benjamin

Thanks for sharing your code and ideas.

Reply

9 chika.tambun December 10, 2008 at 12:17 pm

i’d love to play music with mplayer… through cli
let say that i have song/music folder with many directory inside, how i can play with mplayer with single command. Actually i have one bash script, but i got trouble when facing white space… mplayer wouldn’t play the file

[songRequest.sh]
for i in ` find -name “*mp3″`
do
mplayer $i
done

any help will be appreciated

Reply

10 nixCraft December 10, 2008 at 12:21 pm

Try

find .  -iname "*.mp3" -print0 | while read -d $'\0' file
do
  mplayer "$file"
done

Reply

11 Chris F.A. Johnson December 10, 2008 at 3:40 pm

> A filename can impossible contain newline-chars, or am I wrong?

A filename can contain newlines. It can contain any character except a slash (/) or an ASCII NUL (0×0).

Reply

12 Chris F.A. Johnson December 10, 2008 at 3:45 pm

# use for loop read all filenames
> for (( i=0; i do
> echo “${fileArray[$i]}”
> done

If all you want to do is print the contents of the array:

printf "%s\n" "${fileArray[@]}"

To loop through them, use:

for i in "${fileArray[@]}"
do
: ... whatever
done

Reply

13 chika.tambun December 11, 2008 at 1:24 pm

great… vivek.gite ^^

your script just work on one file… then mplayer killed immediately.
yeah just playing one file only(there are many left).

on playing progress i can’t get control of mplayer, even i can’t turn up/down volume, go to the next song, etc.

all i can do is Ctrl+C

btw thanks

Reply

14 nixCraft December 11, 2008 at 2:12 pm

I guess you can try
mplayer *.mp3

Or send all files to .playlist.txt
ls -1 > .playlist.txt
mplayer -playlist .playlist.txt

Reply

15 mhernandez December 11, 2008 at 3:30 pm

All this information is very useful, but I think that needing it is sympton: ¿shouldn’t you use another language instead?

Nice post, though

Reply

16 Chris F.A. Johnson December 11, 2008 at 9:08 pm

“All this information is very useful, but I think that needing it is sympton: ¿shouldn’t you use another language instead?”

Why? It’s straightforward shell scripting.

Reply

17 mhernandez December 11, 2008 at 9:46 pm

opendir(DIR, “.”) or die(“Error opening directory”);
my @readFiles = readdir(DIR);
foreach (@readFiles) {

Reply

18 Kyle Brandt December 16, 2008 at 7:27 pm

I don’t think there is a need to reset the IFS variable, all you really need to do to avoid problems with word splitting is to put the variables that appear in the loop body in double quotes (I find this to be a good default habit). Once in double quotes the shell will consider whatever the variable expands to as a single argument and pass it to the program as such.

For example:
for i in *.mp3; do rm “$i”; done

mhernandez: Its not a problem with the shell, its just how it works, the bash shell is based on word splitting.

Reply

19 Rob Leach July 12, 2010 at 1:40 pm

This doesn’t seem to work for me. I’m wondering what I’m doing wrong….

If I do this:

$ for file in $(ls | awk -F. ‘{print $1}’); do sox -S “${file}.m4a” “${file}.ogg”; done

it doesn’t work as I would have expected.

Reply

20 Dean Faulkner January 7, 2009 at 4:57 pm

You can reset the IFS with

unset IFS

this way, you do not need to save the state and then recall it from a variable.

Reply

21 Jim Eberle January 10, 2009 at 3:47 am

Kyle is correct. No need to mess w/ your IFS var.
Your 2nd example is problematic in a very subtle way

FILES=/data/*

does not set the variable ‘FILES’ to the names of all the files in /data, it sets it to ‘/data/*’. You can see this w/:

declare -p FILES

A very useful technique in all this, is to employ the ‘set’ builtin:


cd ..; set *.txt; cd ~-
for f; do
echo "../$f" "$f"
done

This lets you scoop up a file list (w/ nasty names and all), and then move somewhere else to process them.

Reply

22 joel February 10, 2009 at 5:30 am

i am doing a project in unix shell scripting.
so my project have messages to be displayed. my problem is that how to add a beep sound to the messages …can any help me to include that also with the coding??pls..its very urgent…i dont know the shell codings…its for my project

Reply

23 Reid February 19, 2009 at 5:17 pm

Chris F.A. Johnson has it right on the nose.

use a quoted @-array: <code“${fileArray[@]}”

A quoted array with an @ subscript expands to a list of quoted elements of that array.

No IFS manipulation required!

Reid

Reply

24 DoXuS July 21, 2009 at 11:51 am

Thanks man, my code is now only 40 lines long, not 136 like before xD^^
nice blog

Reply

25 fvw August 26, 2009 at 4:48 am

mp3(){
local o=$IFS
IFS=$(echo -en “\n\b”)
/usr/bin/beep-media-player “$(cat $@)” &
IFS=o
}
“$(cat $@)” if you use this I think the IFS down work …
I think you want $(cat “$@”)

Reply

26 Rich September 2, 2009 at 8:38 pm

Thank you! I found this post most useful.

Reply

27 gimi October 1, 2009 at 9:06 pm

Instead of telling echo to print a char for new line and (probably) bell and then *not* to print a new line, one can just use echo’s default new line output :)

IFS=`echo`

Thanks for your share!

Reply

28 cfajohnson October 7, 2009 at 9:45 am

Why “IFS=`echo`”? Command substitution is slow (and unnecessary).

IFS=’

IFS=$’\n’ ## bash

Reply

29 Brian November 13, 2009 at 5:47 am

Amazing! Your notation for creating an array from:
> fileArray=($(find $DIR -type f))
is stunning! Thanks for the excellent writeup.

Reply

30 xiaoyuhum November 26, 2009 at 7:07 am

I just use as following
with double quotes
[code]
for m in "`ls`"; do echo "$m" ; done
[/code]

Reply

31 cfajohnson December 2, 2009 at 10:36 pm

for m in "`ls`"; do echo "$m" ; done

Why?

The for command will only see one argument.

Try this:


n=1
for m in "`ls`"
do
echo "$n"
echo "$m"
n=$(( $n + 1 ))
done

You’ll see that it only goes through the loop once.

You don’t need a loop for that, and you don’t need ls:

printf "%s\n" *

Reply

32 Shreyas Kulkarni December 19, 2009 at 5:52 pm

excellent post. exactly what i was looking for. i had been facing this space-field-seperator problem for quite a sometime today. had tried many options to circumvent it, but in vain. will try this $IFS approach now.

thanks a lot for sharing.

shreyas

Reply

33 Chris Thiessen February 8, 2010 at 8:09 pm

My favorite solution, though it can theoretically overmatch:

find . | sed -e “s/ /?/g”
ls | sed -e “s/ /?/g”

Then you can do things like greps, for loops, whatever equally well:
grep hello $(ls | sed -e “s/ /?/g”)
for i in $(find . | sed -e “s/ /?/g) do; ls -l “$i”; done

Reply

34 cfajohnson February 9, 2010 at 8:07 am

“grep hello $(ls | sed -e “s/ /?/g”)”

You don’t need sed or ls; use filename expansion:

grep hello *

“for i in $(find . | sed -e “s/ /?/g) do; ls -l “$i”; done”

That will fail if any filenames contain whitespace.

It is also extremely inefficient to call ls in a loop.

Reply

35 piponazo March 11, 2010 at 3:01 pm

Great post! It has been very useful for me :D

Reply

36 gringo guy April 19, 2010 at 4:18 am

Hey, thanks for this! I never did much bash scripting and was trying to figure out how to parse an array from a bash RC file into a Perl variable. I started out writing a long parser hack, but trying to support array entries with spaces was a big headache. Then thought maybe bash should do the work instead, and your examples helped a lot. Here’s a one-liner bash script that’s run using the Perl `backtick` operator.

$path is the RC file, and $var is the array variable. BASH variables must be escaped, but PERL
variables must not, and new-line chars have to be added so bash is happy with the string.
First step is to source the RC file, and next get the length of the array.

$list = `. $path; L=\${#${var}[@]}\n for (( i=0; i<\${L}; i++ ));\n do\n echo -n \${${var}[\$i]}\,\n done`;
chop( $list ); # strip the trailing comma

foreach $entry (split(",", $list)) { # Prove that it works
print "entry: '$entry'\n";
}

Here's an example of the bash array that this will parse. Cool. Thanks again for your examples.

ARRAY_VAR=( .bashrc .vimrc "foo bar" .java
.mozilla
.Xdefaults "other file name" )

Reply

37 Kelvin Nicholson August 25, 2010 at 9:40 pm

Fixed my little script! Cheers for posting this – always nice to learn something new.

Reply

38 Dennis Gearon September 11, 2010 at 6:24 am

Trying to get the following to work. If I cut and paste the eventual contents of $directoryFiles EITHER in the bash script OR on the command line, it works perfectly. If I try to get it to do it directly in the script, it fails with:

tar –all the file names– : Cannot stat: No such file or directory

I”ve tried about every expansion or quoting that I can think of for the $direcotryFiles variable.

#!/bin/bash
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
directoryFiles=""
for file in $( find /home/gearond -maxdepth 1 -type f)
do
  directoryFiles="$directoryFiles\"$file\" "
  #echo $file
done
tar -cf /tmp/directoryFiles.tar.bz2  $directoryFiles
IFS=$SAVEIFS

Reply

39 nixCraft September 11, 2010 at 8:28 am

How about:

find /home/gearond -maxdepth 1 -type f -print0 | xargs -O -I file your-command-on file

In your case:

find . -maxdepth 1 -type f -print0 | xargs -0 -I {} tar -rjvf /tmp/directoryFiles.tar.bz2 {}
tar -tjvf /tmp/directoryFiles.tar.bz2

Reply

40 Paul Grevink February 8, 2011 at 12:08 pm

Thank you very much, this is very helpful.
I work with Linux for many year, but until today, I did not know this.

Reply

41 jcubic March 7, 2011 at 5:03 pm

What is the difference between

IFS=$(echo -en "\n\b")

and

IFS=''

Which do exactly the same (but don’t call sub shell)

Reply

42 Chris F.A. Johnson March 7, 2011 at 5:50 pm

They do not do the same thing.

The first assigns a newline and a backspace to IFS; the second assigns an empty string.

The first assignment can be done more efficiently with:

IFS=$'\n\b'

Reply

43 Stefan Mueller June 4, 2011 at 7:45 pm

Thanks to all contributors of this outstanding discussion!

Reply

44 8ohmh July 23, 2011 at 2:02 pm

Thanks for your tip. I was not thinking anymore about IFS. Here’s a tip to remove unwanted characters in filenames:

SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
FILES=$1/*
for f in $FILES
do
	#echo	"$f"
	FILENAME="${f//[\?%\+]/_}";
	mv -b --strip-trailing-slashes "$f" "$FILENAME"
done
IFS=$SAVEIFS

replaces ?, % and + with _ (Useful if you copy FAT32 files to linux and back)

Reply

45 Rud Holmgren August 14, 2011 at 6:07 am

For simple operations on multiple files you can use the -exec option on find:

find . -iname ‘*.avi’ -exec mv {} /Some/Where/Else/ \;

This command recursively finds all .avi files in the current directory and moves them to folder /Some/Where/Else. The curly brackets {} are substituted with file name. Consult the man page for more details.

Reply

46 zerocool September 20, 2011 at 12:59 am

array=($(mysql –host=192.168.1.2 –user=XXXX –password=XXXX -s -N -e ‘use xbmc_video; SELECT c22 FROM movie, files WHERE files.idFile=movie.idFile ORDER BY idMovie DESC LIMIT 10;’))

how can i get rid of spaces in this array

Reply

47 Chris F.A. Johnson September 22, 2011 at 5:10 pm
array=( "${array[@]// /}" )

Reply

48 Tom Thurman January 23, 2012 at 8:14 pm

Thanks for the very clear explanation!

Reply

49 amrutha April 24, 2012 at 2:02 pm

Hello there,

I tried the code for storing the filenames into an array variable with a few modifications:

#!/bin/bash
DIR=$HOME/HY_DUMP/
# failsafe - fall back to current directory
#[ "$DIR" == "" ] && DIR="."
# save and change IFS
OLDIFS=$IFS
IFS=$'\n\b'
cd $DIR
# read all file name into an array
fileArray = ( $(`ls -l`) )
# restore it
IFS=$OLDIFS
# get length of an array
tLen=${#fileArray[@]}
echo $tLen
# use for loop read all filenames
for (( i=0; i<${tLen}; i++ ));
do
  echo "${fileArray[$i]}"
done

But, i have been running into errors on this:

error:

script.sh: syntax error at line 12: `(' unexpected

I have no clue how to proceed with this problem. Could anyone tweek this problem?

Thanks.
-A

Reply

50 Chris F.A. Johnson June 18, 2012 at 6:30 pm
fileArray = ( $(`ls -l`) )

should be

fileArray=( * )

Reply

51 manolo August 28, 2012 at 5:28 pm

Hi there,
I used the above example script for concatenating multiple files which works fine for me if the directory structure contains no spaces.However now I am facing a dierctory structure that does have spaces. Setting the IFS doesn’t seem to help here.
The directory structure I am facing is such: ./CH 0000010240/A10/21MT/21mt#001.csv and I want to concatenate all .csv files I find. Here is my current version of the script which I am running on OS X:

#!/bin/bash
DIR="$1"
# failsafe - fall back to current directory
[ "$DIR" == "" ] && DIR="."
# save and change IFS
OLDIFS=$IFS
IFS=$'\n'
# read all file name into an array
fileArray=($(find $DIR -type f))
# restore it
IFS=$OLDIFS
# get length of an array
tLen=${#fileArray[@]}
# use for loop read all filenames
for (( i=0; i> all.csv
done

Thanks for any hints.
Cheers,M

Reply

52 Chris F.A. Johnson September 3, 2012 at 4:49 am
> #!/bin/bash
> DIR="$1"
> # failsafe - fall back to current directory
> [ "$DIR" == "" ] && DIR="."

DIR=${DIR:-.}

> # save and change IFS
> OLDIFS=$IFS
> IFS=$'\n'
> # read all file name into an array
> fileArray=($(find $DIR -type f))
> # restore it
> IFS=$OLDIFS
> # get length of an array
> tLen=${#fileArray[@]}
> # use for loop read all filenames
> for (( i=0; i> all.csv

for file in “${filearray[@]}”
do
: do something with “$file” (in quotes)

> done

Reply

53 Ben September 23, 2012 at 8:12 am

Nixcrift FTW again. Good job.

Reply

54 Tobias September 28, 2012 at 2:42 pm

Very helpful. Thanks

Reply

55 Massimo October 18, 2012 at 7:23 am

It’s very helpful. Thank you

Reply

56 Sapan May 13, 2013 at 7:04 am

This can be simpler

for f in “` `”

just the ” around the comand and that is it

Reply

57 Megabyte June 5, 2013 at 3:16 am

Can anyone explain to me what the -v’s, used with echo and cp, in the “Using old good find command to process file names” section above, are for/do?

Reply

58 Alex October 21, 2013 at 11:21 am

Thank you very much Author for sharing these knowledge with other people!

Reply

59 David A. Wheeler November 6, 2013 at 4:16 am

The filename handling examples are wrong in several ways. Filenames can include newlines, backslashes, backspaces, tabs, asterisks, and other weirdness, which this doesn’t handle.

# For example, wrong – doesn’t handle filenames with \n or \b
IFS=$(echo -en “\n\b”)
for f in *
do
echo “$f”
done

For more on how to do it correctly, see:
http://www.dwheeler.com/essays/filenames-in-shell.html

Reply

Leave a Comment

Tagged as: , , , , , , , , , ,

Previous post:

Next post: