Bash For Loop Spaces

by on March 13, 2008 · 16 comments· LAST UPDATED February 4, 2014

in , ,

How do I use bash for loop with spaces in a file name on Unix or Linux operating system? The following code fails with errors:

#!/bin/bash
files=$(ls *.txt)
dest="/nas/server/dest"
for f in $files
do
cp "$f" $dest
done

How do I fix this problem on bash shell?

Tutorial details
DifficultyEasy (rss)
Root privilegesNo
RequirementsNone
Estimated completion time5m
For demonstration purpose create files as follows. cd to /tmp and use mkdir command to create a directory file called test:

cd /tmp/
mkdir test
cd test

Create a set of files:

echo "foo" > "This is a test.txt"
echo "bar" > "another       file     name   with  lots of   spaces   .txt"
date > "current date and time.txt"
ls -l /etc/*.conf > "My configuration files.lst"
echo "Eat in silence; work in silence." > quote.txt
echo "Eat in silence; work in silence." > quote.txt
echo "Pride is blinding" > "A Long File    Name   .      doc"
 

To list directory contents use ls command as follows:
$ ls -l
Sample outputs:

total 48
-rw-r--r--  1 vivek  wheel    18 Feb  4 16:01 A Long File    Name   .      doc
-rw-r--r--  1 vivek  wheel  1092 Feb  4 15:54 My configuration files.lst
-rw-r--r--  1 vivek  wheel     4 Feb  4 15:54 This is a test.txt
-rw-r--r--  1 vivek  wheel     4 Feb  4 15:54 another       file     name   with  lots of   spaces   .txt
-rw-r--r--  1 vivek  wheel    29 Feb  4 15:54 current date and time.txt
-rw-r--r--  1 vivek  wheel    33 Feb  4 15:58 quote.txt

Understanding problem

Try to read file name with spaces in a for or while loop using following syntax instead of ls command:

Syntax

for f in *
do
  echo "$f"
done

Sample outputs:

A Long File    Name   .      doc
My configuration files.lst
This is a test.txt
another       file     name   with  lots of   spaces   .txt
current date and time.txt
quote.txt

Let us try to copy files using a bash for loop to $dest directory:

#!/bin/bash
dest="/nas/path/to/dest"
################################################################
## Do not use ls command to read file names in shell for loop ##
################################################################
for f in *.txt
do
  # do something with  $f now #
  cp "$f" "$dest"
done
 

Wrap command line args $@ (positional parameters) in double quotes

You can pass command line args too. The following is a bad example:

#!/bin/bash
for f in $@
do
        echo "|$f|"
done

Run it as follows:
./script *.txt
Sample outputs:

|This|
|is|
|a|
|test.txt|
|another|
|file|
|name|
|with|
|lots|
|of|
|spaces|
|.txt|
|current|
|date|
|and|
|time.txt|
|quote.txt|

The following is correct way to deal with command line args in bash for loop:

#!/bin/bash
for f in "$@"
do
        echo "|$f|"
done

Run it as follows:
./script *.txt
Sample outputs:

|This is a test.txt|
|another       file     name   with  lots of   spaces   .txt|
|current date and time.txt|
|quote.txt|

while loop with spaces example

find . | while read -r file
do
  echo "$file"
done

Or better:

find . -type f -print0 | xargs -I {} -0 echo "|{}|"

OR

find . -type f -print0 | xargs -I {} -0 cp "{}" /path/to/dest/

for loop with spaces example using IFS

Warning: Avoid using $IFS variable and is considered as a bad practice, but presented here for historical reasons. See the discussion below for more information.

O=$IFS
IFS=$(echo -en "\n\b")
for f in *
do
  echo "$f"
done
IFS=$O

To process all files passed as command line args:

#!/bin/bash
O=$IFS
IFS=$(echo -en "\n\b")
for f in "$@"
do
  echo "File: $f"
done
IFS=$O
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 16 comments… read them below or add one }

1 TheBonsai March 26, 2009 at 10:22 am

Hi.

The first two examples give the wrong expression that globbing or the positional parameters aren’t “word-aware”.

The first example (globbing) works correctly without touching the internal field separators.

The second example (positional parameters) works correctly when you quote the positional parameter mass-expander using doublequotes:

for x in "$@"; do
...
done

The third example should use the read-switch for “raw read”, and it should use the default REPLY variable: The raw-reading prevents Bash from interpreting special characters like line continuation, using REPLY makes it reading the whole line, instead of trying to interpret words in it.

find . | while read -r; do
echo "$REPLY"
done

However, you will face the problem with the while-loop being run in a subshell: You can’t communicate back to the main shell, e.g. by setting variables. So try Process Substitution:

while read -r; do
echo "$REPLY"
done < <(find .)

References:
Mass-usage of positional parameters
The read builtin command
Introduction to expansions and substitutions

Regards,
Jan

Reply

2 Peter March 26, 2009 at 10:59 am

You should not manipulate IFS for such primitive jobs! Publishing this as a tutorial is not good.

Reply

3 Philippe Petrinko November 21, 2009 at 12:51 pm

Hi Vivek,
This subject is interesting, but false.

If you try this
1) create a file with lots of spaces :
touch “file with spaces (lots) in its name”

2) check if a simple [ for ] loop will process it:
# for f in *; do echo -e “:$f:”; done

this will print OK:
:”file with spaces (lots) in its name:

Therefore proving that there is no need to modify IFS variable,

(The Bonsai and Peter are right)

Keep up the good work, thanks for your site and interesting topics.

— Philippe

Reply

4 george February 4, 2010 at 1:09 pm

The technique with ‘read’ isn’t necessary. It depends on the source of the strings for ‘for’.

The problem:

$ touch “hello there”
$ touch hell_is_for_children

$ for i in `ls hell*`; do echo -e “:$i:” ; done
:hell_is_for_children:
:hello:
:there:

Two solutions:

Use read:

$ ls -1 hell* | while read FILENAME; do echo “:$FILENAME:” ; done
:hell_is_for_children:
:hello there:

Note that this will not work, because echo puts everything on the same line, even though it comes out of ls -1 on multiple lines:
$ echo `ls -1 hell*` | while read FILENAME; do echo “:$FILENAME:” ; done
:hell_is_for_children hello there:

Although for some reason this does work (note the double quotes):
$ echo “`ls -1 hell*`” | while read FILENAME; do echo “:$FILENAME:” ; done

Second solution:

You can’t set IFS to newline for some reason:

$ IFS=$(echo -ne ‘\n’)

$ echo -n :$IFS: | xxd
0000000: 3a3a ::

Unless its followed by something
$ IFS=$(echo -ne ‘\n ‘)
$ echo -n :$IFS: | xxd
0000000: 3a20 3a : :

Zeroing it out doesn’t work:
$ IFS=
$ for i in `ls -1 hell*`; do echo “:$i:” ; done
:hell_is_for_children
hello there:

Note there’s no trailing/leading colons in the above result.

You can get the \n in there if you trail it with another character. How
about 0x0d?

$ IFS=$(echo -en “\n015″)
$ for i in `ls -1 hell*` ; do echo “:$i:” ; done
:hell_is_for_children:
:hello there:

So that should do it. You’ll find another google result that sets IFS to $(echo -en “\n\b”) and I couldn’t figure out why \b for the life of me, until I went through all the above pain myself :)

Reply

5 Philippe Petrinko February 4, 2010 at 3:00 pm

To George,
I don’t get your point.

Let’s create 3 files

touch "hello there"
touch "I feel I am getting too much spaces"
touch hell_is_for_children

There is no problem selecting files beginning with “hell” using this simple line:

for f in hell*; do echo ":${f}:"; done

# which gives:
:hell_is_for_children:
:hello there:

My Unix Guru always told me: Keep It Short and Simple.

To Vivek: regarding “To process all files passed as command line args”:
(and please note the typo “ars” instead of “args”)

Create this script as “sample01.sh”

#!/bin/bash
for f
do
echo ":${f}:"
done

Then run sample01.sh script:

bash sample01.sh "this-one-has-no-spaces" "this one has many spaces" "Some Spaces Too"
#which gives:
:this-one-has-no-spaces:
:this one has many spaces:
:Some Spaces Too:

you can avoid “for f in $@” and stick to “for f”

Thanks for this interesting topic,
— Philippe

Reply

6 george February 5, 2010 at 4:39 am

Philippe – it depends on your SOURCE of filenames. Yes, a simple shell glob will work if you’re just trying to match a pattern:

for x in hell*

but my examples used backticks to demonstrate problems when *commands* are the source of filenames. eg.:

`cat filenames.txt`

In these cases, you may have to change IFS or use read. I used `ls -1` to make the examples easier to reproduce.

Reply

7 Philippe Petrinko February 5, 2010 at 10:03 am

To George:
“but my examples used backticks to demonstrate problems when *commands* are the source of filenames. ”
As the question of this topic is : “How do I use bash for loop with spaces in a file name?”

I am afraid I disagree. The point is using the output of a command in an appropriate way.
And there are ways which do not require modifying IFS variable.

Let’s try this:

touch "Hello, World"
touch "myfile with spaces"
touch somefile

Then you can use [ls -1] this way:


ls -1|while read f; do echo ":${f}:"; done
:Hello, World:
:myfile with spaces:
:somefile:

Next, using a file content.
Let’s create the list of file names

ls -1>list.txt

Again, no problem (and BTW we can skip using [cat] which involves useless overhead)

while read f; do echo ":${f}:"; done < list.txt
:Hello, World:
:myfile with spaces:
:somefile:

Last, but not least, [bash] has a builtin Variable which takes input of [read] function.
(always a good thing to remind !)

That is, one could simplier write

while read; do echo ":${REPLY}:"; done < list.txt

or

ls -1|while read; do echo ":${REPLY}:"; done

Nevertheless, I admit there may be some times when you need to alter IFS – but not in the cases you proposed.

— Philippe

Reply

8 george February 6, 2010 at 8:18 am

Well Philippe, we’re down to opinions, not facts.

1) I don’t think modifying IFS is that big of a deal – that’s what its there for.

2) I don’t think this is very readable:
while read; do echo ":${REPLY}:"; done < list.txt

To the eye, it looks like you’re redirecting list.txt to ‘done’. Definitely not clear that the receiver of STDIN is way back there in the front. The overhead for cat’ing is nothing compared to the worth of having the source of data on the left side where it belongs.

3) Using $REPLY violates many readability and maintainability standards. Its name doesn’t make any sense in the context (who exactly is replying?).

Reply

9 TheBonsai February 6, 2010 at 8:53 am

@george

at 2) it’s more correct to say you redirect to the ‘done’ than to the ‘read’. Both is wrong, of course, because you redirect to the compound command.

at 3) you need REPLY here or you have code that looses spaces

Reply

10 Philippe Petrinko February 6, 2010 at 3:24 pm

To George:
1) Is IFS modification a big deal ?

Yes, you just proved it.

Because of all defensive programming techniques I have learnt,
modifying IFS should kept to avoided unless necessary.

a) Modifying IFS it is useless in this case, it can be solved by basic commands. Keep it short & simple.

b) As your code shows it, it is not for beginners, IFS processing as many side-effects and is complex to handle.
So, it is more prone to programming errors. For instance:
– To forget to restore IFS value. Code may be long, and IFS restoration may be out of view.
– Using commands which rely on Environment variables leads to code that is much more context-dependant – and therefore less readable.

This is exactly what your experience shows, you said: “I couldn’t figure out why for the life of me, until I went through all the above pain myself”

2) Regarding code readability and being puzzled by right-hand redirection:

Well, it’s a fact, and not an opinion, that input redirection with “<" is just basic shell functionality.

Nevertheless, no problemo. Write:

cat list.txt | while read; do echo ":${REPLY}:"; done
or
cat list.txt | while read
do
        echo ":${REPLY}:";
done

3) Using $REPLY …
I did not choose this name – Bash programmers did. Nevertheless, you don’t like it, drop it:

cat list.txt | while read fname
do
        echo "${fname}";
done

that’s it !
See, no IFS modification needed, readable, from left to right.

I just prefer the more efficient (using just shell internals functions) …

while read fname
do
        echo "${fname}";
done < list.txt

…because this is the use shell input redirection was designed for.

Cheers,
— Philippe

Reply

11 TheBonsai February 6, 2010 at 3:33 pm

@Philipe

Same applies to your read. If you don’t use the automatic REPLY, read looses data (as it doesn’t read the line as line, but as set of words).

Reply

12 Philippe Petrinko February 6, 2010 at 3:51 pm

To the Bonsai:
I am not sure of what you really mean, I assume you say this code does not work.
The request is to print file names that may contains spaces – this works.
I have checked this code before. Let’s check it again in my gnome-terminal:

while read fname; do echo ":${fname}:"; done < list.txt
:Hello, World:
:list.txt:
:myfile with big spaces:
:myfile with spaces:
:somefile:

It works. I don’t know if you tried it, but it just works on my Debian Lenny/Bash.
Or may I have misunderstood your comment?
Would you show me where this code does not fit to the purpose ?

Reply

13 TheBonsai February 6, 2010 at 5:13 pm

I mean this:

$ sed 's/^/>>>/;s/$/<<>> leading and trailing spaces <<<

$ while read fname; do echo ":${fname}:"; done < list.txt
:leading and trailing spaces:

Reply

14 TheBonsai February 6, 2010 at 5:14 pm

Sorry, seems I broke the HTML code parser there :/

Reply

15 Philippe Petrinko February 6, 2010 at 6:49 pm

@Bonsai
Yeah, right :-/

$REPLY was designed to take raw input (I mean the whole line without word splitting).

And then yes, in this case, if we do not use $REPLY,
one of the simpliest way to have [read] include leading and trailing spaces (or any FS caracter)
is to modify IFS variable.

Reply

16 chris February 4, 2014 at 8:47 am

nixcraft is now seen as untrustworthy by me

Reply

Leave a Comment

Tagged as: , , , , , , , , ,

Previous Faq:

Next Faq: