Q. How do I run shell loop over set of files stored in a current directory or specified directory?
A. You can use for loop easily over a set of shell file under bash or any other UNIX shell using wild card character.
Using a shell variable and for loop
You can use shell variable to store all file names. For example, store all *.c file in a variable called FILES:
$ FILES="*.c"
Now to loop through all files, enter:
$ for f in "$FILES"; do echo "Processing $f file.."; done
Sample shell script to loop through all files
#!/bin/bash FILES="*" for f in "$FILES" do echo "Processing $f file..." # take action on each file. $f store current file name cat $f done
Featured Articles:
- 20 Linux System Monitoring Tools Every SysAdmin Should Know
- My 10 UNIX Command Line Mistakes
- 10 Greatest Open Source Software Of 2009
- Top 5 Email Client For Linux, Mac OS X, and Windows Users
- Top 20 OpenSSH Server Best Security Practices
- Top 10 Open Source Web-Based Project Management Software
- Top 5 Linux Video Editor Software
Want to read Linux tips and tricks, but don't have time to check our blog everyday? Subscribe to our daily email newsletter to make sure you don't miss a single tip/tricks. Subscribe to our weekly newsletter here!
- Email FAQ to a friend
- Download PDF version
- Printable version
- Comment RSS feed
- Last Updated: 05/11/08
{ 19 comments… read them below or add one }
The scripts are wrong, use `$FILES` (without quotes) in the loop instead of `”$FILES”`.
DAY,
I don’t think so it is wrong. Do you have any problem running script?
Yes, the script does not work for me. Both the one directly typed in the cmd line and the one from a script file. They output of the first one is:
Processing *.c file..
The output of the second is:
Processing * file…
In fact, I just start learn the shell script. Here is another one which can do the same job:
#!/bin/sh
for f in `ls`
do
echo “Processing $f file …”
done
Hmm…not sure why you wouldn’t use
for file in *
or
for file in *.c
DAY,
Hmm, it should work (I hope you have *.c files in current directory) or try as suggested by brock.
Thanks, vivek and Brock. Replacing “$FILES” with * or *.c works as expected. But I am just curious why the original one does not work for me. It seems that the all items following `in’ should not be enclosed in a pair of quotes, otherwise, all of them will be interpreted as one item, the string. I did try the following example, the output is as what I expected:
for f in "hello world"
do
echo "Processing $f file..."
# take action on each file. $f store current file name
done
OK, read sth. in the bash man. page.
Quoting is used to remove the special meaning of certain characters or words to the shell. Quoting can be used to disable special treatment for special characters, to prevent reserved words from being recognized as such, and to prevent parameter expansion.
I think that’s why the original script doesn’t work. “$FILES” is treated as “*”, which does disable special treatment for the special char `*’. It’s similar when you type `echo “*”`. `echo’ would not print out everything in the `pwd`. Instead, it simply prints out `*’.
Actually… if $FILES + the contents of /proc/$pid/environ are together > the output of “getconf ARG_MAX” this will fail.
The proper way to do this that always works is in the “useless use of cat awards” page:
http://partmaps.org/era/unix/award.html#backticks
The for is easier to read, but it is really annoying when your scripts fail with the dreaded “argument list too long” errors.
Forgot to share a little bashism that makes it easy to determine a good guess of the ARG_MAX:
MAXARGS=$(( $(getconf ARG_MAX) – $(env | wc -c) ))
I use loops like this a lot,
for x in * ; do
test -f “$x” || continue
COMMAND “$x”
done
helps to easily ignore subdirectories
Double quotes disable the special meaning of most enclosed characters. They do not disable the interpretation of variables with a leading $. To do this yopu need single quotes.
FILES=”*” is wrong unless you want the value of $FILES to be *. The same is true of “*.c”. Lose the quotes to get what you want.
I have always just used -
for F in *
do
… etc
for F in `ls`
is OK except that
for F in ls -1 (one)
is better, but both are more cumbersome and less elegant that
for F in * (or *.c and so on)
> FILES=”*” is wrong unless you want the value of $FILES to be *. The same is true of “*.c”. Lose the quotes to get what you want.
I dont think his wrong. He just gave those of us who are new to Bash scripting a placeholder for other commands or values. In fairness, the script did what it said it would do. Thanks for explaining the difference with using quotes and doing away with them.
For others reading this. Dont just take our words for it. Use the script – test it for yourself. Play with it. Thanks and congratulations to nixcraft for sharing.
Peace!
FILES=*; for f in $FILES; do… is WRONG.
for f in `ls`; do… is even WORSE.
Both break with filenames including whitespaces, newlines etc.
Since this is about bash use array if you want files in variables:
files=(*.c)
for f in “${files[@]}”; do cmd “$f”; done
Or just use glob:
for f in *.c; do cmd “$f”; done
I was initially a little confused by the thread. It’s not useful to assign * to a variable if you only intend to use it for a loop. Furthermore, as others have stated, putting quotes around the variable name prevent the glob, in this case *, from expanding.
* Vivek is not correct that $FILES should be quoted.
* DAY’s initial response that $FILES should be unquoted is not wrong, but using the variable at all is not useful.
* DAY’s second idea of looping over the output of `ls` is a very common mistake. It’s wrong because of wordsplitting.
* Brock Noland’s instinct to use
for file in *.c...is spot-on.* Jeff Schroeder is right to avoid ARG_MAX in general, but it only applies when you call exec*() via the kernel. Since for is a shell builtin, ARG_MAX doesn’t apply here.
* David Thompson and Baz’s comments are OK, but to Baz I would reiterate to avoid using the
lscommand for anything except human-readable output.* As for Chris’ comment:
FILES="*"andFILES=*are equivalent since sh-compliant shells don’t expand globs during variable assignment.One curious problem. If there are NO files matching the glob, you get this:
$ for file in *.jpg; do echo " the file variable is now '$file' " ; done
the file variable is now ‘*.jpg’
I would have expected that the contents of the for loop would not be executed at all, since there was no jpg file.
Found the problem with my previous example. The nullglob shell option needs to be set:
shopt -s nullglob; for file in *.jpg; do echo " the file variable is now '$file' " ; doneproduces no output, as expected. As opposed to:
shopt -u nullglob; for file in *.jpg; do echo " the file variable is now '$file' " ; donethe file variable is now '*.jpg'
@Jeff Schroeder:
The for is easier to read, but it is really annoying when your scripts fail with the dreaded “argument list too long” errors.
This won’t happen on a for-loop statement, since no exec() is done for the loop itself.
I need some in writing a script that will delete a specific line from a bunch of file in a directory.
your help is appreciated
@gurpur
Again, offtopic, go to our forum at nixcraft.com to post all your shell scripting related queries.