≡ Menu

Bash Shell Loop Over Set of Files

How do I run shell loop over set of files stored in a current directory or specified directory?

You can use for loop easily over a set of shell file under bash or any other UNIX shell using wild card character.

Syntax

The general syntax is as follows:

 
for f in file1 file2 file3 file5
do
 echo "Processing $f"
 # do something on $f
done

You can also use shell variables:

 
FILES="file1
/path/to/file2
/etc/resolv.conf"
for f in $FILES
do
	echo "Processing $f"
done
 

You can loop through all files such as *.c, enter:

$ for f in *.c; do echo "Processing $f file.."; done

Sample Shell Script To Loop Through All Files

#!/bin/bash
FILES=/path/to/*
for f in $FILES
do
  echo "Processing $f file..."
  # take action on each file. $f store current file name
  cat $f
done

Filename Expansion

You can do filename expansion in loop such as work on all pdf files in current directory:

 
for f in *.pdf
do
	echo "Removing password for pdf file - $f"
done
 

However, there is one problem with the above syntax. If there are no pdf files in current directory it will expand to *.pdf (i.e. f will be set to *.pdf"). To avoid this problem add the following statement before the for loop:

#!/bin/bash
# Usage: remove all utility bills pdf file password 
shopt -s nullglob
for f in *.pdf
do
	echo "Removing password for pdf file - $f"
        pdftk "$f" output "output.$f" user_pw "YOURPASSWORD-HERE"
done
 

Using A Shell Variable And While Loop

You can read list of files from a text file. For example, create a text file called /tmp/data.txt as follows:

file1
file2
file3

Now you can use the while loop as follows to read and process each by one by one:

#!/bin/bash
        while IFS= read -r file
        do
                [ -f "$file" ] && rm -f "$file"
        done < "/tmp/data.txt"
 

Here is another example which removes all unwanted files from chrooted lighttpd / nginx or Apache webserver:

 
#!/bin/bash
_LIGHTTPD_ETC_DEL_CHROOT_FILES="/usr/local/nixcraft/conf/apache/secure/db/dir.etc.list"
secureEtcDir(){
        local d="$1"
        local _d="/jails/apache/$d/etc"
        local __d=""
        [ -f "$_LIGHTTPD_ETC_DEL_CHROOT_FILES" ] || { echo "Warning: $_LIGHTTPD_ETC_DEL_CHROOT_FILES file not found. Cannot secure files in jail etc directory."; return; }
        echo "* Cleaning etc FILES at: \"$_d\" ..."
        while IFS= read -r file
        do
                __d="$_d/$file"
                [ -f "$__d" ] && rm -f "$__d"
        done < "$_LIGHTTPD_ETC_DEL_CHROOT_FILES"
}
 
secureEtcDir "nixcraft.net.in"

Processing Command Line Arguments

 
#!/bin/bash
# make sure you always put $f in double quotes to avoid any nasty surprises i.e. "$f"
for f in $*
do
  echo "Processing $f file..."
  # rm "$f"
done
 

OR

 
#!/bin/bash
# make sure you always put $f in double quotes to avoid any nasty surprises i.e. "$f"
for f in $@
do
  echo "Processing $f file..."
  # rm "$f"
done
 

Please note that $@ expanded as "$1" "$2" "$3" ... "$n" and $* expanded as "$1y$2y$3y...$n", where y is the value of IFS variable i.e. "$*" is one long string and $IFS act as an separator or token delimiters.
The following example use shell variables to store actual path names and then files are processed using the for loop:

#!/bin/bash
_base="/jail/.conf"
_dfiles="${base}/nginx/etc/conf/*.conf"
 
for f in $_dfiles
do
        lb2file="/tmp/${f##*/}.$$"   #tmp file
        sed 's/Load_Balancer-1/Load_Balancer-2/' "$f" > "${lb2file}"   # update signature 
        scp "${lb2file}" nginx@lb2.nixcraft.net.in:${f}   # scp updated file to lb2
        rm -f "${lb2file}"
done
 

Updated for accuracy!

Tweet itFacebook itGoogle+ itPDF itFound an error/typo on this page?

{ 39 comments… add one }

  • DAY May 11, 2008, 12:35 pm

    The scripts are wrong, use `$FILES` (without quotes) in the loop instead of `”$FILES”`.

  • nixCraft May 11, 2008, 1:57 pm

    DAY,

    I don’t think so it is wrong. Do you have any problem running script?

  • DAY May 11, 2008, 3:14 pm

    Yes, the script does not work for me. Both the one directly typed in the cmd line and the one from a script file. They output of the first one is:

    Processing *.c file..

    The output of the second is:

    Processing * file…

    In fact, I just start learn the shell script. Here is another one which can do the same job:

    #!/bin/sh
    for f in `ls`
    do
    echo “Processing $f file …”
    done

  • Brock Noland May 11, 2008, 7:36 pm

    Hmm…not sure why you wouldn’t use

    for file in *

    or

    for file in *.c

  • nixCraft May 11, 2008, 8:54 pm

    DAY,

    Hmm, it should work (I hope you have *.c files in current directory) or try as suggested by brock.

  • DAY May 12, 2008, 2:47 am

    Thanks, vivek and Brock. Replacing “$FILES” with * or *.c works as expected. But I am just curious why the original one does not work for me. It seems that the all items following `in’ should not be enclosed in a pair of quotes, otherwise, all of them will be interpreted as one item, the string. I did try the following example, the output is as what I expected:

    for f in "hello world"
    do
    echo "Processing $f file..."
    # take action on each file. $f store current file name
    done

  • DAY May 12, 2008, 4:13 am

    OK, read sth. in the bash man. page.

    Quoting is used to remove the special meaning of certain characters or words to the shell. Quoting can be used to disable special treatment for special characters, to prevent reserved words from being recognized as such, and to prevent parameter expansion.

    I think that’s why the original script doesn’t work. “$FILES” is treated as “*”, which does disable special treatment for the special char `*’. It’s similar when you type `echo “*”`. `echo’ would not print out everything in the `pwd`. Instead, it simply prints out `*’.

  • Jeff Schroeder May 12, 2008, 5:08 pm

    Actually… if $FILES + the contents of /proc/$pid/environ are together > the output of “getconf ARG_MAX” this will fail.

    The proper way to do this that always works is in the “useless use of cat awards” page:
    http://partmaps.org/era/unix/award.html#backticks

    The for is easier to read, but it is really annoying when your scripts fail with the dreaded “argument list too long” errors.

  • Jeff Schroeder May 12, 2008, 5:10 pm

    Forgot to share a little bashism that makes it easy to determine a good guess of the ARG_MAX:
    MAXARGS=$(( $(getconf ARG_MAX) – $(env | wc -c) ))

  • David Thompson May 16, 2008, 4:46 am

    I use loops like this a lot,

    for x in * ; do
    test -f “$x” || continue
    COMMAND “$x”
    done

    helps to easily ignore subdirectories

  • Baz June 8, 2008, 12:12 am

    Double quotes disable the special meaning of most enclosed characters. They do not disable the interpretation of variables with a leading $. To do this yopu need single quotes.

    FILES=”*” is wrong unless you want the value of $FILES to be *. The same is true of “*.c”. Lose the quotes to get what you want.

    I have always just used –

    for F in *
    do
    … etc

    for F in `ls`
    is OK except that
    for F in ls -1 (one)
    is better, but both are more cumbersome and less elegant that
    for F in * (or *.c and so on)

  • Chris December 11, 2008, 1:00 am

    > FILES=”*” is wrong unless you want the value of $FILES to be *. The same is true of “*.c”. Lose the quotes to get what you want.

    I dont think his wrong. He just gave those of us who are new to Bash scripting a placeholder for other commands or values. In fairness, the script did what it said it would do. Thanks for explaining the difference with using quotes and doing away with them.

    For others reading this. Dont just take our words for it. Use the script – test it for yourself. Play with it. Thanks and congratulations to nixcraft for sharing.

    Peace!

  • Foo February 11, 2009, 9:47 pm

    FILES=*; for f in $FILES; do… is WRONG.
    for f in `ls`; do… is even WORSE.
    Both break with filenames including whitespaces, newlines etc.

    Since this is about bash use array if you want files in variables:
    files=(*.c)
    for f in “${files[@]}”; do cmd “$f”; done

    Or just use glob:
    for f in *.c; do cmd “$f”; done

  • Michael Smith February 11, 2009, 10:00 pm

    I was initially a little confused by the thread. It’s not useful to assign * to a variable if you only intend to use it for a loop. Furthermore, as others have stated, putting quotes around the variable name prevent the glob, in this case *, from expanding.
    * Vivek is not correct that $FILES should be quoted.
    * DAY’s initial response that $FILES should be unquoted is not wrong, but using the variable at all is not useful.
    * DAY’s second idea of looping over the output of `ls` is a very common mistake. It’s wrong because of wordsplitting.
    * Brock Noland’s instinct to use for file in *.c... is spot-on.
    * Jeff Schroeder is right to avoid ARG_MAX in general, but it only applies when you call exec*() via the kernel. Since for is a shell builtin, ARG_MAX doesn’t apply here.
    * David Thompson and Baz’s comments are OK, but to Baz I would reiterate to avoid using the ls command for anything except human-readable output.
    * As for Chris’ comment: FILES="*" and FILES=* are equivalent since sh-compliant shells don’t expand globs during variable assignment.

  • rduke15 April 12, 2009, 2:40 pm

    One curious problem. If there are NO files matching the glob, you get this:

    $ for file in *.jpg; do echo " the file variable is now '$file' " ; done

    the file variable is now ‘*.jpg’

    I would have expected that the contents of the for loop would not be executed at all, since there was no jpg file.

  • rduke15 April 12, 2009, 2:53 pm

    Found the problem with my previous example. The nullglob shell option needs to be set:
    shopt -s nullglob; for file in *.jpg; do echo " the file variable is now '$file' " ; done

    produces no output, as expected. As opposed to:

    shopt -u nullglob; for file in *.jpg; do echo " the file variable is now '$file' " ; done
    the file variable is now '*.jpg'

  • TheBonsai May 5, 2009, 4:57 am

    @Jeff Schroeder:


    The for is easier to read, but it is really annoying when your scripts fail with the dreaded “argument list too long” errors.

    This won’t happen on a for-loop statement, since no exec() is done for the loop itself.

  • gurpur December 17, 2009, 3:29 am

    I need some in writing a script that will delete a specific line from a bunch of file in a directory.
    your help is appreciated

    • nixCraft December 17, 2009, 6:29 am

      @gurpur

      Again, offtopic, go to our forum at nixcraft.com to post all your shell scripting related queries.

  • Xa December 19, 2010, 4:14 am

    for i in ‘ls’
    did NOT work for me. What does work is this:
    for i in $(ls)

    • mwcz April 5, 2013, 5:56 pm

      That’s because you used single quotes instead of backticks.

      It’s not this:

      for i in ‘ls’

      It’s this:

      for i in `ls`

  • she March 7, 2011, 4:21 pm

    can somebody help me develop a program that has the option whether to overwrite a document or not. when making a file, it will ask the user if he would want to create a new file on the same file name or overwrite the said filename.. please i need this badly.

  • Benjamin June 24, 2011, 8:12 pm

    Nice compilation, thanks! One suggestion, though: I think many people would be looking for still another thing: how to get file extensions and how to get filenames without extensions.

  • monr4 July 15, 2011, 9:09 pm

    hello, a have a problem, i need to do a script por a rutine something like

    for i in $(cat list1) but with 2 list
    for i in $(cat list1) AND b in $(cat list2);do mknod /dev/$i/group c64[$b];done

    somebody helpme with that please

  • Hari December 7, 2011, 12:21 pm

    Hello I have a problem .i want to run a program N times by give N number of different .txt files from a directory. i want to give all the files in single time by using loops can any one help me to script .

  • Saad January 10, 2012, 6:48 am

    Is it possible to know how many files will this loop run on,

    e.g. in the following loop

    for f in *.pdf
    do

    done;

    can i know how many files are present in *.pdf glob, and can i use that to know how many files have been processed out of the total number

    • RDuke January 10, 2012, 3:32 pm

      @saad: maybe something like

      count=$( ls -1 *.pdf | wc -l )

    • nixCraft January 11, 2012, 8:27 am

      Pure bash code and nothing else :)

       filecount=(*.pdf)
       echo "Total files in $PWD: ${#filecount[@]}"
      
  • Potato January 11, 2012, 12:26 am

    try COUNT=`ls -1 *.pdf | wc -l`

  • ziming May 12, 2012, 10:41 am

    Very good tutorials. Helps me lot. Thx mate.

  • Manuel Rodriguez May 30, 2012, 4:42 pm

    good stuff

  • Mike February 14, 2013, 7:53 am

    thanks this was helpfull (still!!) ;) and I have been scripting for 20 years!

  • Inukaze October 11, 2013, 7:51 pm

    Hi there im trying to say to Bash , wait for a Process Start / Begin

    im trying by this way for example

    notepad=`pidof notepad.exe`
    until [ $notepad > 0 ]
    do
        taskset -p 03 $notepad
        renince -n 5 -p $notepad
    done
    

    well i have 2 questions:
    First -> why this generate a file called “0” (the file are empty)
    i dont wanna make a new file , just wait for the PID to check execution.

    Second -> This is a Loop , but if the 2 commands are execute correclty , 1 time , how i can continue to done ???

  • Aaron Burns April 1, 2014, 9:12 pm

    I keep seeing “for f in “…. what is the f for. I don’t see it defined anywhere.

  • Giri March 19, 2015, 9:31 am

    Hi All,
    I want to write a shell script for processing the input files(file.csv) from one path and redirect to the other mount or same mount for required query based on the acknowledgment file name (same as input file but file.done) .Please help me

  • JKT April 20, 2015, 8:07 pm

    It should be noted that none of the above examples work if you want to explicitly specify a directory to iterate through (instead of the current directory, aka *) that has a space in it. I still can’t figure out how to do that. If I set a variable to the explicit path, neither quotes nor delimiters allow iteration through that directory; the script’s attempt to navigate to that directory stops at the space as if a new argument started after it. And since this isn’t that unusual a situation, it’s unfortunate nobody covered it above. Everyone got stuck on using the current directory, which I would think is actually a less-versatile construct.

Leave a Comment