Bash Shell Loop Over Set of Files

How do I run shell loop over set of files stored in a current directory or specified directory?

You can use for loop easily over a set of shell file under bash or any other UNIX shell using wild card character.

Syntax

The general syntax is as follows:

for f in file1 file2 file3 file5
do
 echo "Processing $f"
 # do something on $f
done

You can also use shell variables:

FILES="file1
/path/to/file2
/etc/resolv.conf"
for f in $FILES
do
	echo "Processing $f"
done

You can loop through all files such as *.c, enter:

$ for f in *.c; do echo "Processing $f file.."; done

Sample Shell Script To Loop Through All Files

#!/bin/bash
FILES=/path/to/*
for f in $FILES
do
  echo "Processing $f file..."
  # take action on each file. $f store current file name
  cat $f
done

Filename Expansion

You can do filename expansion in loop such as work on all pdf files in current directory:

for f in *.pdf
do
	echo "Removing password for pdf file - $f"
done

However, there is one problem with the above syntax. If there are no pdf files in current directory it will expand to *.pdf (i.e. f will be set to *.pdf”). To avoid this problem add the following statement before the for loop:

#!/bin/bash
# Usage: remove all utility bills pdf file password 
shopt -s nullglob
for f in *.pdf
do
	echo "Removing password for pdf file - $f"
        pdftk "$f" output "output.$f" user_pw "YOURPASSWORD-HERE"
done

Using A Shell Variable And While Loop

You can read list of files from a text file. For example, create a text file called /tmp/data.txt as follows:

file1
file2
file3

Now you can use the while loop as follows to read and process each by one by one:

#!/bin/bash
        while IFS= read -r file
        do
                [ -f "$file" ] && rm -f "$file"
        done < "/tmp/data.txt"

Here is another example which removes all unwanted files from chrooted lighttpd / nginx or Apache webserver:

#!/bin/bash
_LIGHTTPD_ETC_DEL_CHROOT_FILES="/usr/local/nixcraft/conf/apache/secure/db/dir.etc.list"
secureEtcDir(){
        local d="$1"
        local _d="/jails/apache/$d/etc"
        local __d=""
        [ -f "$_LIGHTTPD_ETC_DEL_CHROOT_FILES" ] || { echo "Warning: $_LIGHTTPD_ETC_DEL_CHROOT_FILES file not found. Cannot secure files in jail etc directory."; return; }
        echo "* Cleaning etc FILES at: \"$_d\" ..."
        while IFS= read -r file
        do
                __d="$_d/$file"
                [ -f "$__d" ] && rm -f "$__d"
        done < "$_LIGHTTPD_ETC_DEL_CHROOT_FILES"
}
 
secureEtcDir "nixcraft.net.in"

Processing Command Line Arguments

#!/bin/bash
# make sure you always put $f in double quotes to avoid any nasty surprises i.e. "$f"
for f in $*
do
  echo "Processing $f file..."
  # rm "$f"
done

OR

#!/bin/bash
# make sure you always put $f in double quotes to avoid any nasty surprises i.e. "$f"
for f in $@
do
  echo "Processing $f file..."
  # rm "$f"
done

Please note that $@ expanded as “$1” “$2” “$3” … “$n” and $* expanded as “$1y$2y$3y…$n”, where y is the value of IFS variable i.e. “$*” is one long string and $IFS act as an separator or token delimiters.
The following example use shell variables to store actual path names and then files are processed using the for loop:

#!/bin/bash
_base="/jail/.conf"
_dfiles="${base}/nginx/etc/conf/*.conf"
 
for f in $_dfiles
do
        lb2file="/tmp/${f##*/}.$$"   #tmp file
        sed 's/Load_Balancer-1/Load_Balancer-2/' "$f" > "${lb2file}"   # update signature 
        scp "${lb2file}" nginx@lb2.nixcraft.net.in:${f}   # scp updated file to lb2
        rm -f "${lb2file}"
done

Updated for accuracy!

🐧 If you liked this page, please support my work on Patreon or with a donation.
🐧 Get the latest tutorials on SysAdmin, Linux/Unix, Open Source/DevOps topics:
CategoryList of Unix and Linux commands
File Managementcat
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Network Utilitiesdig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg jobs killall kill pidof pstree pwdx time
Searchinggrep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
47 comments… add one
  • Benjamin Jun 24, 2011 @ 20:12

    Nice compilation, thanks! One suggestion, though: I think many people would be looking for still another thing: how to get file extensions and how to get filenames without extensions.

  • monr4 Jul 15, 2011 @ 21:09

    hello, a have a problem, i need to do a script por a rutine something like

    for i in $(cat list1) but with 2 list
    for i in $(cat list1) AND b in $(cat list2);do mknod /dev/$i/group c64[$b];done

    somebody helpme with that please

  • Hari Dec 7, 2011 @ 12:21

    Hello I have a problem .i want to run a program N times by give N number of different .txt files from a directory. i want to give all the files in single time by using loops can any one help me to script .

  • Saad Jan 10, 2012 @ 6:48

    Is it possible to know how many files will this loop run on,

    e.g. in the following loop

    for f in *.pdf
    do

    done;

    can i know how many files are present in *.pdf glob, and can i use that to know how many files have been processed out of the total number

    • RDuke Jan 10, 2012 @ 15:32

      @saad: maybe something like

      count=$( ls -1 *.pdf | wc -l )

    • 🐧 nixCraft Jan 11, 2012 @ 8:27

      Pure bash code and nothing else :)

       filecount=(*.pdf)
       echo "Total files in $PWD: ${#filecount[@]}"
      
  • Potato Jan 11, 2012 @ 0:26

    try COUNT=`ls -1 *.pdf | wc -l`

  • ziming May 12, 2012 @ 10:41

    Very good tutorials. Helps me lot. Thx mate.

  • Manuel Rodriguez May 30, 2012 @ 16:42

    good stuff

  • Mike Feb 14, 2013 @ 7:53

    thanks this was helpfull (still!!) ;) and I have been scripting for 20 years!

  • Inukaze Oct 11, 2013 @ 19:51

    Hi there im trying to say to Bash , wait for a Process Start / Begin

    im trying by this way for example

    notepad=`pidof notepad.exe`
    until [ $notepad > 0 ]
    do
        taskset -p 03 $notepad
        renince -n 5 -p $notepad
    done
    

    well i have 2 questions:
    First -> why this generate a file called “0” (the file are empty)
    i dont wanna make a new file , just wait for the PID to check execution.

    Second -> This is a Loop , but if the 2 commands are execute correclty , 1 time , how i can continue to done ???

  • Aaron Burns Apr 1, 2014 @ 21:12

    I keep seeing “for f in “…. what is the f for. I don’t see it defined anywhere.

  • Giri Mar 19, 2015 @ 9:31

    Hi All,
    I want to write a shell script for processing the input files(file.csv) from one path and redirect to the other mount or same mount for required query based on the acknowledgment file name (same as input file but file.done) .Please help me

  • JKT Apr 20, 2015 @ 20:07

    It should be noted that none of the above examples work if you want to explicitly specify a directory to iterate through (instead of the current directory, aka *) that has a space in it. I still can’t figure out how to do that. If I set a variable to the explicit path, neither quotes nor delimiters allow iteration through that directory; the script’s attempt to navigate to that directory stops at the space as if a new argument started after it. And since this isn’t that unusual a situation, it’s unfortunate nobody covered it above. Everyone got stuck on using the current directory, which I would think is actually a less-versatile construct.

  • The Calibre Jun 9, 2015 @ 16:11

    Very good tutorial, Thanks.

  • MiaS Jul 14, 2015 @ 15:22

    It seems to be a nice thread.

    I’m having one similar issue and I guess Unix Loop might help.

    I need a write a cleanup script in unix shell to delete files and folders older than 3 days but the problem is that I need to exclude .snp files/folders. And that can be present any where in the directory. I tried with normal “find” command but that is not excluding the file from nested directory…
    It seems only loop can help…

    Any idea?

    • Captain Obvious Jan 19, 2016 @ 15:57

      Continue to use your find command. You can match the inverse of patterns with grep. The following will return everything-but what is specified in the pattern.

      grep -v somepatternIwanttoexclude

      so for example,
      find blah blah | grep -v .snp | grep -v .snpFolder

  • Alan Santos Jul 17, 2015 @ 13:52

    Very usefull!

    Thank you!

    att,

  • Arun Feb 17, 2016 @ 15:59

    shopt -s nullglob helped me. Thanks

  • Old Unix Dude Apr 4, 2016 @ 19:35

    It is worth noting that none of these examples that use * will scale. It is much more reliable to use the find command. To “loop” over every file in /dir and run some_command :

    find /dir -exec some_command {} \;

    HINT: find replaces {} with the current filename and \; marks the end of the command.

Leave a Reply

Your email address will not be published. Required fields are marked *

Use HTML <pre>...</pre>, <code>...</code> and <kbd>...</kbd> for code samples.