65 comment

1. Hi, I don’t fully understand how this…

   BASH for loop works nicely under UNIX / Linux / Windows and OS X

   works nicely under Windows…

   Oh! btw, thank you.

2. Install cygwin from http://www.cygwin.com/

   Install bash and your are done!

3. For the shell equivalent of python’s splitlines, I always did it like this:

   oldifs="$IFS"
   IFS="
   "
   for line in $(< line-with-spaces-between-fields.txt); do
      ...
   done
   IFS="$oldifs"

   or while read ... instead of for when the list exceeds $(getconf ARG_MAX)

   Your way works too, but is more typing

4. Thanks really so much,
   It’s so useful and I think it’d save me a lot of time and help me do better shell coding..

5. This is a great blog!

   Instead of using following:

   find . -print0 | while read -d $'' file
   do
     echo -v "$file"
   done

   I just use:

   find . | while read file
   do
     echo "$file"
   done

   Never had problems with it. A filename can impossible contain newline-chars, or am I wrong?

   Best regards,
   Benjamin Schmidt

   1. File names can contain newline characters.

      Gnome for example displays them correctly. Couldn’t manage to create such a file in Gnome. I tested creation in C and Python iirc.

      1. In bash you can create a file with a newline in the name using, e.g:

         touch $’weirdnfile’

6. @Jeff/Benjamin

   Thanks for sharing your code and ideas.

7. i’d love to play music with mplayer… through cli
   let say that i have song/music folder with many directory inside, how i can play with mplayer with single command. Actually i have one bash script, but i got trouble when facing white space… mplayer wouldn’t play the file

   [songRequest.sh]
   for i in ` find -name “*mp3″`
   do
   mplayer $i
   done

   any help will be appreciated

8. Try

   find .  -iname "*.mp3" -print0 | while read -d $'' file
   do
     mplayer "$file"
   done

9. > A filename can impossible contain newline-chars, or am I wrong?

   A filename can contain newlines. It can contain any character except a slash (/) or an ASCII NUL (0x0).

10. # use for loop read all filenames
    > for (( i=0; i do
    > echo “${fileArray[$i]}”
    > done

    If all you want to do is print the contents of the array:

    printf "%sn" "${fileArray[@]}"

    To loop through them, use:

    for i in "${fileArray[@]}"
do
: ... whatever
done

11. great… vivek.gite ^^

    your script just work on one file… then mplayer killed immediately.
    yeah just playing one file only(there are many left).

    on playing progress i can’t get control of mplayer, even i can’t turn up/down volume, go to the next song, etc.

    all i can do is Ctrl+C

    btw thanks

12. I guess you can try
    mplayer *.mp3

    Or send all files to .playlist.txt
    ls -1 > .playlist.txt
mplayer -playlist .playlist.txt

13. All this information is very useful, but I think that needing it is sympton: ¿shouldn’t you use another language instead?

    Nice post, though

14. “All this information is very useful, but I think that needing it is sympton: ¿shouldn’t you use another language instead?”

    Why? It’s straightforward shell scripting.

15. opendir(DIR, “.”) or die(“Error opening directory”);
    my @readFiles = readdir(DIR);
    foreach (@readFiles) {
    …

16. I don’t think there is a need to reset the IFS variable, all you really need to do to avoid problems with word splitting is to put the variables that appear in the loop body in double quotes (I find this to be a good default habit). Once in double quotes the shell will consider whatever the variable expands to as a single argument and pass it to the program as such.

    For example:
    for i in *.mp3; do rm “$i”; done

    mhernandez: Its not a problem with the shell, its just how it works, the bash shell is based on word splitting.

    1. This doesn’t seem to work for me. I’m wondering what I’m doing wrong….

       If I do this:

       $ for file in $(ls | awk -F. ‘{print $1}’); do sox -S “${file}.m4a” “${file}.ogg”; done

       it doesn’t work as I would have expected.

17. You can reset the IFS with
    
unset IFS

    this way, you do not need to save the state and then recall it from a variable.

18. Kyle is correct. No need to mess w/ your IFS var.
    Your 2nd example is problematic in a very subtle way

    FILES=/data/*

    does not set the variable ‘FILES’ to the names of all the files in /data, it sets it to ‘/data/*’. You can see this w/:

    declare -p FILES

    A very useful technique in all this, is to employ the ‘set’ builtin:

    
cd ..; set *.txt; cd ~-
for f; do
echo "../$f" "$f"
done


    This lets you scoop up a file list (w/ nasty names and all), and then move somewhere else to process them.

19. i am doing a project in unix shell scripting.
    so my project have messages to be displayed. my problem is that how to add a beep sound to the messages …can any help me to include that also with the coding??pls..its very urgent…i dont know the shell codings…its for my project

20. Chris F.A. Johnson has it right on the nose.

    use a quoted @-array: <code“${fileArray[@]}”

    A quoted array with an @ subscript expands to a list of quoted elements of that array.

    No IFS manipulation required!

    Reid

21. Thanks man, my code is now only 40 lines long, not 136 like before xD^^
    nice blog

22. mp3(){
    local o=$IFS
    IFS=$(echo -en “nb”)
    /usr/bin/beep-media-player “$(cat $@)” &
    IFS=o
    }
    “$(cat $@)” if you use this I think the IFS down work …
    I think you want $(cat “$@”)

23. Thank you! I found this post most useful.

24. Instead of telling echo to print a char for new line and (probably) bell and then *not* to print a new line, one can just use echo’s default new line output 🙂

    IFS=`echo`

    Thanks for your share!

25. Why “IFS=`echo`”? Command substitution is slow (and unnecessary).

    IFS=’
    ‘

    IFS=$’n’ ## bash

26. Amazing! Your notation for creating an array from:
    > fileArray=($(find $DIR -type f))
    is stunning! Thanks for the excellent writeup.

27. I just use as following
    with double quotes
    [code]
    for m in “`ls`”; do echo “$m” ; done
    [/code]

28. for m in "`ls`"; do echo "$m" ; done

    Why?

    The for command will only see one argument.

    Try this:

    
n=1
for m in "`ls`"
do
echo "$n"
echo "$m"
n=$(( $n + 1 ))
done


    You’ll see that it only goes through the loop once.

    You don’t need a loop for that, and you don’t need ls:

    printf "%sn" *

29. excellent post. exactly what i was looking for. i had been facing this space-field-seperator problem for quite a sometime today. had tried many options to circumvent it, but in vain. will try this $IFS approach now.

    thanks a lot for sharing.

    shreyas

30. My favorite solution, though it can theoretically overmatch:

    find . | sed -e “s/ /?/g”
    ls | sed -e “s/ /?/g”

    Then you can do things like greps, for loops, whatever equally well:
    grep hello $(ls | sed -e “s/ /?/g”)
    for i in $(find . | sed -e “s/ /?/g) do; ls -l “$i”; done

31. “grep hello $(ls | sed -e “s/ /?/g”)”

    You don’t need sed or ls; use filename expansion:

    grep hello *

    “for i in $(find . | sed -e “s/ /?/g) do; ls -l “$i”; done”

    That will fail if any filenames contain whitespace.

    It is also extremely inefficient to call ls in a loop.

32. Great post! It has been very useful for me 😀

33. Hey, thanks for this! I never did much bash scripting and was trying to figure out how to parse an array from a bash RC file into a Perl variable. I started out writing a long parser hack, but trying to support array entries with spaces was a big headache. Then thought maybe bash should do the work instead, and your examples helped a lot. Here’s a one-liner bash script that’s run using the Perl `backtick` operator.

    $path is the RC file, and $var is the array variable. BASH variables must be escaped, but PERL
    variables must not, and new-line chars have to be added so bash is happy with the string.
    First step is to source the RC file, and next get the length of the array.

    $list = `. $path; L=${#${var}[@]}n for (( i=0; i<${L}; i++ ));n don echo -n ${${var}[$i]},n done`;
    chop( $list ); # strip the trailing comma

    foreach $entry (split(",", $list)) { # Prove that it works
    print "entry: '$entry'n";
    }

    Here's an example of the bash array that this will parse. Cool. Thanks again for your examples.

    ARRAY_VAR=( .bashrc .vimrc "foo bar" .java
    .mozilla
    .Xdefaults "other file name" )

34. Fixed my little script! Cheers for posting this – always nice to learn something new.

35. Trying to get the following to work. If I cut and paste the eventual contents of $directoryFiles EITHER in the bash script OR on the command line, it works perfectly. If I try to get it to do it directly in the script, it fails with:

    tar –all the file names– : Cannot stat: No such file or directory

    I”ve tried about every expansion or quoting that I can think of for the $direcotryFiles variable.

    #!/bin/bash
    SAVEIFS=$IFS
    IFS=$(echo -en "nb")
    directoryFiles=""
    
    for file in $( find /home/gearond -maxdepth 1 -type f)
    do
      directoryFiles="$directoryFiles"$file" "
      #echo $file
    done
    
    tar -cf /tmp/directoryFiles.tar.bz2  $directoryFiles
    
    IFS=$SAVEIFS
    

    1. How about:

       find /home/gearond -maxdepth 1 -type f -print0 | xargs -O -I file your-command-on file

       In your case:

       find . -maxdepth 1 -type f -print0 | xargs -0 -I {} tar -rjvf /tmp/directoryFiles.tar.bz2 {}
       tar -tjvf /tmp/directoryFiles.tar.bz2

36. Thank you very much, this is very helpful.
    I work with Linux for many year, but until today, I did not know this.

37. What is the difference between

    IFS=$(echo -en "nb")
    

    and

    IFS=''
    

    Which do exactly the same (but don’t call sub shell)

38. They do not do the same thing.

    The first assigns a newline and a backspace to IFS; the second assigns an empty string.

    The first assignment can be done more efficiently with:

    IFS=$'nb'
    

39. Thanks to all contributors of this outstanding discussion!

40. Thanks for your tip. I was not thinking anymore about IFS. Here’s a tip to remove unwanted characters in filenames:

    SAVEIFS=$IFS
    IFS=$(echo -en "nb")
    FILES=$1/*
    for f in $FILES
    do
    	#echo	"$f"
    	FILENAME="${f//[?%+]/_}";
    	mv -b --strip-trailing-slashes "$f" "$FILENAME"
    done
    IFS=$SAVEIFS
    

    replaces ?, % and + with _ (Useful if you copy FAT32 files to linux and back)

41. For simple operations on multiple files you can use the -exec option on find:

    find . -iname ‘*.avi’ -exec mv {} /Some/Where/Else/ ;

    This command recursively finds all .avi files in the current directory and moves them to folder /Some/Where/Else. The curly brackets {} are substituted with file name. Consult the man page for more details.

42. array=($(mysql –host=192.168.1.2 –user=XXXX –password=XXXX -s -N -e ‘use xbmc_video; SELECT c22 FROM movie, files WHERE files.idFile=movie.idFile ORDER BY idMovie DESC LIMIT 10;’))

    how can i get rid of spaces in this array

    1. array=( "${array[@]// /}" )
       

43. Thanks for the very clear explanation!

44. Hello there,

    I tried the code for storing the filenames into an array variable with a few modifications:

    #!/bin/bash
    DIR=$HOME/HY_DUMP/
    
    # failsafe - fall back to current directory
    #[ "$DIR" == "" ] && DIR="."
    
    # save and change IFS
    OLDIFS=$IFS
    IFS=$'nb'
    cd $DIR
    # read all file name into an array
    fileArray = ( $(`ls -l`) )
    
    # restore it
    IFS=$OLDIFS
    
    # get length of an array
    tLen=${#fileArray[@]}
    echo $tLen
    # use for loop read all filenames
    for (( i=0; i<${tLen}; i++ ));
    do
      echo "${fileArray[$i]}"
    done
    
    

    But, i have been running into errors on this:

    error:

    script.sh: syntax error at line 12: `(' unexpected

    I have no clue how to proceed with this problem. Could anyone tweek this problem?

    Thanks.
    -A

45. fileArray = ( $(`ls -l`) )

    should be

    fileArray=( * )

46. Hi there,
    I used the above example script for concatenating multiple files which works fine for me if the directory structure contains no spaces.However now I am facing a dierctory structure that does have spaces. Setting the IFS doesn’t seem to help here.
    The directory structure I am facing is such: ./CH 0000010240/A10/21MT/21mt#001.csv and I want to concatenate all .csv files I find. Here is my current version of the script which I am running on OS X:

    #!/bin/bash
    DIR="$1"
    
    # failsafe - fall back to current directory
    [ "$DIR" == "" ] && DIR="."
    
    # save and change IFS
    OLDIFS=$IFS
    IFS=$'n'
    
    # read all file name into an array
    fileArray=($(find $DIR -type f))
    
    # restore it
    IFS=$OLDIFS
    
    # get length of an array
    tLen=${#fileArray[@]}
    
    # use for loop read all filenames
    for (( i=0; i> all.csv
    done
    

    Thanks for any hints.
    Cheers,M

47. > #!/bin/bash
    > DIR="$1"
    > # failsafe - fall back to current directory
    > [ "$DIR" == "" ] && DIR="."
    

    DIR=${DIR:-.}

    > # save and change IFS
    > OLDIFS=$IFS
    > IFS=$'n'
    > # read all file name into an array
    > fileArray=($(find $DIR -type f))
    > # restore it
    > IFS=$OLDIFS
    > # get length of an array
    > tLen=${#fileArray[@]}
    > # use for loop read all filenames
    > for (( i=0; i> all.csv
    

    for file in “${filearray[@]}”
    do
    : do something with “$file” (in quotes)

    > done
    

48. Nixcrift FTW again. Good job.

49. Very helpful. Thanks

50. It’s very helpful. Thank you

51. This can be simpler

    for f in “` `”

    just the ” around the comand and that is it

52. Can anyone explain to me what the -v’s, used with echo and cp, in the “Using old good find command to process file names” section above, are for/do?

53. Thank you very much Author for sharing these knowledge with other people!

54. The filename handling examples are wrong in several ways. Filenames can include newlines, backslashes, backspaces, tabs, asterisks, and other weirdness, which this doesn’t handle.

    # For example, wrong – doesn’t handle filenames with n or b
    IFS=$(echo -en “nb”)
    for f in *
    do
    echo “$f”
    done

    For more on how to do it correctly, see:
    http://www.dwheeler.com/essays/filenames-in-shell.html

55. Excelente! Tank You!!!

56. Thanks! you’re my savior.

57. Thanks. I had been trying to do:
    > for f in `ls`;
    and
    > for f in $( ls );
    Both were problematic. But
    > for f in *; works, regardless of $IFS AFAICT.

58. This site should have a bitcoin donation link.

59. Thanks for this. Worked a treat. I wanted to delete dozens of folders in iTunes containing the word “Riddim”, and ended up with this:

    SAVEIFS=$IFS

    IFS=$(echo -en “nb”)

    for d in `ls` ; do g=`ls “$d” | grep Riddim` ; if [ “” != “$g” ] ; then echo “$d – $g” ; fi ; done

    IFS=$SAVEIFS

60. Every time i forget something and promptly Google for it, I land on your website! Thanks for all your efforts. Brilliant.

    Have a question? Post it on our forum!