Bash Shell: Replace a String With Another String In All Files Using sed and Perl -pie Options

Posted on in Categories , , , last updated March 9, 2011

How do I replace a string with another string in all files? For example, ~/foo directory has 100s of text file and I’d like to find out xyz string and replace with abc. I’d like to use sed or any other tool to replace all occurrence of the word.

The sed command is designed for this kind of work i.e. find and replace strings or words from a text file under Apple OX, *BSD, Linux, and UNIX like operating systems. The perl can be also used as described below.

sed replace word / string syntax

The syntax is as follows:
sed -i 's/old-word/new-word/g' *.txt

GNU sed command can edit files in place (makes backup if extension supplied) using the -i option. If you are using an old UNIX sed command version try the following syntax:

sed 's/old/new/g' input.txt > output.txt

You can use old sed syntax along with bash for loop:

#!/bin/bash
OLD="xyz"
NEW="abc"
DPATH="/home/you/foo/*.txt"
BPATH="/home/you/bakup/foo"
TFILE="/tmp/out.tmp.$$"
[ ! -d $BPATH ] && mkdir -p $BPATH || :
for f in $DPATH
do
  if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
   sed "s/$OLD/$NEW/g" "$f" > $TFILE && mv $TFILE "$f"
  else
   echo "Error: Cannot read $f"
  fi
done
/bin/rm $TFILE

A Note About Bash Escape Character

A non-quoted backslash \ is the Bash escape character. It preserves the literal value of the next character that follows, with the exception of newline. If a \newline pair appears, and the backslash itself is not quoted, the \newline is treated as a line continuation (that is, it is removed from the input stream and effectively ignored). This is useful when you would like to deal with UNIX paths. In this example, the sed command is used to replace UNIX path “/nfs/apache/logs/rawlogs/access.log” with “__DOMAIN_LOG_FILE__”:

#!/bin/bash
## Our path
_r1="/nfs/apache/logs/rawlogs/access.log"
 
## Escape path for sed using bash find and replace 
_r1="${_r1//\//\\/}"
 
# replace __DOMAIN_LOG_FILE__ in our sample.awstats.conf
sed -e "s/__DOMAIN_LOG_FILE__/${_r1}/" /nfs/conf/awstats/sample.awstats.conf  > /nfs/apache/logs/awstats/awstats.conf 
 
# call awstats
/usr/bin/awstats -c /nfs/apache/logs/awstats/awstats.conf

The $_r1 is escaped using bash find and replace parameter substitution syntax to replace each occurrence of / with \/.

perl -pie Syntax For Find and Replace

The syntax is as follows:
perl -pie 's/old-word/new-word/g' input.file > new.output.file

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin and a trainer for the Linux operating system/Unix shell scripting. He has worked with global clients and in various industries, including IT, education, defense and space research, and the nonprofit sector. Follow him on Twitter, Facebook, Google+.

63 comment

  1. Ravi,

    Glancing at this I see a typo:
    sed “s/$OLD/$NEW/g” “$f” > $TFILE && mv TFILE “$f”

    Should be:
    sed “s/$OLD/$NEW/g” “$f” > $TFILE && mv $TFILE “$f”

    Also, you didn’t seem to mention that sed can auto-create backup files using syntax like this:
    sed -i.bak ‘s/old-word/new-word/g’ filename.txt

    That will edit filename.txt and copy the original to filename.txt.bak

  2. Recursively replace a word in multiple files in multiple directories?
    I couldn’t find a recursive option in sed, so …

    grep -ilr ‘old-word’ * | xargs [email protected] sed -i ‘s/old-word/new-word/g’ @

    Which will find each file with the old-word in it, complete with path, then pipe through xargs and run sed.
    Does the trick.

    1. I had to do about 5 things to make this work in OSX Snow Leopard. NOTE THAT COPYING DIRECTLY MIGHT NOT WORK. This is because the single quote characters could turn into curly quotes after posted. What I ended up with is

      grep -ilr ‘old-word’ * | xargs [email protected] sed -i ” ‘s/old-word/new-word/g’ @

      1. Curly single quotes had to become straight single quotes.
      2. Lowercase -i doesn’t appear in xargs in this OSX version, had to use uppercase -I (and spent a long time trying to figure out that @ wasn’t the problem because it didn’t have some special meaning).
      3. OSX didn’t like the missing backup suffix, it interpreted this to mean that replacement pattern was the suffix and that @ was the command. Now that I think of it, how does any OS get that the pattern isn’t the suffix? Maybe the those were accent marks that were changed into curly single quotes?

      Ok, only three things, but it felt like an eternity trying to figure it out. Actually, it was probably over an hour.

  3. Hello,
    I’m a linux newbie.
    I’m trying to change a string which is a windows path file (e.g. C:\Documents and Settings\path) into a unix path-file (e.g. /home/path). The file containing the string I want to change is a textual file (a bibtex file: file.bib). This files containes the addresses of other files, and they appear as: “C\:\\Documents and Setting\\path\\”). I’m trying with the “sed” command but apparently unix do not like the “\\”. Someone knows the solution?

  4. This wont work when I try to use it for replacing a work having “/” in it.
    I use a script to change password for other machine. But once it a while the problem comes like
    New password is “dkCHQ4/uTfSIU” old paswd “14eMXYEwUD7Cc”

    sed ‘s/14eMXYEwUD7Cc/dkCHQ4/uTfSIU’ /etc/passwd > $newpassfile
    it throws error.

    1. You can change the / from the sed option to any char you want. Try this

      sed ‘s|14eMXYEwUD7Cc|dkCHQ4/uTfSIU’ /etc/passwd > $newpassfile

      I think it’s better to understand then escape all the ‘/’s in the string like Mattheu suggested.

  5. hi,
    i need to remove the SOP in the code. My code contains a call to another function immediately after SOP call. please suggest how can i find the first occurance of ‘;’ in the line and remove the text till first occurance of ‘;’

    System.out.println(“06”); String sDebitTxnmemo= cm.getString(“********”);

  6. Hi,

    I have a template file, and would like to replace some words in that file with words taken from another file like:
    File1
    Hi $1 I miss being with you at $2
    File2
    Johan London
    So the output will be a new file contains
    Hi Johan I miss being with you at London.

    1. Try sed

      var1="Johan"
      var2="London"
      sed -e "s/\$1/$var1" -e "s/\$2/$var2" file1 > file3

      You can read data into variable using cut, sed, awk, and any other shell builtin:

      data="$(cat file2)"
      set -- $data
      var1="$1"
      var2="$2"
      sed -e "s/\$1/$var1" -e "s/\$2/$var2" file1 > file3
      
  7. Hi All,
    We had a situation to find and replace a word in huge number of files. But if we create a new file with the sed command, it may create space issue. So we are asked to remove the old file once this replacement happens and retain only the modified file.
    You may make use of the following script:
    > vi change.sh

    #!/bin/bash
    OLD="No State"
    NEW="XX"
    for f in `ls \home\Folder\FileName*.csv`
    do
       echo "processing $f `date`" >>Change.log
       sed "s/$OLD/$NEW/g" $f > 1.csv && mv 1.csv $f
       echo "processed $f `date`" >>Change.log
    done
    

    > nohup sh change.sh (run this in background)
    > tail -f Change.log (To track changes)

  8. Thank you for these useful tips. Same question as Suvankar with a twist:
    A=/path1
    B=/path2
    sed -i ‘s/$A/$B/g’ f.txt
    throws an error.
    I tried escaping with A=\/path1 to no avail.
    Any further tip for such a case? Thanks
    PS Also, what is the difference between using single and double quotes above? Thanks again.

    1. A=/path1
      B=/path2
      sed -i ‘s/$A/$B/g’ f.txt

      Should be as follows:

      A='\/path1'
      B='\/path2'
      sed -i "s/$A/$B/g" f.txt

      You need to escape / so that sed will not confuse with s/ command. See the difference between single and double quote here:

      a] Enclosing characters in single quotes preserves the literal value of each character within the quotes
      b] Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of ‘$’, ‘`’, ‘\’, and, when history expansion is enabled, ‘!’. The characters ‘$’ and ‘`’ retain their special meaning within double quotes.

  9. That works perfectly (even with spaces in the path). Thank you.
    So the single quotes in A=’\/path1′ prevent interpreting the escape character until it is used within double quotes, right?

    Now my script has matured a bit. Here is where I am at. This attempts to delete files in a tree if they are absent from another directory tree.

    ########## Try to declare path variables in various ways
    A='\/tank\/tests 1\/a'
    B='\/tank\/tests 1\/b'
    AA="/tank/tests 1/a"
    BB="/tank/tests 1/b"
    
    ########## This works, but most paths are hard-coded
    echo \# Running this file will recursively erase files and directories in  > "/tank/tests 1/b/diff.txt"
    echo \# $BB                                                               >> "/tank/tests 1/b/diff.txt"
    echo \# and not in                                                        >> "/tank/tests 1/b/diff.txt"
    echo \# $AA                                                               >> "/tank/tests 1/b/diff.txt"
    diff -r "/tank/tests 1/a" "/tank/tests 1/b" | grep "/tank/tests 1/b" >> "/tank/tests 1/b/diff.txt"  # lis differences
    sed -i "s/Only in /cd \"/" "/tank/tests 1/b/diff.txt"         # replace    Only in   with   cd "
    sed -i "s/: /\" \&\& rm -r \"/" "/tank/tests 1/b/diff.txt"    # replace    :         with   && rm -r "
    sed -i "/#/!s/$/\"/" "/tank/tests 1/b/diff.txt"               # add final  "         except for comments
    
    ######### This is one of many attempts to use soft-coded paths
    #outfile="/tank/tests 1/b/diff.txt"
    #diff -r $AA $BB | grep "$BB" >> $outfile
    #sed -i "s/Only in /cd \"/g" $outfile
    

    Any clue how to soft-code the paths? I have run out of ideas to try.
    Thanks again.

  10. I have a problem could anyone tell me a way to open the access.log file which does not have permission to read or write. I need to read that file for monitoring purpose. I cannot use chmod command to change the permission as it says permission denied. I came to know that there is a shell command to open it. It will be great help if anyone give any idea in perl or php to read that information.

    thanks in advance

  11. I could not understand the following line can u explain its functionality.

    if [ -f $f -a -r $f ]; then //explain only this one whole script are below

    #!/bin/bash
    OLD=”xyz”
    NEW=”abc”
    DPATH=”/home/you/foo/*.txt”
    BPATH=”/home/you/bakup/foo”
    TFILE=”/tmp/out.tmp.$$”
    [ ! -d $BPATH ] && mkdir -p $BPATH || :
    for f in $DPATH
    do
    if [ -f $f -a -r $f ]; then
    /bin/cp -f $f $BPATH
    sed “s/$OLD/$NEW/g” “$f” > $TFILE && mv $TFILE “$f”
    else
    echo “Error: Cannot read $f”
    fi
    done

  12. Vivek,

    I have a dictionary file:
    sme,replace
    me,place

    And on the processing file, i have:
    123456,sme,89
    678901,me,78

    Need to replace values in processing file with corresponding values from dictionary file. e.g. sme -> replace

    Would appreciate on this.

    Thanks

  13. Hi All,
    I am facing an issue wherein i need to replace a word with another one. But my case is a bit different here. Only a part of the word is constant which I know. The other half is variable ( too many possibilities ). I need to change the complete word with a new word. For example
    The possible old words may be
    abc123ad, abc123bc, abc123ak and so on….
    i need to replace any of the above in my file with a new word.

    Please help :(.
    Thanks in advance!!!

  14. BEWARE:

    If any of the files fed through xargs are binary, both sed and perl will change the timestamps of the files that all have the string, binary files or text files, but no changes are made (in at least the one file *I* was looking at at the top of the file directory hierarchy)

    So filter out binary files.

  15. Hi All,
    I want to replace a string in a file and redirect the output to a new file,but this is slightly tricky.The input file contains data as A|B|C|D|E|F|G|H|I|K , now i want to replace alphabet “F” to “:” and redirect the output file to a file.The final output must look like this:A|B|C|D|E|:|G|H|I|K. Please help it’s urgent

  16. I need help immediately. I need to monitor multiple web servers access.log file. I have completed the coding for single server completely but the problem we are facing is let us imagine two servers 192.47.155.200 and 192.47.155.36. The server 192.47.155.200 has database now how to fetch the information from 192.27.155.36 being in that server and insert in that database. I want to create a link between two servers that it does not ask for password everytime fetch the data by itself as i am running cron job. I have access to both the server as i am the admin of it. I am doing coding in php.

  17. Ok , sorry for this stupid question but someone can tell me how i can “delete” “/home/” for example i use

    echo $HOME
    /home/user_name

    i need to delete “/home/” to get “username” without “whoami” or something like that

  18. please can someone help me with the following?

    I have a text file which i have produced which is a count of how many times a pattern has occurred. This looks like this:

    network_errors 3
    vfstab_errors 2
    fs_errors 10

    I want to write a loop that extracts the results 3,2,10 and appends this result to the relevant entry in a csv file.

    the csv file looks like this:

    network_errors=1,5,8,11,4
    vfstab_errors=0,4,12,16,2
    fs_errors=3,7,1,0,0

    I want to be able to do append 3,2,10 to the entry in cvs file so in this example it would look like:
    network_errors=1,5,8,11,4,3
    vfstab_errors=0,4,12,16,2,2
    fs_errors=3,7,1,0,0,10

    Please note though the count file I am using as input will have different results each day so I want to be able to append the result to the csv file whatever the value is

    Any help would be greatly appreciated. Thanks

    1. wrong tool for the job mate, use a db. If you still want to do it like that … open file, split by line, foreach line split by space, then replace ‘(firstpart).*’ with ‘$1(,secondpart)’ in your summary file.

      Still a very bad idea as your data will not be easy to extract as it’s not even proper CSV.

      Much better but still bad would be having a correct csv file like this >
      network_errors,vfstab_errors,fs_errors
      1,0,3
      5,4,7
      8,12,1
      11,16,0
      4,2,0

      Then you can have a much simpler update script, with no replace or whatever, just get the d* out of each line of your file and add commas.

  19. Hi All

    I have got a file in dis format (tabular format in rows and columns)

    1 2 3 4 5
    aa aa aa aa aa
    bb bb bb bb bb
    cc cc cc cc cc

    I want 2 replace a particular column value with a special character
    For ex i want 2 replace column 2 first character with $
    output file:

    1 $ 3 4 5
    aa $a aa aa aa
    bb $b bb bb bb
    cc $c cc cc cc

    Thanx in advance

  20. Hi All

    I have got a file in dis format (tabular format in rows and columns)

    1 2 3 4 5
    aa aa aa aa aa
    bb bb bb bb bb
    cc cc cc cc cc

    I want 2 replace a particular column value with a special character
    For ex i want 2 replace column 2 first character with $
    output file:

    1 $ 3 4 5
    aa $a aa aa aa
    bb $b bb bb bb
    cc $c cc cc cc

    Thanx in advance

    Hi All

    I have got a file in dis format (tabular format in rows and columns)

    1 2 3 4 5
    aa aa aa aa aa
    bb bb bb bb bb
    cc cc cc cc cc

    I want 2 replace a particular column value with a special character
    For ex i want 2 replace column 2 first character with $
    output file:

    1 $ 3 4 5
    aa $a aa aa aa
    bb $b bb bb bb
    cc $c cc cc cc

    Thanx in advance

  21. Hi all!
    I have following problem: I need to replace string containing specified letters (e.g. May12) in one file, using string from another file containing same letters (May12).
    Any1 got ideas how it can be done?
    Thx in advance.

  22. Can any one help me on this..

    write a script that detects when Full GC starts in Ven file.
    Once Full GC starts,
    the script should start capturing in a separate file, about the list of objects being scanned and memory occupied by them, in docs.log or cach.log file; till Full GC completes its execution.

  23. Hello

    And thank you for putting up a nice info page for all the newbies, like me (and not only).
    In addition to all the comments, there is a problem I’m facing. I’ll simplify and try to make it clear:

    – There are a ton of text files in recursive folders.
    – All of them have lines of text (EOL doesn’t matter) — imagine a large poem.
    – Many of them have certain lines containing some strings — let’s say OLD1 and OLD2.

    What needs to be done is to search all the files for the occurence of OLD1 and OLD2, replace it with NEW1 and NEW2 but(!), in addition, for every OLD1 replaced by a NEW1 there has to be added a new line below the second containing the string BLA.

    I bet it’s confusing so I’ll give a small example:

    This is the old file:


    aaa OLD1 sss
    qqq www eee
    zzz OLD2 xxx

    This is the new file, after replacing and adding:


    aaa NEW1 sss
    qqq www eee
    BLA
    zzz NEW2 xxx

    I hope I managed to explain. If you can help me, it would be really great.
    Thank you in advance for any possible try.

  24. Hello Experts,
    Need your quick help to tune the linux script. Where i am trying to replace the text from inside the file contain. which is like below example.
    But when $NEW conatins like (‘)…. etc then i am getting some error

    ./kbsre.sh: line 3: syntax error near unexpected token `(‘
    ./kbsre.sh: line 3: `NEW=”=”(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=lnxdb-prd-208-vip.cisco.com)(PORT=1681))(CONNECT_DATA=(SERVICE_NAME=CTSPRD_SRVC_SAIB.cisco.com)(Server=Dedicated)))”‘

    But while passing simple string its getting successfully. Please refer the below linux script. Please guide me asap. Appreciate your help.
    ——————————————————————————-
    OLD=”sco.cisco.com:1546:kprd”
    #NEW=”=”(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=lnxdb-prd-208-vip.cisco.com)(PORT=1681))(CONNECT_DATA=(SERVICE_NAME=CTSPRD_SRVC_SAIB.cisco.com)(Server=Dedicated)))”
    NEW=”lnxdb-prd-208-vip.cisco.com”
    for f in `ls *.kbs`
    do
    echo “processing $f `date`” >>Change.log
    sed “s/$OLD/$NEW/g” $f > 1.kbs && mv 1.kbs $f
    echo “processed $f `date`” >>Change.log
    done
    —————————————————-

  25. Because it is zipped, you only real choice is to unzip it.
    You CAN extract a single file, make changes, and ‘update’ it, ‘replace’ it. BUT what tar REALLY does is just add it to the end. The tar program then uses ALL versions of a file for any operation, and the LAST one overwrites the others. So your archive can get REALLY large, if you go through a lot of the files.

    The recommended way is to totally extract it, make changes, re-tar/zip it.

  26. Hello,

    I have one solution, but I don’t know how to use it to process files in multithreading mode.
    gzip -dc old_file.gz | head -100 | regexpress_cut_string.pl | gzip > new_file.gz

    Each gzip file contains one text file. I have less 20 files. File size in GZIP is 1 GB, and file size in UNGZIP is 20GB.

    Can you help me?

  27. why are you choping the last 100 characters off of the file? What is you rprogram ‘regexpress_cut_string.pl’ do? This page is about replacement of strings in side of files. Is that what you are doing? About multithreading, the only two things I know about multithreading relative to this is: A/ There are several programs that do multithreading compression. If you want gzip, though, you probably have to use http://www.fsarchiver.org/Compression#Multi-threading. B/ Multithreading compressing on a laptop CAN and often DOES cause them to overheat and crash. Personal experience.

    When I backup the raw partition of my laptop and compress it, I max out the memory speed, the four cores, and the hard drive speed. It crashes in 30 to 120 minutes. I have to only use 2 of the four cores, and have recently cleated the heatsink of lint if I want multithreading to work.

  28. This is example of regular expressions which I would like to use.

    #!/usr/bin/perl
    while($file_line=)
    {
    
    	for($file_line)
    	{
    		s/(\d)(\.0+)(\]|~)/\1\3/g;
    		s/(\d\.\d*[1-9])(0+)(\]|~)/\1\3/g;
    	}
    	print STDOUT $file_line;
    }
    
  29. I have multiple files in a direcotry which contain a string like enduserchk-End User Check. Test the DSL, ATM layers from the TAM to the user.—–;
    I wont to delete this string from the files.There are around 500 files like this.Can some one tell me how this can be done using a script.This files are on aix machine

  30. I would like to replace a old line with new line, in a file using sed, can someone help here?

    old line:- /usr/bin/enaix64 -d

    new line:- /usr/bin/enaix64 -d -p /usr/bin -l 4445

    from a file test123

    I would like to use sed instead vi

    sed XXXXXXXX test123 –>like this

  31. I would need some help in parsing this string as follows:

    Variable “COLUMNS_LIST” is assigned as follows:

    COLUMNS_LIST=CD, NM, GRP_CD, GRP_NM

    Desired Output is:
    ‘CD’, ‘NM’, ‘GRP_CD’, ‘GRP_NM’

    Appreciate the help!

    Thanks

Leave a Comment