Howto: Linux command line utilities for removing blank lines from text files

by on September 13, 2006 · 30 comments· LAST UPDATED August 10, 2008

in , ,

Q. I want to change the formatting of a file. I just wanted to remove all blank lines from text file. How do I achieve this task w/o spending much time?

A. Yes, you do not have to waste your time making manual changes to files. Both Linux and UNIX systems come with file manipulation tools that can be used to remove all blank lines very quickly.

Task: Remove blank lines using sed

Type the following command:
$ sed '/^$/d' input.txt > output.txt

Task: Remove blank lines using grep

$ grep -v '^$' input.txt > output.txt

Both grep and sed use special pattern ^$ that matchs the blank lines. Grep -v option means print all lines except blank line.

Let us say directory /home/me/data/*.txt has all text file. Use following for loop (shell script) to remove all blank lines from all files stored in /home/me/data directory:

#!/bin/sh
files="/home/me/data/*.txt"
for i in $files
do
  sed '/^$/d' $i > $i.out
  mv  $i.out $i
done
 

Updated for accuracy.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 30 comments… read them below or add one }

1 James Herr March 13, 2008 at 5:08 pm

Is that last shell script wrong? The for loop is using variable f, and inside the loop, everything is using variable i. I think the script should look more like:

#!/bin/sh
files=”/home/me/data/*.txt”
# Next line is changed to use variable i, not f
for i in $files
do
sed ‘/^$/d’ $i > $i.out
mv $i.out $i
done

Reply

2 anicka November 28, 2011 at 1:58 pm

Hello!

Little fix in this BASH script:

#!/bin/bash
#!/bin/sh
files=”/home/tiger/test_moduls/*.txt”
for i in $files
do
sed ‘/^ *$/d’ $i > $i.out
mv $i.out $i
done

Good Luck! :-)

Reply

3 Deepak August 10, 2008 at 8:40 am

its nice..appreciable

Reply

4 nixCraft August 10, 2008 at 9:00 am

James,

Thanks for the heads up. The faq has been updated.

Reply

5 Rick October 28, 2008 at 5:37 pm

When writing to the same file, or a different file and using the redirect > the file ends up blank.

This removes blank lines from input.txt

sed ‘/^$/d’ -i input.txt

Reply

6 Prashant Deshani November 26, 2008 at 1:19 pm

Hi,

I tried to use the command sed given above to remove blank line from my file.

The file looks good except one strange behaviour:
“Always” the last line of the file is removed..for example:
test.p is the file and following are the contents of the file:

BEFORE RUNNING THE SED COMMAND:
“this is test 1.

this is test 2.

this is test 3.”

AFTER RUNNING THE SED COMMAND:
“this is test 1.
this is test 2.”

As you can see the last line (“this is test 3.”) is removed after running the command.

If I put the blank line at the end it removes the blank line i.e.
“this is test 1.

this is test 2.

this is test 3.

becomes…….
“this is test 1.
this is test 2.
this is test 3.”

Do let me know what I should do in this case?

Waiting for your prompt reply…!

Thanks and Best Regards,
Prashant Deshani

Reply

7 Dennis January 28, 2009 at 2:34 pm

Actually, grep’s -v (–invert-match) option inverts the sense of matching, thus selecting non-matching lines.
Therefore, only when combining it with the pattern ‘^$’ does it mean to select everything except blank lines.
Just thought the way you explained it might be confusing for beginners. When I was new to shell and Linux, most of the guides or forums were useless because they all assumed that the person that needs help has some high understanding of shell or Linux already, and thus they use a lot of terms that person would have never heard of before or they don’t explain anything. Once you have a basic understanding, the learning comes easier, but in the beginning, the learning curve has an almost horizontal slope.

Reply

8 REDDY SIVA SARAN April 15, 2009 at 10:46 am

Hi,
I am Siva Saran
I am new to linux environment
can any on tell plz that how to get a particular line from a file using linux commands
For example say that inputfile has lines as follows
total 0
d [RWCEAFMS] MRohit 512 Apr 15 03:15 3.1_SP1_Beta1_127
d [RWCEAFMS] MRohit 512 Apr 14 03:26 3.1_SP1_125 d [RWCEAFMS] MRohit 512 Apr 14 03:26 3.1_SP1_259
d [RWCEAFMS] MRohit 512 Apr 14 03:26 3.0_SP4_IR3_62

Now I want the line which is having SP1_125 ie., line number 3 and store it in an variable
plz can any tell the answer plz
can u mail to {snip_no_email_ids}

Reply

9 Renz June 18, 2009 at 6:22 am

what if there’s spaces and tab in the line I want to delete?

thanks,
Renz

Reply

10 Angwe August 26, 2010 at 2:44 pm

This is old, but in case no-one told you:

\w is the special regular expression for whitespace. Both the sed and grep matches should be changed to:

/^\w*$/

-Angwe

Reply

11 Niju N B July 13, 2009 at 11:47 am

Hi,

Using the command
grep -R “SP1_125″
you can get the lines with SP1_125.
if you want to assign to a variable then use
var_a=$(grep -R “SP1_125″ )

where is the name of file

Reply

12 Niju N B July 13, 2009 at 11:49 am

var_a = $(grep -R “SP1_125″ <filename>)

Reply

13 Niju N B July 13, 2009 at 11:52 am

Hi,
var_a=$(grep -R “SP1_125″ filename)

Reply

14 ezz November 4, 2009 at 4:48 pm

This is quite helpful. Thanks!!

Reply

15 sobi3ch April 16, 2010 at 7:16 am

how count all lines in all files in current dir and omit empty lines with wc, grep, cut and bc commands

echo `wc -l * | grep total | cut -f2 -d’ ‘` – `grep -in “^$” * | wc -l ` | bc

Reply

16 Walter April 16, 2011 at 7:03 pm

@sobi3ch
LOL, wut? Wanna win an obfuscation contest?

grep -v '^\w*$' * | wc -l

would be better, because it doesn’t count lines with only whitespaces, and uses only two instead of seven(!) programs.

for f in *; do awk '!/^\w*$/ && !/^\w*#/ {x++} END {print FILENAME" :"x" lines"}' $f; done

will print you the number of lines, excluding empty lines and lines starting with “#” for every file separately in the current directory.

Reply

17 Simplesol May 13, 2011 at 6:59 pm

All you are trying to do is this:

grep -v “^$” * | wc -l

Reply

18 Walter April 16, 2011 at 7:34 pm

The example should read

sed -i '/^\w*$/d' input.txt
grep -v '^\w*$' input.txt > output.txt && mv output.txt input.txt

and

#!/bin/sh
files="/home/me/data/*.txt"
for i in $files; do
    sed -i '/^\w*$/d' $i
done

Because
a) using “^\w*$” instead of “^*$” removes also lines consisting only of whitespaces
b) sed <em-i (=in-place) will edit the files directly, so that there is no need for output redirection and file renaming.

If you have an non-GNU sed version, which doesn’t support in-place editing, than at least change the sed/mv lines to

sed '/^\w*$/d' $i > $i.out && mv $i.out $i

otherwise you could end up with a lot of empty files, if the sed command goes wrong.

Reply

19 Walter April 16, 2011 at 7:35 pm

“sed <em-i" should read sed i

Reply

20 Walter April 16, 2011 at 8:14 pm

“sed <em-i" should read "sed -i

Reply

21 pj July 13, 2011 at 6:44 am

I would like to know how i can write a shell script to delete “datetime message” and a particular patern exists? in short I want my file to this following output.

### original file ###
datetime message
2011-07-13 13:45:35 Hello World of War Craft
2011-07-13 13:48:43 This is a Test of text

### output file ###
20110713134535 | Hello World of War Craft
20110713134843 | This is a Test of text

thanks

Reply

22 Tryme July 13, 2011 at 7:30 am

Type

echo '2011-07-13 13:45:35 Hello World of War Craft'  | sed -e 's/-//g' -e 's/://g' | awk '{ print $1$2 " | " substr($0, index($0,$3))}'

Result:

20110713134535 | Hello World of War Craft

You need to process the file using while loop:

while IFS= read -r line
do
   echo "$line" | sed -e 's/-//g' -e 's/://g' | awk '{ print $1$2 " | " substr($0, index($0,$3))}'
done < "input.txt"

There may be a better solution to reduced awk, but I’m too lazy to try it out ;)

Reply

23 pj July 13, 2011 at 7:48 am

Thanks for fast reply,

when I run this shell script
# while IFS= read -r line; do echo “$line” | sed -e ‘s/-//g’ -e ‘s/://g’ | awk ‘{ print $1$2 ” | ” substr($0, index($0,$3))}’; done testdb1.pl;

the file output is below: the line 1 not remove and added datetimemessage, is there any way to remove the line 1 which is “datetimemessage | datetime message”

datetimemessage | datetime message
20110713134535 | Hello World of War Craft
20110713134843 | This is a Test of text

Reply

24 pj July 13, 2011 at 7:50 am

while IFS= read -r line; do echo “$line” | sed -e ‘s/-//g’ -e ‘s/://g’ | awk ‘{ print $1$2 ” | ” substr($0, index($0,$3))}’; done testdb1.pl;

Reply

25 pj July 13, 2011 at 7:51 am

correction
‘; done testdb.pl “>testdb.pl”

Reply

26 pj July 13, 2011 at 10:45 am

Hi all,

I don’t if is the right to post this issue,
is there any way to dump the new insert data to a textfile with shell script, for ie. after insertion to the mysql database execute the shell script to append and append the new record?

thanks

Reply

27 rceny December 11, 2011 at 7:02 am

Hi all,

Is there a way to add an empty line for every 5 lines of data using grep?

line1
line2
line3
line4
line5
(empty line)
line6
line7
line8
line9
line10
(empty line)

Reply

28 wordpress March 13, 2012 at 8:00 am

you can also do cat output.txt | awk ‘NF’ >> output.txt

Reply

29 Maciej March 19, 2013 at 9:24 am

grep -v “^[[:blank:]]*$”

Reply

30 reza June 14, 2013 at 7:15 am

i have a text file that in end of file have a blank line but i cant delet in by sed i have to vi it and use dd command to delet it

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , , , , , , , ,

Previous Faq:

Next Faq: