Comments on: Shell: How To Remove Duplicate Text Lines

By: Rakesh

Rakesh — Wed, 17 Aug 2011 17:01:36 +0000

Rows means dupliacte text data is comma seperated
example:
this,1,country,567,1,1,1,1
that,2,country,678,2,2,2,2
this,1,country,567,3,3,3,3

from the above data, it shoulb be check for duplicate data upto 4 values(this,1,country,567) and also redirect the particular duplicate lines into some another fil, pls help me………………

By: Rakesh

Rakesh — Wed, 17 Aug 2011 16:56:25 +0000

I want to redirect only duplicate rows into another file, anyone help me pls?

By: web designing chennai

web designing chennai — Thu, 11 Aug 2011 04:58:41 +0000

Thanks and it helped me a lot.

By: Pranab Rana

Pranab Rana — Mon, 06 Jun 2011 12:20:39 +0000

Thanks for this post.

By: stacy

stacy — Thu, 28 Oct 2010 01:27:31 +0000

just pipe the output to a new filename:

sort abc.txt | uniq > abc_1.txt

By: stacy

stacy — Thu, 28 Oct 2010 01:24:29 +0000

Also i needed both files so my command was more like this:

sort garbage.file | uniq > garbage_no_duplicates.file

By: stacy

stacy — Thu, 28 Oct 2010 01:22:13 +0000

Thanks for the help!! the uniq command took a 1372 line file down to a 32 line file, which was much less daunting!

By: Joseph

Joseph — Fri, 30 Jul 2010 20:59:42 +0000

Thanks for this. Helped me a great deal. For some reason it did leave duplicates but was still hugely helpful and removed a few duplicates manually is much better then making a 1483 line file into a 379 line file like it was supposed to be without duplicates.

By: Vivek

Vivek — Sun, 27 Jun 2010 06:36:57 +0000

How to find the duplicates alone & print in a text file? example i i have in abc.txt

abc dr545.xml
dsf fg456.xml
abc sfg34.xml

I need a text file with it’s output

abc dr545.xml
abc sfg34.xml

By: Nathan

Nathan — Tue, 08 Jun 2010 16:53:12 +0000

Thank you so much, this has helped me big time!

By: rh

rh — Mon, 31 Aug 2009 14:10:22 +0000

GOT IT:

# write dup files into text file
sort file1.txt file2.txt | uniq -d > duplicity.txt
# remove duplicated records from both files
for domain in `cat duplicity.txt`; do sed -e “/${domain}/d” -i file1.txt; sed -e “/${domain}/d” -i file2.txt; done

By: rh

rh — Mon, 31 Aug 2009 10:50:00 +0000

Does anyone know how should i remove both duplicated lines and put list of all duplicates
into text file
as there will be output

garbage.txt:
food that are killing you
wings of fire
we hope that the labor spent in creating this software
unix ips as well as enjoy our blog

garbage.duplicates.txt:
this is a test
?

By: Amber

Amber — Mon, 10 Aug 2009 06:33:05 +0000

that would be uniq followed by filename.

By: Amber

Amber — Mon, 10 Aug 2009 06:32:24 +0000

Martin 09.20.08 at 8:41 am

How can change your example so the output would be (without duplicate lines, but one of duplicates is still there)?

this is a test
food that are killing you
wings of fire
we hope that the labor spent in creating this software
unix ips as well as enjoy our blog

Thank you.
3 Deb 09.20.08 at 10:12 am

uniq -c will do it Martin.

uniq -c removes duplicates and leaves one of the lines that have been duplicated. But it also prefixes each line with the number of duplicates that have been removed.
You must use
uniq
without any of the options and your job is done.

By: Mikey

Mikey — Fri, 10 Jul 2009 00:33:13 +0000

Read the RUTE. Root users Tutorial and Exposition. This teaches you many of the basics of Linux system administration.

By: dude

dude — Sun, 05 Apr 2009 20:48:44 +0000

sorry… was going a bit too fast…
sort -u -o garbage.txt garbage.txt
Obviously =)

By: dude

dude — Sun, 05 Apr 2009 20:46:46 +0000

@lamp:
sort -u -o new_file old_file
e.g. sort -u -o moregarbage.txt garbage.txt

By: LAMP docs

LAMP docs — Wed, 03 Dec 2008 05:51:44 +0000

What will be the command to save the output of this command to the same file? Or maybe a script?

By: Jadu Saikia

Jadu Saikia — Sun, 21 Sep 2008 03:34:54 +0000

One more approach keeping the order of lines same as input. The good thing about this is that it can be applied if we need to remove duplicate based on a field or fields.

$ awk ‘!x[$0]++’ garbage.txt

Output:
this is a test
food that are killing you
wings of fire
we hope that the labor spent in creating this software
unix ips as well as enjoy our blog

By: Deb

Deb — Sat, 20 Sep 2008 10:12:50 +0000

uniq -c will do it Martin.