About Linux FAQ

Browse More FAQs:

Shell: How To Remove Duplicate Text Lines

Posted by Vivek Gite [Last updated: September 19, 2008]

Q. I need to sort data from a log file but there are too many duplicate lines. How do I remove all duplicate lines from a text file under GNU/Linux?

A.. You need to use shell pipes along with following two utilities:

a] sort command - sort lines of text files

b] uniq command - report or omit repeated lines

Removing Duplicate Lines With Sort, Uniq and Shell Pipes

Use the following syntax:
sort {file-name} | uniq -u
sort file.log | uniq -u

Here is a sample test file called garbage.txt:

this is a test
food that are killing you
wings of fire
we hope that the labor spent in creating this software
this is a test
unix ips as well as enjoy our blog

Type the following command to get rid of all duplicate lines:
$ sort garbage.txt | uniq -u
Sample output:

food that are killing you
unix ips as well as enjoy our blog
we hope that the labor spent in creating this software
wings of fire

Where,

  • -u : check for strict ordering, remove all duplicate lines.

E-mail    Print    Can't find an answer to your question? Contact us

Related Other Helpful FAQs:

Discussion on This FAQ

  1. Bahaa Soliman Says:

    you can use
    command: sort -u filename
    it gives you the same result

  2. Martin Says:

    How can change your example so the output would be (without duplicate lines, but one of duplicates is still there)?


    this is a test
    food that are killing you
    wings of fire
    we hope that the labor spent in creating this software
    unix ips as well as enjoy our blog

    Thank you.

  3. Deb Says:

    uniq -c will do it Martin.

  4. Jadu Saikia Says:

    One more approach keeping the order of lines same as input. The good thing about this is that it can be applied if we need to remove duplicate based on a field or fields.

    $ awk ‘!x[$0]++’ garbage.txt

    Output:
    this is a test
    food that are killing you
    wings of fire
    we hope that the labor spent in creating this software
    unix ips as well as enjoy our blog

  5. LAMP docs Says:

    What will be the command to save the output of this command to the same file? Or maybe a script?

Leave a Reply

We encourage your comments, and suggestions. But please stay on topic, be polite, and avoid spam. Thank you very much for stopping by our site!

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Tags: , , , , , , ,

Copyright © 2006-2008 nixCraft. All rights reserved - TOS/Disclaimer - Privacy policy - Powered by Open source software.