Linux fdupes: Get Rid (Delete) Of Double Duplicate Files In Directory

by nixCraft on February 3, 2010 · 13 comments· last updated at February 7, 2010

How do I find duplicate files in a given set of directories and delete them using a shell script or a command line options? How do I get rid of double duplicates files stored in ~/foo and /u2/foo directory?

You need to use a tool called fdupes. It will searche the given path for duplicate files. Such files are found by comparing file sizes and MD5 signatures, followed by a byte-by-byte comparison. fdupes is a nice tool to get rid of duplicate files.

Install fdupes

Type the following command under Debian / Ubuntu Linux:
# apt-get install fdupes
Type the following command under Redhat / RHEL / Fedota / CentOS Linux, enter (turn on rpmforge repo before running the following yum command):
# yum install fdupes

How Do I Use fdupes?

Find duplicate files in /etc/ directory, enter:
# fdupes /etc
Sample outputs:

/etc/vimrc
/etc/virc

How Do I Delete Unwanted Files?

You can force fdupes to prompt you for files to preserve, deleting all others (use this with care otherwise you may loss data): # fdupes -d /etc
Sample outputs:

[1] /etc/vimrc
[2] /etc/virc
Set 1 of 1, preserve files [1 - 2, all]: 1
   [+] /etc/vimrc
   [-] /etc/virc

How Do Recursively Search Directory?

You can recursively search every directory given follow subdirectories encountered within the -r option, enter:
# fdupes -r /dir1

How Do I Find Dupes In Two Directories?

Type the command as follows:
# fdupes /dir1 /dir2
OR
# fdupes -r /etc /data/etc /nas95/etc

How Do I See Size Of Duplicate Files?

Type the following command with the -S option:
# fdupes -S /etc
Sample outputs:

1533 bytes each:
/etc/vimrc
/etc/virc

Featured Articles:

{ 13 comments… read them below or add one }

1 jason February 3, 2010 at 3:05 pm

cool. thx

2 Tech Observer February 4, 2010 at 5:40 pm

Thanks forgot that one, there’s always a guarantee that Debian will make it easier.
Do you know an equivalent with a GUI front end on Ubuntu?

Tech Observer

3 Vivek Gite February 4, 2010 at 6:08 pm: I don’t think so,
apt-cache search dupes
fdupes – identifies duplicate files within given directories
findimagedupes – Finds visually similar or duplicate images
Reply

4 Philippe Petrinko February 6, 2010 at 1:58 pm

  Hi Vivek
    Nice topic,
    Best regards,
–Philippe

5 Wido February 6, 2010 at 5:35 pm

Cool tool but, does it takes consideration in hard links? they are the same file, no a dup, but with different name

6 Name February 6, 2010 at 9:46 pm

I use fdupes with sed to output a shell file that will delete unwanted duplicates:
fdupes -r -n -S /directory | sed -r “s/^/#rm \”/” | sed -r “s/$/\”/” >duplicate-files.sh

The shell file created by this has each line commented out. It’s just a matter of uncommenting the files you want to delete. I don’t recall where I first saw this idea – credit to original poster though.

7 Rajagopal February 7, 2010 at 11:29 am

I think the “yum install fdupes” is missing in the post

Great post, as usual.

Regards,

Rajagopal

8 Vivek Gite February 7, 2010 at 12:41 pm

@Rajagopal,

Thanks for the heads up!

9 Vivek Gite February 7, 2010 at 6:33 pm

@Wido,

Yes, take a look at man page you will get the option to control soft and hard links.

10 gio February 12, 2010 at 2:36 pm

I’ve read about a similar tool, included in Ubuntu distro, with a GUI front end whose name is

fslint

11 荒野无灯 May 4, 2010 at 7:38 am

hello, happy to see your article.
I am intrested in what tools you are using to create PDF for posts .
could you tell ?
“Download PDF version”

12 haasdas February 29, 2012 at 9:08 pm

Try FSlint maybe. It has a GUI if you insist.

Hope it helps

13 James March 15, 2012 at 5:14 pm

I’m runnung CentOS 5 with fdupes v1.4. fdupes runs fine if i’m checking for duplicates on the Linux box itself. But I need to check for duplicates on an NFS mounted drive. When I run the command nothing seems to happen other than I get the command line again. I’m typing fdupes -r /dir1 /dir2
My nfs mount was mounted with the follwoing command mount -t nfs -o tcp ipaddress:/ifs /dupeChk

Thanks in advance for any help.