agedu: Unix / Linux Command For Tracking Down Wasted Disk Space

The most sysadmin will run low on disk space or storage space. Users will demand more disk space, and you need to free storage space. You will find out files that are a waste of space and delete it or move to an archive medium. However, how do you find the right files to delete that can help recover maximum space? Say hello to agedu tool (pronounced as ‘age dee you’) – it scans a directory tree and produces reports about how much disk space is used in each directory and subdirectory, and also how that usage of disk space corresponds to files with last-access times a long time ago. In other words, this command might help you to free up disk space.

du vs agedu

The du command summarize disk usage of each file, recursively for directories. This tool is like du, but unlike du, it also distinguishes between large collections of data which are still in use and ones which have not been accessed for months or years – for instance, extensive archives downloaded, unpacked, used once, and never cleaned up. Where du helps you find what’s using your disk space, agedu helps you find what’s wasting your disk space.

How does agedu works?

From the man page:

Most Unix file systems, in their default mode, helpfully record when a file was last accessed. Not just when it was written or modified, but when it was even read. So if you generated a large amount of data years ago, forgot to clean it up, and have never used it since, then it ought in principle to be possible to use those last-access time stamps to tell the difference between that and a large amount of data you’re still using regularly.

agedu is a program which does this. It does basically the same sort of disk scan as du, but it also records the last-access times of everything it scans. Then it builds an index that lets it efficiently generate reports giving a summary of the results for each subdirectory, and then it produces those reports on demand.

In this post you will learn how to install agedu on Linux, MacOS X, FreeBSD, and Unix-like system to track down wasted disk space usage.

Install agedu

Deiban / Ubuntu Linux user type the following apt-get command to install agedu:
# apt-get install agedu
Sample outputs:

Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  agedu
0 upgraded, 1 newly installed, 0 to remove and 13 not upgraded.
Need to get 46.4 kB of archives.
After this operation, 131 kB of additional disk space will be used.
Get:1 http://debian.osuosl.org/debian/ squeeze/main agedu amd64 8928-1 [46.4 kB]
Fetched 46.4 kB in 1s (29.8 kB/s)                  
Selecting previously deselected package agedu.
(Reading database ... 274216 files and directories currently installed.)
Unpacking agedu (from .../agedu_8928-1_amd64.deb) ...
Processing triggers for man-db ...
Setting up agedu (8928-1) ...

FreeBSD unix user can use the port as follows:
# cd /usr/ports/sysutils/agedu/
# make install clean
OR alternatively use the binary package provided by FreeBSD:
# pkg install agedu
RHEL / CentOS / Fedora / Scientific Linux user turn on EPEL repo and type the following yum command to install agedu:
# yum install agedu
Sample outupts:

Loaded plugins: auto-update-debuginfo, product-id, protectbase, rhnplugin,
              : subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
0 packages excluded due to repository protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package agedu.x86_64 0:0-2.r9153.el6 will be installed
--> Finished Dependency Resolution
 
Dependencies Resolved
 
================================================================================
 Package         Arch             Version                  Repository      Size
================================================================================
Installing:
 agedu           x86_64           0-2.r9153.el6            epel            47 k
 
Transaction Summary
================================================================================
Install       1 Package(s)
 
Total download size: 47 k
Installed size: 83 k
Is this ok [y/N]: y
Downloading Packages:
agedu-0-2.r9153.el6.x86_64.rpm                           |  47 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : agedu-0-2.r9153.el6.x86_64                                   1/1 
Installed products updated.
  Verifying  : agedu-0-2.r9153.el6.x86_64                                   1/1 
 
Installed:
  agedu.x86_64 0:0-2.r9153.el6                                                  
 
Complete!

Arch Linux users, type the following command to install agedu:
$ sudo yaourt -S agedu
MacOS X users, type the following command to install agedu:
$ brew install agedu
Sample outputs:

==> Downloading https://homebrew.bintray.com/bottles/agedu-20160920.sierra.bottl
Already downloaded: /Users/veryv/Library/Caches/Homebrew/agedu-20160920.sierra.bottle.tar.gz
==> Pouring agedu-20160920.sierra.bottle.tar.gz
🍺  /usr/local/Cellar/agedu/20160920: 5 files, 119.1K

How do I use agedu command?

First, you need to scan your disk and build an index file containing a special data structure, enter:
$ agedu -s /home/vivek
$ sudo agedu -s /var
$ agedu -s /nas05
Sample outputs:

Built pathname index, 103484 entries, 8642180 bytes of index                                                                                                 
Faking directory atimes
Building index
Final index file size = 23654856 bytes

In above examples, I started agedu by telling it to do a scan of a directory tree and build an index for /home/vivek, /var, and /nas05 directories. Next, logical step is to query the index by typing the following command:
$ agedu -w
Sample outputs:

Using Linux /proc/net magic authentication
URL: http://127.0.0.1:42113/

Fire up a graphical web browser, type the following url:
http://127.0.0.1:42113/
Sample outputs:

Fig.01: agedu report

Username: agedu
Password: 696cv6r297upqzmt
URL: http://192.168.1.5:9000/

agedu will normally make up a username and password for you. But, you can set your own user name and password:
$ agedu -w --address 192.168.1.5:9000 --auth basic --auth-fd 0
Sample outputs:

vivek:cAnd1Bar

The authentication details should consist of the username (vivek), followed by a colon (:), followed by the password (cAnd1Bar), followed immediately by end of file (press [CTRL]+[D]).

Can I access agedu reports using terminal mode?

Type the following command (replace /home/wwwroot with actual path):
$ agedu -t /home/wwwroot
Sample outputs:

53569312    /home/wwwroot
30427672    /home/wwwroot/logs
83997004    /home/wwwroot/images

You will get a summary of the disk usage in /home/wwwroot and its subdirectories. The output is much the same format as du command. To see how much old data is there, pass the the -a option to show only files last accessed a certain length of time ago. For example, to see only files which haven’t been looked at in twelve months or more:
$ agedu -t /home/wwwroot -a 12m
Sample outputs:

2220        /home/wwwroot
2236        /home

How do I see only the disk space taken by MP3 files or .AVI files?

Run the command as follows in current directory:
$ agedu -s . --exclude '*' --include '*.mp3'
$ agedu -s . --exclude '*' --include '*.avi'
To view reports, run:
$ agedu -w
The reports and first two command will cause everything to be omitted from the scan, but then the MP3 files to be put back in. If you then wanted only a subset of those MP3s, you could then exclude some of them again by adding, say, –exclude-path ‘./steviewonder/*’
$ agedu -s ~/Downloads/drm-free/music --exclude '*' --include '*.mp3' --exclude-path './steviewonder/*'

How do I delete files and reclaim disk space again?

In my case /home/vivek/iso-images/ was taking too much space. I found out older vmware-workstation, Linux / Unix iso and binary files. I deleted all those old unwanted files and recovered 16.5GB disk space using nothing but simple rm command:
$ rm ~/iso-images/vmware*
$ rm ~/iso-images/centos-4* ~/iso-images/centos-5*

How do I remove agedu index file?

Use the ls command to see the size of agedu index file:
$ ls agedu.dat -lh
Sample outputs:

-rw------- 1 vivek vivek 23M Oct 21 01:55 agedu.dat

To remove index file, enter:
$ agedu -R
OR
$ rm agedu.dat
However, you can also put -R on the end of a command line to indicate that agedu should delete its index file after it finishes performing other operations such as displaying web pages:
$ agedu -w -R

This blog post gives you the quick tour of what agedu does. This command has many more options for complex situations that comes with usual array of unix command line options. So, I recommend that you read the man page for more information or visit the project home page to grab the latest source code:
$ man agedu

Related posts:

• Unix / Linux: See colourised filesystem disk space usage.
• Debian/Ubuntu Linux remove unwanted packages and files to reclaim disk space/a>.
• MySQL bin files eating lots of disk space.

If you’ve got a favorite command or hack I didn’t mention, let us know about it in the comments below.

Editor’s note: Due to some xml markup error you may missed our previous blog entry in rss reader/email news letter. This entry is available at – Howto: Setup Linux hard disk encryption with LUKS page.