We are using NFS v4 server under RHEL 6.x+. How do I configure CacheFS for NFS under Red Hat Enterprise Linux or CentOS to speed up file access and reduce load on our NFS server?

Linux comes with CacheFS which is developed by David Howells. The Linux CacheFS currently is designed to operate on Andrew File System and Network File System. You need to install a package called cachefilesd which is a CacheFiles userspace management daemon. The cachefilesd daemon manages the caching files and directory that are used by network filesystems such a AFS and NFS to do persistent caching to the local disk.

Step #1: Install cachefilesd

Use the yum command to install cachefilesd – cacheFiles userspace management daemon:
# yum -y install cachefilesd
Sample outputs:

Loaded plugins: product-id, protectbase, rhnplugin, subscription-manager
Updating certificate-based repositories.
0 packages excluded due to repository protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package cachefilesd.x86_64 0:0.10.2-1.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

 Package               Arch             Version                Repository                      Size
 cachefilesd           x86_64           0.10.2-1.el6           rhel-x86_64-server-6            35 k

Transaction Summary
Install       1 Package(s)

Total download size: 35 k
Installed size: 0  
Downloading Packages:
cachefilesd-0.10.2-1.el6.x86_64.rpm                                          |  35 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : cachefilesd-0.10.2-1.el6.x86_64                                                  1/1 
Installed products updated.

  cachefilesd.x86_64 0:0.10.2-1.el6                                                                 


Step #2: Configure cachefilesd

You need to edit /etc/cachefilesd.conf, enter:
# vi /etc/cachefilesd.conf
Sample configuration file:

dir /ssd/fscache
tag mycache
brun 10%
bcull 7%
bstop 3%
frun 10%
fcull 7%
fstop 3%

# Assuming you're using SELinux with the default security policy included in
# this package
secctx system_u:system_r:cachefiles_kernel_t:s0


  • dir /ssd/fscache – The default directory is set to /var/cache/fscache. This directory act as the root of the cache. In this example, my nfs client is mounted on RAID-1 and cache is on single ssd disk at mounted at /ssd/. The cache is good while reading files but I have way too many small files and it causes the cache opposite. Particularly on slow harddrives. So I put cache to another harddrive. If you can afford used ssd disks.
  • tag mycache This command specifies a tag to FS-Cache to use in distinguishing multiple caches. This is only required if more than one cache is going to be used. The default is “CacheFiles”.
  • secctx system_u:system_r:cachefiles_kernel_t:s0 – Specify an LSM security context as which the kernel will perform operations to access the cache. The default is to use cachefilesd’s security context. If you change the directory you need to use the these instructions to set new LSM security context.
  • brun 10%, bcull 7%, bstop 3%, frun 10%, fcull 7%, fstop 3% – The cache may need culling occasionally to make space. This involves discarding objects from the cache that have been used less recently than anything else. Culling is based on the access time of data objects. Empty directories are culled if not in use. See cachefilesd.conf man page or this url for more information.

Step #3: How do I start / stop / restart cachefilesd?

Simply type the following commands:

## start it ##
/sbin/service cachefilesd  start
# or #
/etc/init.d/cachefilesd start
## stop it ##
/sbin/service cachefilesd  stop
## restart it ##
/sbin/service cachefilesd  restart
# or #
/etc/init.d/cachefilesd restart
## get status ##
/sbin/service cachefilesd  status

Step #4: How do I mount nfs client with CacheFS support?

You need to pass the fsc option to mount command as follows:

mount -t nfs -o fsc,optio2 nas01:/export/dir1/ /destination/mnt/point

In this example mount /var/www/html from nas042 on local /var/www/html using fsc (and other options) as follows:
# mount -t nfs4 -o rsize=32768,wsize=32768,intr,hard,proto=tcp,sync,fsc nas042:/var/www/html /var/www/html

Step #5: How do I verify that nfs client is working with CacheFS?

Cd to /ssd/fscache (or default /var/cache/fscache) directory and type the following command:
# cd /ssd/fscache
# ls -Z

Sample outputs:

drwx------. root root system_u:object_r:cachefiles_var_t:s0 cache
drwx------. root root system_u:object_r:cachefiles_var_t:s0 graveyard

When the cache is set up correctly, you will see two directories as above. You can type the following command to list files and cache size:
# find
# du -sh

Sample outputs:


How do I see FS-Cache statistics?

Simply type the following command:
# cat /proc/fs/fscache/stats
Sample outputs:

FS-Cache statistics
Cookies: idx=30 dat=7895 spc=0
Objects: alc=7164 nal=0 avl=7164 ded=4261
ChkAux : non=0 ok=3727 upd=0 obs=3
Pages  : mrk=59000 unc=37195
Acquire: n=7925 nul=0 noc=0 ok=7925 nbf=0 oom=0
Lookups: n=7164 neg=3429 pos=3735 crt=3429 tmo=0
Updates: n=0 nul=0 run=0
Relinqs: n=5022 nul=0 wcr=0 rtr=22
AttrChg: n=0 ok=0 nbf=0 oom=0 run=0
Allocs : n=0 ok=0 wt=0 nbf=0 int=0
Allocs : ops=0 owt=0 abt=0
Retrvls: n=7307 ok=3324 wt=1540 nod=3243 nbf=740 int=0 oom=0
Retrvls: ops=6567 owt=1672 abt=0
Stores : n=32683 ok=32683 agn=0 nbf=0 oom=0
Stores : ops=5842 run=38525 pgs=32683 rxd=32683 olm=0
VmScan : nos=181 gon=0 bsy=0 can=0
Ops    : pend=1695 run=12409 enq=38525 can=0 rej=0
Ops    : dfr=21 rel=12409 gc=21
CacheOp: alo=0 luo=0 luc=0 gro=0
CacheOp: upo=0 dro=0 pto=0 atc=0 syn=0
CacheOp: rap=0 ras=0 alp=0 als=0 wrp=0 ucp=0 dsp=0

You can see the caching state of individual mountpoints can be examined through other /proc files.
# cat /proc/fs/nfsfs/servers
Sample outputs:

v4 0a0a1d44  801   1 nas042

# cat /proc/fs/nfsfs/volumes
Sample outputs:

NV SERVER   PORT DEV     FSID              FSC
v4 0a0a1d44  801 0:20    147f55ffe88547d2  yes

Note down the output of the “FSC” column. It always says “yes” when the system has been asked to cache a particular NFS, and “no” when it hasn’t.

How do I test CacheFS?

Copy files from NFS server to local hard drive:

time cp /path/to/nfs/mnt/point/bigfile.gz /tmp
#### this should speed up as bigfile.gz is in cache now #####
time cp /path/to/nfs/mnt/point/bigfile.gz /dev/null

In this example copy data.tar.bz2 (78MB) from nfs to /tmp (init cache):
$ time cp data.tar.bz2 /tmp
Sample outputs:

real	0m7.023s
user	0m0.000s
sys	0m0.185s

Copy data.tar.bz2 to /dev/null from NFS again (i.e. get it from cache):
$ time cp data.tar.bz2 /dev/null
Sample outputs:

real	0m0.027s
user	0m0.000s
sys	0m0.026s

See also:

  1. FS-Cache: A Network Filesystem Caching Facility by David Howells (PDF)
  2. man pages – mount, cachefilesd(8), cachefilesd.conf(5)

🐧 Get the latest tutorials on Linux, Open Source & DevOps via RSS feed or Weekly email newsletter.

🐧 12 comments so far... add one

CategoryList of Unix and Linux commands
Disk space analyzersdf duf ncdu pydf
File Managementcat cp mkdir tree
FirewallAlpine Awall CentOS 8 OpenSUSE RHEL 8 Ubuntu 16.04 Ubuntu 18.04 Ubuntu 20.04
Modern utilitiesbat exa
Network UtilitiesNetHogs dig host ip nmap
OpenVPNCentOS 7 CentOS 8 Debian 10 Debian 8/9 Ubuntu 18.04 Ubuntu 20.04
Package Managerapk apt
Processes Managementbg chroot cron disown fg glances gtop jobs killall kill pidof pstree pwdx time vtop
Searchingag grep whereis which
User Informationgroups id lastcomm last lid/libuser-lid logname members users whoami who w
WireGuard VPNAlpine CentOS 8 Debian 10 Firewall Ubuntu 20.04
12 comments… add one
  • Amol Jun 4, 2012 @ 17:30


    Your website is probably one of the most useful sites out on the internet for linux commands. Keep it up.

  • Sanjith Jun 15, 2012 @ 9:52

    I ran the command..
    #yum install cachfilesd
    but got the error
    “No package cachefilesd available. Nothing to do”

    • Wayne Jul 19, 2013 @ 16:12

      When I looked for it in CENTOS5.9 I couldn’t find it either. It is available in RHEL6/CENTOS6.

  • jbd Jun 20, 2012 @ 14:58

    You should drop the multiple caches between your two run of cp and post the updated results.

    # echo 3 > /proc/sys/vm/drop_caches
    # time cp data.tar.bz2 /dev/null
    # echo 3 > /proc/sys/vm/drop_caches
    # time cp data.tar.bz2 /dev/null

    • time_eco Jul 11, 2013 @ 8:26

      As told , I have did the run second time copy. The OS will reboot itself. Anybody faced the same problem in Centos 6.4 ?

      # echo 3 > /proc/sys/vm/drop_caches
      # time cp data.tar.bz2 /dev/null
      # echo 3 > /proc/sys/vm/drop_caches
      # time cp data.tar.bz2 /dev/null

  • Supermasita Oct 17, 2012 @ 21:35

    Is there a rpm for CentOS 5.6?

    Ive tried compiling and i get the following error:

    [root@server src]# cachefilesd -s -d -n -f /etc/cachefilesd.conf
    Unable to open /dev/cachefiles: errno 2 (No such file or directory)


    • jakommo Apr 4, 2013 @ 19:10

      Had the same error on Centos6.4 after istalling the rpm with yum.
      Turns out the module wasn’t loaded. You can try:
      # modprobe cachefiles
      # lsmod |grep cachefiles
      cachefiles 36906 1
      fscache 53874 2 cachefiles,nfs
      # service cachefilesd start

      after this it worked.

      • time_eco Jul 25, 2013 @ 9:54

        Using FS-Cache as a caching proxy server for Samba ? Did anyone tried ?

  • Fan Feb 24, 2014 @ 21:17

    Anyone know how to tune the size of the cache file? My default cache file’s size is always 12K (/var/cache/fscache/cache) and I don’t know how to change it. Another question is in this tutorial, when comparing the $ time cp data.tar.bz2 /tmp and $ time cp data.tar.bz2 /dev/null, how do you know you are actually using the fs-cache and not using linux default caching? When I run trace on( strace -T -tt cp data.tar.bz2/tmp) I found the most of time is waiting for close. Is there any thing wrong with my testing?

  • sushil Jul 7, 2016 @ 10:38

    How to clear the fscache if it is causing full disk issue,without losing any data?

    • Chris Nov 7, 2016 @ 15:05

      Did you ever get an answer to this? I’ve got a disk that is 90% full due to cachefilesd filling up.

  • fabian Sep 14, 2016 @ 19:53

    thanks for this very nice tutorial.
    Is it possible to use the ram for caching?
    I have around 500MB of data and want to use a very fast cache.
    I’ll use it in a virtualised system via nfs harddisks (For High Availability)

Leave a Reply

Your email address will not be published.

Use HTML <pre>...</pre> for code samples. Still have questions? Post it on our forum