CentOS / RHEL CacheFS: Speed Up Network File System (NFS) File Access

by on June 4, 2012 · 9 comments· LAST UPDATED June 4, 2012

in ,

We are using NFS v4 server under RHEL 6.x+. How do I configure CacheFS for NFS under Red Hat Enterprise Linux or CentOS to speed up file access and reduce load on our NFS server?

Linux comes with CacheFS which is developed by David Howells. The Linux CacheFS currently is designed to operate on Andrew File System and Network File System. You need to install a package called cachefilesd which is a CacheFiles userspace management daemon. The cachefilesd daemon manages the caching files and directory that are used by network filesystems such a AFS and NFS to do persistent caching to the local disk.

Step #1: Install cachefilesd

Use the yum command to install cachefilesd - cacheFiles userspace management daemon:
# yum -y install cachefilesd
Sample outputs:

Loaded plugins: product-id, protectbase, rhnplugin, subscription-manager
Updating certificate-based repositories.
0 packages excluded due to repository protections
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package cachefilesd.x86_64 0:0.10.2-1.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================
 Package               Arch             Version                Repository                      Size
====================================================================================================
Installing:
 cachefilesd           x86_64           0.10.2-1.el6           rhel-x86_64-server-6            35 k
Transaction Summary
====================================================================================================
Install       1 Package(s)
Total download size: 35 k
Installed size: 0
Downloading Packages:
cachefilesd-0.10.2-1.el6.x86_64.rpm                                          |  35 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : cachefilesd-0.10.2-1.el6.x86_64                                                  1/1
Installed products updated.
Installed:
  cachefilesd.x86_64 0:0.10.2-1.el6
Complete!

Step #2: Configure cachefilesd

You need to edit /etc/cachefilesd.conf, enter:
# vi /etc/cachefilesd.conf
Sample configuration file:

dir /ssd/fscache
tag mycache
brun 10%
bcull 7%
bstop 3%
frun 10%
fcull 7%
fstop 3%
# Assuming you're using SELinux with the default security policy included in
# this package
secctx system_u:system_r:cachefiles_kernel_t:s0

Where,

  • dir /ssd/fscache - The default directory is set to /var/cache/fscache. This directory act as the root of the cache. In this example, my nfs client is mounted on RAID-1 and cache is on single ssd disk at mounted at /ssd/. The cache is good while reading files but I have way too many small files and it causes the cache opposite. Particularly on slow harddrives. So I put cache to another harddrive. If you can afford used ssd disks.
  • tag mycache This command specifies a tag to FS-Cache to use in distinguishing multiple caches. This is only required if more than one cache is going to be used. The default is "CacheFiles".
  • secctx system_u:system_r:cachefiles_kernel_t:s0 - Specify an LSM security context as which the kernel will perform operations to access the cache. The default is to use cachefilesd’s security context. If you change the directory you need to use the these instructions to set new LSM security context.
  • brun 10%, bcull 7%, bstop 3%, frun 10%, fcull 7%, fstop 3% - The cache may need culling occasionally to make space. This involves discarding objects from the cache that have been used less recently than anything else. Culling is based on the access time of data objects. Empty directories are culled if not in use. See cachefilesd.conf man page or this url for more information.

Step #3: How do I start / stop / restart cachefilesd?

Simply type the following commands:

 
## start it ##
/sbin/service cachefilesd  start
# or #
/etc/init.d/cachefilesd start
 
## stop it ##
/sbin/service cachefilesd  stop
 
## restart it ##
/sbin/service cachefilesd  restart
# or #
/etc/init.d/cachefilesd restart
 
## get status ##
/sbin/service cachefilesd  status
 

Step #4: How do I mount nfs client with CacheFS support?

You need to pass the fsc option to mount command as follows:

 
mount -t nfs -o fsc,optio2 nas01:/export/dir1/ /destination/mnt/point
 

In this example mount /var/www/html from nas042 on local /var/www/html using fsc (and other options) as follows:
# mount -t nfs4 -o rsize=32768,wsize=32768,intr,hard,proto=tcp,sync,fsc nas042:/var/www/html /var/www/html

Step #5: How do I verify that nfs client is working with CacheFS?

Cd to /ssd/fscache (or default /var/cache/fscache) directory and type the following command:
# cd /ssd/fscache
# ls -Z

Sample outputs:

drwx------. root root system_u:object_r:cachefiles_var_t:s0 cache
drwx------. root root system_u:object_r:cachefiles_var_t:s0 graveyard

When the cache is set up correctly, you will see two directories as above. You can type the following command to list files and cache size:
# find
# du -sh

Sample outputs:

142M

How do I see FS-Cache statistics?

Simply type the following command:
# cat /proc/fs/fscache/stats
Sample outputs:

FS-Cache statistics
Cookies: idx=30 dat=7895 spc=0
Objects: alc=7164 nal=0 avl=7164 ded=4261
ChkAux : non=0 ok=3727 upd=0 obs=3
Pages  : mrk=59000 unc=37195
Acquire: n=7925 nul=0 noc=0 ok=7925 nbf=0 oom=0
Lookups: n=7164 neg=3429 pos=3735 crt=3429 tmo=0
Updates: n=0 nul=0 run=0
Relinqs: n=5022 nul=0 wcr=0 rtr=22
AttrChg: n=0 ok=0 nbf=0 oom=0 run=0
Allocs : n=0 ok=0 wt=0 nbf=0 int=0
Allocs : ops=0 owt=0 abt=0
Retrvls: n=7307 ok=3324 wt=1540 nod=3243 nbf=740 int=0 oom=0
Retrvls: ops=6567 owt=1672 abt=0
Stores : n=32683 ok=32683 agn=0 nbf=0 oom=0
Stores : ops=5842 run=38525 pgs=32683 rxd=32683 olm=0
VmScan : nos=181 gon=0 bsy=0 can=0
Ops    : pend=1695 run=12409 enq=38525 can=0 rej=0
Ops    : dfr=21 rel=12409 gc=21
CacheOp: alo=0 luo=0 luc=0 gro=0
CacheOp: upo=0 dro=0 pto=0 atc=0 syn=0
CacheOp: rap=0 ras=0 alp=0 als=0 wrp=0 ucp=0 dsp=0

You can see the caching state of individual mountpoints can be examined through other /proc files.
# cat /proc/fs/nfsfs/servers
Sample outputs:

NV SERVER   PORT USE HOSTNAME
v4 0a0a1d44  801   1 nas042

# cat /proc/fs/nfsfs/volumes
Sample outputs:

NV SERVER   PORT DEV     FSID              FSC
v4 0a0a1d44  801 0:20    147f55ffe88547d2  yes

Note down the output of the "FSC" column. It always says "yes" when the system has been asked to cache a particular NFS, and "no" when it hasn't.

How do I test CacheFS?

Copy files from NFS server to local hard drive:

 
time cp /path/to/nfs/mnt/point/bigfile.gz /tmp
 
#### this should speed up as bigfile.gz is in cache now #####
time cp /path/to/nfs/mnt/point/bigfile.gz /dev/null
 

In this example copy data.tar.bz2 (78MB) from nfs to /tmp (init cache):
$ time cp data.tar.bz2 /tmp
Sample outputs:

real	0m7.023s
user	0m0.000s
sys	0m0.185s

Copy data.tar.bz2 to /dev/null from NFS again (i.e. get it from cache):
$ time cp data.tar.bz2 /dev/null
Sample outputs:

real	0m0.027s
user	0m0.000s
sys	0m0.026s

See also:

  1. FS-Cache: A Network Filesystem Caching Facility by David Howells (PDF)
  2. man pages - mount, cachefilesd(8), cachefilesd.conf(5)
TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 9 comments… read them below or add one }

1 Amol June 4, 2012 at 5:30 pm

Vivek,

Your website is probably one of the most useful sites out on the internet for linux commands. Keep it up.

Reply

2 Sanjith June 15, 2012 at 9:52 am

I ran the command..
#yum install cachfilesd
but got the error
“No package cachefilesd available. Nothing to do”

Reply

3 Wayne July 19, 2013 at 4:12 pm

When I looked for it in CENTOS5.9 I couldn’t find it either. It is available in RHEL6/CENTOS6.

Reply

4 jbd June 20, 2012 at 2:58 pm

You should drop the multiple caches between your two run of cp and post the updated results.

# echo 3 > /proc/sys/vm/drop_caches
# time cp data.tar.bz2 /dev/null
# echo 3 > /proc/sys/vm/drop_caches
# time cp data.tar.bz2 /dev/null

Reply

5 time_eco July 11, 2013 at 8:26 am

As told , I have did the run second time copy. The OS will reboot itself. Anybody faced the same problem in Centos 6.4 ?

# echo 3 > /proc/sys/vm/drop_caches
# time cp data.tar.bz2 /dev/null
# echo 3 > /proc/sys/vm/drop_caches
# time cp data.tar.bz2 /dev/null

Reply

6 Supermasita October 17, 2012 at 9:35 pm

Is there a rpm for CentOS 5.6?

Ive tried compiling and i get the following error:

[root@server src]# cachefilesd -s -d -n -f /etc/cachefilesd.conf
Unable to open /dev/cachefiles: errno 2 (No such file or directory)

Thanks!!

Reply

7 jakommo April 4, 2013 at 7:10 pm

Had the same error on Centos6.4 after istalling the rpm with yum.
Turns out the module wasn’t loaded. You can try:
# modprobe cachefiles
# lsmod |grep cachefiles
cachefiles 36906 1
fscache 53874 2 cachefiles,nfs
# service cachefilesd start

after this it worked.

Reply

8 time_eco July 25, 2013 at 9:54 am

Using FS-Cache as a caching proxy server for Samba ? Did anyone tried ?

Reply

9 Fan February 24, 2014 at 9:17 pm

Anyone know how to tune the size of the cache file? My default cache file’s size is always 12K (/var/cache/fscache/cache) and I don’t know how to change it. Another question is in this tutorial, when comparing the $ time cp data.tar.bz2 /tmp and $ time cp data.tar.bz2 /dev/null, how do you know you are actually using the fs-cache and not using linux default caching? When I run trace on( strace -T -tt cp data.tar.bz2/tmp) I found the most of time is waiting for close. Is there any thing wrong with my testing?

Reply

Leave a Comment

Tagged as: , , , , , , , , , , , , , , , , ,

Previous Faq:

Next Faq: