NFS Stale File Handle error and solution

by on October 9, 2006 · 33 comments· LAST UPDATED May 14, 2008

in , ,

Sometime NFS can result in to weird problems. For example NFS mounted directories sometimes contain stale file handles. If you run command such as ls or vi you will see an error:
$ ls
.: Stale File Handle

First let us try to understand the concept of Stale File Handle. Managing NFS and NIS, 2nd Edition book defines filehandles as follows (a good book if you would like to master NFS and NIS):
A filehandle becomes stale whenever the file or directory referenced by the handle is removed by another host, while your client still holds an active reference to the object. A typical example occurs when the current directory of a process, running on your client, is removed on the server (either by a process running on the server or on another client).

So this can occur if the directory is modified on the NFS server, but the directories modification time is not updated.

How do I fix this problem?

a) The best solution is to remount directory from the NFS client using mount command:
# umount -f /mnt/local
# mount -t nfs nfsserver:/path/to/share /mnt/local

First command (umount) forcefully unmount a disk partition /mnt/local (NFS).

(b) Or try to mount NFS directory with the noac option. However I don't recommend using noac option because of performance issue and Checking files on NFS filesystem referenced by file descriptors (i.e. the fcntl and ioctl families of functions) may lead to inconsistent result due to the lack of consistency check in kernel even if noac is used.

TwitterFacebookGoogle+PDF versionFound an error/typo on this page? Help us!

{ 33 comments… read them below or add one }

1 Colin Brace January 2, 2008 at 10:46 pm

I encounter these errors on my Ubuntu client when I reboot my Fedora server with the NFS shares. I find I need to first umount the shares on the client, then restart nsf on the server, then remount the shares on the client. I don’t claim to understand the logic behind this — but it works.

Reply

2 Sander Deryckere January 29, 2009 at 8:16 am

I had this error on my home laptop, so it had nothing to do with servers. it was when I tried t install songbird on my computer, something went wrong and when I wanted to reinstall songbird my computer gave this error, I’ve tried to reboot, to delete the /usr/share/songbird directory but nothing worked. Finally I let it like it was and just ran the scripts while they weren’t in place (right from my home folder) and now, a few weeks later the problem hes resolved himself and I could reinstall songbird without problems. If you could say how this happened please let me know.

Reply

3 Daemeon January 29, 2009 at 5:11 pm

When a client mounts a server’s exported NFS mount point to the client’s specific mount point, client and server negotiate a unique id for that event (i.e. the id is unique for the mount, mount point, server, server exported file system, etc.), there is therefore a new unique ID for every successful mount request. All communications between the client and the server include this unique ID. When a server reboots, the server (intentionally) will have no record of the unique ID, hence the client will get an error when it tries to access the remote file system. The stale file system error is telling you that this has occurred.

Unmounting (force) and remounting from the client will resolve this problem IF there are no other references within the client that are retaining the handle/ID. I.e. this is a problem for processes that are running and have open file handles when you do the force unmount. When you force unmount/remount and still get the error, you have some program that is probably (?) not following proper file handle semantics about closing files.

However, watch out for a related problem, in autofs I have seen timeout options. These can cause a stale NFS handle in some combinations of NFS (version 2 also) client to server communications, and processes that have (correctly) open file handles.

So, a stale NFS handle occurring on a client after a server reboot, resolved on the client by an un/remount of the client’s file system is proper behaviour.

Reply

4 artila February 16, 2009 at 10:51 am

I just bot laptop with Limpus Linux an it and the first time I started gave me this notice
fale NFS Stale File Handle error : I dont have much experience with linux how can I solve the problem , or instal windows xp (when I tried it gave mi the similar error like stoped to not damage sistem)?

Reply

5 brephophagist May 16, 2009 at 1:54 am

Usually I find some operation that will force the inode tables on client and server to update de-zombifies stale NFS data. Something as simple as remaking the mount point directory on the client (seen that solve some really weird stuff) or renaming a directory on the server and forcing the client to try to use it, then renaming it back. This is all assuming simple umount / mount and/or kill / restart stuff doesn’t work.

*cough*network failure system*cough*

Reply

6 Dot2Kode January 30, 2010 at 3:56 am

Nice thanks for the tip…Simple 2 commands to fix it..Made a little bash script for them…thanks again.

Reply

7 John J April 1, 2010 at 4:12 pm

using rpc.statd is better as it constantly checks the state of the nfs system and keeps the mount status up to date.

Reply

8 Rafael August 24, 2010 at 5:20 pm

If you have the fstab entry, you can simply umount and then mount again using the mount point. Something like this:
umount -f /mnt/local
mount /mnt/local

Reply

9 stillinschool September 15, 2010 at 3:51 pm

I am running slax. I had the same problem after install a module. I uninstalled the module and the problem was gone

Reply

10 DL October 14, 2010 at 8:22 pm

ls -alh in the offending directory regularly clears this problem up for me.

ls -alh /mnt/volume/directory won’t do it
ls -alh /mnt/volume/directory/ will.

Reply

11 chandra October 28, 2010 at 6:10 pm

Done! in one shot as umount -f mount point then mounted it

Cheers!

Reply

12 chandra October 28, 2010 at 6:15 pm

I was getting below error….
mount.nfs: Stale NFS file handle

#umount -f /mount point
# mount -t nfs nfs:server /mount point
#

cheers!

Reply

13 Pravinkumar Chavan February 9, 2011 at 11:18 am

I have this problem on my system running on windows. I am running an embedded program loaded on chip and not at all connected to server. when I will do ls it shows the error “Stale NFS file handle”. I am using Putty / hyperterminal . The solution mention above not working for me. please help me out………..

Thanking you in advanced.

Reply

14 pravinkumar chavan February 12, 2011 at 9:02 am

I have my file system stored at hda2 partation which is of type ext2 and getting folling error
mount /dev/hda2 /mnt/hda2
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
#
# cd /mnt/hda2
#
# ls
ls: ./^J#: Stale NFS file handle
ls: ./AppInfo: Stale NFS file handle
ls: ./C: Stale NFS file handle

Settings selinux
alsa-mae-init launchup sys
b# lib test_snd
bin linuxrc tmp
config_init lost+found tmp_bac
data mnt
dev opt usr
etc proc var
i root
init sbin

Reply

15 Nirmal Pathak July 29, 2011 at 12:37 pm

# umount -l /mnt/point

# mount nfs:/path/mount /mnt/point

Above worked for me!

I guess this lazy umount option should be added in article. :)

Reply

16 Leonid August 5, 2011 at 1:04 pm

Only this key help also for my!
Thank you very much!

Reply

17 Pradeep Kumar December 7, 2013 at 7:48 am

You saved my day Nirmal, I was not able to umount with -f until i saw your solution. -l worked perfectly for me.. Thanks Nirmal again:)

Reply

18 Naveen August 22, 2011 at 9:08 am

Thanks a lot. My work is done with ur help.

Reply

19 Pete September 13, 2011 at 4:05 pm

The umount/mount fix is a *really* big hammer. Besides, it only fixes the symptoms, it doesn’t fix the disease. Hasn’t anyone written a fix so that when the Stale condition is detected, the filehandle/dirhandle in question is refreshed (made non-Stale)? Seems like a simple enough fix. We get the Stale handles between RedHat Ent. and Tru64, sometimes in a matter of seconds of making the mount. Neither file system has been restarted. A umount/mount will fix it for a short time, but it returns quickly. Using the NOAC option significantly reduces the frequency of occurrence, but also significantly reduces write speed to the mounted files, even within one open/write/close session, the write is greatly slowed. This is especially curious, since the file attributes should be immaterial during the writes, IMO. Our biggest problem is that we can’t pay a trained monkey to sit around watching for the Stale handle incidents and do the umount/mount 24/7. Besides the umount will affect other running processes. It’s hardly a win. Does anyone know of better solutions? Has anyone else had success with the `ls -alh’ fix?

Reply

20 Alex Atkin UK November 6, 2011 at 2:23 pm

Perhaps I am missing something, but I have never ONCE been able to get the above to work.

I always get stuck with:
# umount -f /mnt/home-ext
umount2: Device or resource busy
umount.nfs: /mnt/home-ext: device is busy

Linux can be so stupid sometimes, I mean how can it be busy if the mount is stale and so by its nature NOTHING is able to access it?

Reply

21 Zsombor January 23, 2012 at 7:49 am

A process – maybe your own bash shell – has /mnt/home-ext as the current directory.
lsof |grep /mnt/home-ext shows what uses – or wants to use – that dir.

Reply

22 Morala November 30, 2011 at 4:20 pm

Always good tips

Reply

23 Jozef December 30, 2011 at 1:51 pm

Can this error occure when NFS has short network outage ?

Reply

24 Alejandro Sanchez-Flores March 20, 2014 at 2:53 am

Hi Jozef,

I really don’t know but I’m having a problem with a server which directories are mounted in a different server. My guess is like you said, a problem with the network, but so far, haven’t found the way of testing it.

Did you have a similar problem or did you get any answer to your question?

Cheers.

Reply

25 Tommi July 1, 2012 at 7:55 pm

Thanks to Nirmal Pathak. -f was not enough…

Reply

26 passingby August 4, 2012 at 5:56 pm

if you are running nautilus file manager, you’ll probably find the problem is nautilus, not NFS at all. try “killall nautilus” from any shell prompt. works for me, so far..

Reply

27 Davinken August 31, 2012 at 10:08 pm

This problem is haunting me in newer Fedora 17 installs. Autofs works fine, but is not timing out the mounts after the resource is left. So they seem to stay alive until something as the remote host being rebooted happens.
Then the stale mount thing…
But after forcing the umount (which seems to fix things partially, no more stale mount alerts), now I get:
Too many levels of symbolic links

Any hint on this ?
Thanks

Reply

28 Nicola September 12, 2012 at 11:54 am

I got same issue (mount.nfs: Stale NFS file handle) the first time I’ve attempted to mount a shared folder.
I dont really have anything to umount of in busy state

Any idea appreciated

thx
Nicola

Reply

29 NoFingers September 30, 2013 at 10:02 am

The `ls -alh’ always works fine with me. I just apply it at the parent directory of the one causing the error. After that, I can access to the directory/file without any problem.

Reply

30 Paul Freeman January 6, 2014 at 10:17 am

Try:

exportfs -f

on the server first, if that does not work then on the client try :

mount -o remount  /path

if that fails with device is busy/in-use, find the offending processes with:

fuser -fvm /path

and retry remount

Reply

31 Rhys February 1, 2014 at 9:01 am

I found that i could not umount -f /path always getting the stale message.
when i looked at my server i could see that in the exportfs -av that the ip address listed was not what i was connected on anymore. Looking at my router i found that i had a Dynamic DHCP address. I added and a reservastion for my MAC address on my old IP address and reconnected my wireless then mounted and everything worked as before.

Reply

32 Amit March 6, 2014 at 4:54 am

I had this problem and I was sure that files/directory are still available at host.
So, I just back traced one directory and came back on current directory, everything was working then.

Reply

33 Anthony March 20, 2014 at 4:49 pm

Thank you. My Yum was locking up and when I did trace it was a stale nfs.

strace yum -y update {package}

I unmounted and remounted and it worked again. Explains why yum and server issues.

Reply

Leave a Comment

Previous post:

Next post: