Linux User does not exist – weird problem and solution

Recently I received a support request as follows:
After adding a user using useradd command:
useradd -s /bin/false -d foo
passwd foo

Following error reported
user foo does not exist

However, after sometime user foo allowed to login. What is going on? Is my system hacked?

First I went through system password database located at /etc/passwd, /etc/shadow, /etc/group – there exists an entry for foo user.

After close investigation I found name service cache daemon (nscd). It is a daemon that provides a cache for the most common name service requests including caching of /etc/passwd file (thanks for top and lsof command for hint).

So the solution is to stop the nscd service add user and start nscd again. This time it worked.
# /etc/init.d/nscd stop
# useradd -s /bin/false -d bar
# passwd bar
# /etc/init.d/nscd start


Now bar is allowed to login immediately :).

Refer to nscd man page for detailed information (hint setup correct TTL to get rid of this problem and upgrade to latest version):
man nscd
man nscd.conf