Linux: Recovering deleted /etc/shadow password file

Sometime by accident you may delete /etc/shadow file. If you boot into single user mode, system will ask root password for maintenance, and just imagine you do not have a backup of /etc/shadow file. How do you fix such problem in a production environment where time is critical factor? I will explain how to recover deleted /etc/shadow file in five easy steps. It will take around 10 min. to fix the problem.

Well all it started when one of our client accidentally deleted /etc/shadow file from co-located Debian Linux server. As a result, all account login disabled. However, ftp was working fine because proftpd was build using MySQL database for authentication and quota management.

Boot server into single user mode

1) Reboot server

2) Next, you will see grub-boot loader screen. Select Recovery mode the version of the kernel that you wish to boot and type e for edit. Select the line that starts with kernel and type e to edit the line.

3) Go to the end of the line and type init=/bin/bash as a separate one word (press the spacebar and then type init=/bin/bash). Press enter key to exit edit mode.
init=/bin/bash

4) Back at the GRUB screen, type b to boot into single user mode. This causes the system to boot the kernel and run /bin/bash instead of its standard init. This will allow us gain root privileges (w/o password) and a root shell.

Make sure you can access system partition(s)

1) Mount partitions in read write mode
Since / is currently mounted read-only and many disk partitions have not been mounted yet, you must do the following to have a reasonably functioning system.
# mount -rw -o remount /
Do not forget to (re)mount your rest of all your partitions in read/write (rw) mode such as /usr /var etc (if any)

Rebuild /etc/shadow file from /etc/passwd

1) You need to use pwconv command; it creates /etc/shadow from /etc/passwd and an optionally existing shadow.
# pwconv

2) Use passwd command to change root user password:
# passwd

Note you may need to type same password twice with passwd command. If you have an admin account, then setup password for that account. On most production, servers direct root login is disabled. In our situation, admin was the only account allowed to use su and sudo command.
# passwd admin

3) Now root and admin accounts are ready to go in multi-user mode. Reboot the system in full multiuser mode:
# sync
# reboot

Step # 4 Block all non-root login

Block all non-root (normal) users until we fix all password related problems. Since rest of account do not have any password, it is necessary to prevent non-root users from logging into the system. You need to create /etc/nologin file, it will allow access only to root. Other users will be shown the contents of this file and their logins will denied (refused)

1) Login as root user (terminal login only)

2) Create /etc/nologin file
cat > /etc/nologin
System is down due to temporary problem. We will restore your access
within 30 minutes time. If you have any questions please contact tech
support at XXX-XXXX or techsupport@mycorp.com

Update all users password in batch mode

1) Create random password for each non-root user using chpasswd utility. It update passwords in batch mode. chpasswd reads a list of user name and password pairs from file and uses this information to update a group of existing users. Each line is of the format:

user_name:password

Remember by default the supplied password must be in clear-text format. This command is intended to be used in a large system environment where many accounts are created at a single time or in emergency like this. First, we need to find out all non-root accounts using awk command:
awk -F: '{ if ( $3 >1000 ) print $1}' /etc/passwd > /root/tmp.pass

Make sure /root/tmp.pass file contains non-root usernames only.

2) Create random password with pwgen
By default, pwgen utility is not installed so with the help of apt-get install it:
# apt-get install pwgen

The pwgen program generates passwords which are designed to be easily memorized by humans, while being as secure as possible. For example following command print the generated password:
# pwgen -1 -n 8

Download complete working script that updates user password in batch mode. Execute script batch-update-password.sh:
# chmod +x batch-update-password.sh
# ./batch-update-password.sh

Now update user passwords with chpasswd, by default script creates file in /root/batch.passwd file:
# chpasswd

3) Email new password to server admin or all end users. You can write a script to email password end users.

4) Your system is ready to accept login, just remove /etc/nologin file:
# rm /etc/nologin

There are other ways to recover /etc/shadow file, depend upon your setup and backup frequency you can use any one of the following method too:

• By default, your /etc/passwd and /etc/shadow file are backup to /var/backups under Debian Linux. You can just copy shadow.bak file after step # 1:

# cp /var/backups/shadow.bak /etc/shadow

• Some time /etc/shadow- file can be use to replace /etc/shadow
• If you have a backup of /etc/shadow on tape or cdrom then you can copy back /etc/shadow file after step #1.

I guess it explains the important of regular backup of both data and key files.

See also:

• Understanding UNIX/Linux file system
• Linux : How to delete file securely
• Linux/UNIX: Recover deleted files

Featured Articles:

• 20 Linux System Monitoring Tools Every SysAdmin Should Know
• 20 Linux Server Hardening Security Tips
• Linux: 20 Iptables Examples For New SysAdmins
• My 10 UNIX Command Line Mistakes
• 25 PHP Security Best Practices For Sys Admins
• The Novice Guide To Buying A Linux Laptop
• Top 5 Email Client For Linux, Mac OS X, and Windows Users
• Top 20 OpenSSH Server Best Security Practices
• Top 10 Open Source Web-Based Project Management Software