≡ Menu

rescue mode

Linux / Unix: Disable OpenSSH Host Key Checking

I've a remote Unix server running with OpenSSH remote login service. The openssh is configured for passwordless login using ssh keys. Our ISP allows to boot all Linux servers into the rescue mode. It allow us to bring a server online remotely in order to troubleshoot system problems that would normally only be resolved by an OS Reload (such as accidentally deleting files or wrong firewall configurations blocking ssh access). When server boots into a remote rescue mode I can connect using SSH. They SSH keys will not be the same in the rescue mode so I get key mismatch messages as SSH keys are re-generated on each boot:

Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
Please contact your system administrator.
Add correct host key in /home/user/.ssh/known_hosts to get rid of this message.
Offending key in /home/user/.ssh/known_hosts:1
RSA host key for www.cyberciti.biz has changed and you have requested strict checking.
Host key verification failed.

How do I ignore OpenSSH hos key checking from my Apple OS X laptop while login using the ssh?
[click to continue…]

Linux LVM error VG meta data inconsistent and Solution

Q. How do I fix the following error under CentOS Enterprise Linux:

LVM error VG meta data inconsistent

How do I fix this problem?

A. You need to use lvm command, which provides the command-line tools for LVM2. In order to fix this problem boot your Linux into rescue mode or a single user mode:

First, you need to restore the VG Meta data, enter:
lvm vgcfgrestore VG

Now, you need to rescan for physical devices and vol groups (scan all disks for physical volume), enter:
lvm pvscan
lvm vgscan

Finally, make the volume group active, enter:
lvm vgchange -ay VG

For more information read lvm man page:
man lvm

What do I do if my Linux Server / Workstation does not boot?

Q. I'm using Debian Linux and CentOS operating system. My question is what do I do if my server refused to boot? How do I troubleshoot the problem?

A. Usually most Linux distribution provides rescue mode. It allows to access to disk partition and you are allowed to solve the problem.

To use the rescue mode of the install CD, follow these steps:
a) Boot from 1st CD / DVD media

b) Enter rescue at the prompt. (use 'dd rescue' if you need to load the SCSI driver)

c) You will get at a shell prompt. Now you can reinstall boot loader or check disk for errors and so on.

Please note that Linux distributions such as Mandriva or Ubuntu Linux offers menu based troubleshooting options. See our previous articles for detailed instructions:

Boot Ubuntu Linux into Rescue mode to fix system

Q. How do I boot my Ubuntu Linux server into Rescue mode to fix system?

A. You can boot Ubuntu Linux into rescue mode when things go wrong, there are several ways to work on fixing them. However, you must understand your system and what to fix in case of emergency.

Use this mode only if your system is unbootable. It is also recommended that you make backup of important data.

=> Start Computer / reboot computer

=> Put your Ubuntu install CD

=> Make sure BIOS is set to boot from CD

=> Now booting will start from Ubuntu install CD

=> You will see various option screen - select Recover a broken system

You will system is booting in to rescue mode; system will ask about Network and other information. This is required to setup rescue environment.

Now you will prompted to select root partition or device.

Next it will try to mount root device and it will notify the same.

Now you will see common rescue mode option such as Reinstall GRUB or open / execute a shell etc.

Now you are in rescue mode and you need to use your own knowledge to rescue the system. When things go wrong, there are several ways to work on fixing them.

Few things to remember....

The rescue-mode CD mounts your root filesystem at /target directory. Open or execute a shell and change directory to /target
# ls /target
# cd /target
# mount

If you need to run fsck command use fdisk -l command to find out partition name. Usually they are located at /dev/discs/discX/partY. For example repair 2nd partition (not mounted as /target) you may type command:
# ls -l /dev/discs/disc0
# fsck.ext3 /dev/discs/disc0/part1

To exit type command :
# sync ;sync
# exit
# reboot

As I said earlier in rescue mode, there is no single command that I can recommend to fix problem. You need to use your own knowledge to rescue the system and don't forget to read command documentation before using any command.

Can I run fsck or e2fsck when Linux file system is mounted?

Q. Can I run fsck or e2fsck command when Linux file system is mounted? Do you advice to run fsck on a live file system? I am using Cent OS.

A. No. Do not run fsck on a live or mounted file system. fsck is used to check and optionally repair one or more Linux file systems. Running fsck on a mounted filesystem can usually result in disk / data corruption. So please do not do it. You have two choices
(a) Take down system to single user mode and unmout system

(b) Boot from the installation CD into rescue mode

(a) Take down system to single user mode and unmout system

Following are steps :
=> Use init (process control initialization ) command to change runlevel 1 (singe user mode)
=> Use umount command to unmount /home file system
=> Run fsck using fsck command

Let us say you would like to run fsck on /home (/dev/sda3):

# init 1
# umount /home
# umount /dev/sda

# fsck /homeOR
# fsck /dev/sda3OR
# e2fsck -y /dev/sda3

(b) Boot from the installation CD into rescue mode

If you are using Cent OS/Fedora Core/RHEL linux, boot from first CD and at boot prompt type linux rescue nomount:
boot: linux rescue nomount

Now make new node for disk and partition 3:
# mknod /dev/sda
# mknod /dev/sda3
# fsck /dev/sda3

# e2fsck -y /dev/sda3

Don't forget to reboot the system:
# exit;exit
# reboot

Linux reset forgotten root password

Q. I forgot my root password, how can I get into my system?

A. You can reset forgotten root password under Linux by booting system into single user mode or emergency mode (also known as rescue mode).

My boot loader is GRUB (see LILO boot loader below)

)Following is the procedure to reset root password if you are using GRUB as a boot loader:

  1. Select the kernel
  2. Press the e key to edit the entry
  3. Select second line (the line starting with the word kernel)
  4. Press the e key to edit kernel entry so that you can append single user mode
  5. Append the letter S (or word Single) to the end of the (kernel) line
  6. Press ENTER key
  7. Now press the b key to boot the Linux kernel into single user mode
  8. At prompt type passwd command to reset password:

You need to mount at least / and other partitions:
# mount -t proc proc /proc
# mount -o remount,rw /

Change the root password, enter:
# passwd
Finally reboot system:
# sync
# reboot

My boot loader is LILO

At LILO boot loader type linux single and press [ENTER] key:
Boot: linux single
When you get the # prompt you will need to type passwd root to reset password:
# passwd
Reboot system:
# sync
# reboot

See also: