Lab 11 System Rescue and Troubleshooting

Goal: To build skills in system rescue procedures.

Estimated Duration: 2 hours

Lab Setup: When using rescue mode on a Xen enabled system, you may be prompted to select which root partition to mount. If you are prompted in this lab to select a root partition, choose the logical volume, not the /dev/{s,h}daX partition (which is the root used by your Xen domU).

Sequence 1: Repairing the MBR in the rescue environment

Scenario: The rescue environment provides a last resort for repairing an unbootable machine, even when the bootloader or the root filesystem is damaged or misconfigured. In order to access the rescue environment, you will need either a boot.iso cdrom on a network that has the Red Hat installation tree (the Server directory) available via NFS, or a Red Hat Enterprise Linux CDROM.

Instructions:

1. Use the following command to overwrite the first stage of GRUB in your Master Boot Record with zeros. Specify the block size carefully. If you write too many zeros, you will overwrite your partition table as well, and this will become a much more difficult exercise. (Note that the command below assumes you are using IDE drives. You might need to modify the destination device.)

After typing the following command, check it three times and hit Enter but once.

# dd if=/dev/zero of=/dev/hda bs=256 count=1 && reboot

You have just wiped out your boot sector, but your primary partition table will still be intact. Attempt a reboot to confirm that your system is unbootable. Use the Red Hat rescue environment to repair the system.

a. Load the rescue environment by booting from a Red Hat installation media (either CDROM or PXE), and typing linux rescue at the boot prompt. Proceed with the normal installation defaults. Choose NFS image for the media type and use the following NFS information:

• NFS server : server1.example.com

• NFS directory : /var/ftp/pub

b. The rescue environment will ask if you wish to mount the hard drive's filesystems. Select Continue to mount the filesystems in read-write mode. Examine the output of mount to confirm that the filesystem was correctly reconstructed. You might want to refresh your memory by examining your disk's partitions with fdisk.

c. Note that your hard drive has been reconstructed under the mount point /mnt/sysimage. Examine grub.conf (on your hard drive) to confirm that it is appropriately configured.

# cat /mnt/sysimage/boot/grub/grub.conf

d. To reinstall GRUB, you must shift contexts, so that grub-install believes that the root of your filesystem is the /mnt/sysimage directory. Spawn a chrooted shell, run grub-install, and then exit.

# chroot /mnt/sysimage

# grub-install /dev/hda

# exit

(Or, should the above fail to execute properly)

Exit the chroot environment and then type the command: grub at the bash prompt. This will place you into grub's command shell where you can enter the following commands:

grub> root (hd0,0)

grub> setup (hd0)

grub> quit

e. Now exit your rescue shell. Note that the rescue environment will unmount any partitions that you mounted. Eject the CD.

Sequence 2: Installing software in rescue mode

Instructions:

1. Use the following command to overwrite the mount command.

# cp /bin/date /bin/mount

You have just wiped out a key executable on your system. Upon attempting a reboot, you should find your system unbootable. Use the Red Hat rescue environment, along with its version of the rpm command and the library of RPMs provided by the installation tree, to repair the system. Hint: /bin/mount is part of the util-linux RPM.

a. Load the rescue environment as in the previous exercise.

b. The rescue environment will attempt to automatically mount the hard drive's filesystems. Examine the output of mount to confirm that the filesystem was correctly reconstructed.

c. Discover what package the /bin/mount command comes from and then verify that util-linux package on your hard drive, all using a chrooted invocation of rpm. Do not forget to exit the chroot or the rpm installation will fail.

# chroot /mnt/sysimage

# rpm -qf /bin/mount

# rpm -V util-linux

# exit

d. rpm should report that the /bin/mount executable has been modified. Reinstall the util-linux RPM from your installation tree (which has been NFS mounted under /mnt/source).

# cd /mnt/source/Server

# rpm -ivh --force --root /mnt/sysimage util-linux*

Note that the util-linux package was installed (the hash marks indicate this), although you may see some errors at the end of the process. As it turns out, these are harmless errors, although in a production environment, you would want to test this out fully.

e. Now exit your rescue shell. Note that the rescue environment will unmount any partitions that you mounted.

Sequence 3: Troubleshooting Practice

System Setup: 1. Turn off iptables and mount the /var/ftp/pub directory from server1 if it is not currently mounted.

service iptables stop

chkconfig iptables off

mkdir /mnt/server1

mount server1:/var/ftp/pub /mnt/server1

2. Install the Troubleshooting Practice RPM:

rpm -ihv /mnt/server1/gls/RPMS/rhce-ts-*

3. Ensure that your computer is configured as closely as possible to the following specifications:

• Authenticate users from your local /etc/passwd file. That is, do not run any network authentication scheme such as NIS or LDAP.

• Use 192.168.0.254 (server1.example.com) as your name server.

• Confirm that /usr/local/bin is part of your PATH environment variable.

The network problems assume that you remember your network information. Your network information is as follows:

IP: 192.168.0.X

SUBNET: 255.255.255.0

GATEWAY: 192.168.0.254

DNS: 192.168.0.254

The following items are required for some, but not all, troubleshooting problems. You may still do most problems if some of these items are missing.

• Change to runlevel 3, not runlevel 5. Confirm that the X server is not running (no startx). Only the local problems require this.

• Confirm that /home is a separate filesystem from the root filesystem and is local to the system (not an NFS mounted filesystem).

Instructions:

1. The Troubleshooting Practice problems come in three parts, each invoked by a separate command. The sections, commands, and number of problems in each section vary; therefore, run command count , to determine the number of problems for each

troubleshooting command:

• For Local: tslocal count

• For Networking: tsnetwork count

• For Booting: tsboot count

2. Invoke the first local problem by running:

# tslocal 1

This command will set up the problem and will explain the goal. The goal will be stored in the file /etc/ts for later reference. Spend three to eight minutes trying to solve the problem.

3. If you have not yet solved the problem, you may need a hint. Hints can be displayed by

running the tshint command:

# tshint local 1 1

This will display the first hint for the first tslocal problem. Continue to invoke hints until you get enough information to solve the problem or until you run out of hints:

tshint local 1 2

tshint local 1 3

[ and so on ...]

The tshint command will tell you when you have reached the end of the hints. Again, do not spend more than five to ten additional minutes on this problem.

If a problem requires that you attempt to resolve it in rescue mode, to access the hints, you will need to add a directory to the path:

export PATH=$PATH:/mnt/sysimage/user/local/bin

4. Whether or not you have solved the problem, run the tslesson command:

# tslesson local 1

This command will tell the lessons intended to be taught by the problem. Some tslesson messages also give step-by-step instructions on how to approach a particular problem.

5. If, after reading the hints and the lesson, you are unable to solve the problem, call the instructor for assistance.

6. Proceed with the remaining problems in the same way. For example, tsnetwork 1 sets up the first network problem and tshint network 1 1 shows the first hint for the first network problem.