Grub on a mirrored boot partition

Grub on a mirrored boot partition

The purpose of this document is to suggest a way of configuring grub on a system where the operating system is mirrored between two disks, so that if the first disk fails completely, the system will boot automatically from the second disk.

Whether this actually works depends on how precisely the system fails, so there are also instructions on how to get the system up if it still isn't booting. Note that not everything in this document has been fully tested, and the grub documentation suggests there are machines on which they definitely won't work.

Setup

The assumption is that we have two identically partitioned disks, containing a mirrored boot partition ^[1], and we will be booting from the master boot record on each disk ^[2]. The examples assume that the first partition on each disk make up the mirrored disk mounted at /boot, and that grub numbers these disks (hd0) and (hd1). Grub numbers disks in order of detection by the bios. The order is probably the same as the operating system, but in some cases the ordering may not be obvious, for example if you have a mixture of IDE and SCSI disks.

First make sure that the BIOS setup of your machine will attempt to boot from the second disk if it can't find a valid boot sector on the first disk. Run the grub command from your root prompt and within the grub shell type the following (Note that the install commands may be broken for legibility but should be entered as one long line):

Here the --stage2 value is the location of the grub stage2 file within the operating system, which means it writes the changes to the mirrored partition, rather than one or the other raw disk, and hence avoids corrupting the mirror. The other file references are in grub format, ie. a partition reference followed by the location of the file within that partition. We don't specify an explicit partition for the grub.conf file as the "p" flag tells grub to remember which partition it found the stage2 file on and make it the default, and we therefore avoid setting the disk explicitly and use the current disk, which is presumably the good half of the mirror. If we didn't have a separate /boot partition, all the file references would start /boot/grub rather than /grub.

The grub.conf file (sometimes called menu.lst) in /boot/grub/ should have no "root" commands, at least for the default target, and there should be no explicit partition references, since we are trying to use relative references. For example:

Hopefully this means the system will boot even if the first disk is corrupted or removed, though it may not if the problem means that the system can read some, but not all for the boot files.

If you upgrade your linux distribution or your grub package directly, you will probably need to redo these instructions (your distribution may try to update the grub setup, but the chances are it won't do it this way). You may also have to fix your grub.conf file after upgrading the kernel.

Picking up the pieces

Depending on the problem, the system may still not boot, or only partially boot (eg. as far as a grub prompt), in which case some manual intervention is required. If there is a grub prompt we can use it, otherwise we can get a grub prompt by booting from a floppy disk with grub installed^[3]. All we need to do here is to select the good boot partition with the root command, and give the kernel and initrd lines as in a normal boot eg.:

However, you may have forgotten which boot partition to use and what arguments you need. In this case the grub commands find, cat and configfile are useful, and grub also has tab completion of files and partitions, which can be used to work out which disk is which. Use find /grub/grub.conf to work out which partition(s) has a valid copy. Set this partition to be the active one with root (hd1,0). Then you can view the configuration file with cat /grub/grub.conf and enter the commands by hand, or use configfile /grub/grub.conf to load it. Note you may have to edit a menu item by adding a root command to get the system to run from it.

If you have any unmirrored partitions on the failed disk, the system is likely not to boot fully when it looks for the missing partition, though commenting out the appropriate line in /etc/fstab should work if the disk isn't vital.

Footnotes:

1. Actually we don't need to be quite this strict. The partition containing the boot software must be in the same place on the two disks, since the raw disk layout information contained in the shared stage2 file must be the same^[4]. Also any unmirrored swap partitions should be in corresponding partitions, and all other disks must be raided or mounted via the LABEL= format, or else the system could get confused when the first disk is pulled out, as the the second will be renamed (eg. from /dev/hdb to /dev/hda).
2. The problem with using the boot sector of the boot partition is that it is still within the mirror, and whereas the boot sectors can be identical, setting it up safely can be difficult as you are liable to end up writing directly to one of the disk rather than to the mirror, which a bad idea. The safest way to do this is to break the mirror, set up the boot sector, and recreate the mirror afterwards^[5].
3. To make a grub boot disk, from a running linux system, insert a formatted floppy disk. If the command /sbin/grub-floppy exists run it, otherwise (preferably not as root so that you can't overwrite the systems grub configuration by mistake) create a file system (mke2fs /dev/fd0), mount it (mount /mnt/floppy) and run /sbin/grub-install --root-directory=/mnt/floppy /dev/fd0 Remember to unmount the disk with umount /mnt/floppy afterwards.
4. Even this limitation can be relaxed if you are prepared to use two different stage2 files. For example if the mirrored boot partition is on /dev/hda1 and /dev/hdb2, the following should work (where stage2a and stage2b are copied from stage2 beforehand):
   install --stage2=/boot/grub/stage2a (hd0,0)/grub/stage1 (hd0) (hd0,0)/grub/stage2a p /grub/grub.conf

   install --stage2=/boot/grub/stage2b (hd1,1)/grub/stage1 (hd1) (hd1,1)/grub/stage2b p /grub/grub.conf
5. The way to break a mirror seems to be to use the raidsetfaulty command. To recover afterwards, you can use raidhotremove to remove the dead disk from the mirror if necessary, and then raidhotadd to replace it and resynchronize. Look at /proc/mdstat to see the mirror status.