Procedure:HostDiskMigration

The host disk migration procedure is to be used in the event that

• a host's disk is failing and it must be moved to a good disk
• expanded host storage on the root disk is needed

WARNING: This procedure is obviously not without risks, backups should be available first!

If everything is done correctly, this process will be read-only from the original disk's perspective, however mistakes can be made and if grub is reinstalled wrong (or the wrong disk gets repartitioned) the results would be gruesome.

New disk preparation

• Hot install new disk in the host; It is recognized and appears in the /dev/sd* table
• Format the new disk to have the same set of partitions as the old disk
  • If the new disk is larger, feel free to enlarge
  • make sure to mark the /boot partition as bootable
  • mark the partition types as the same
• mkfs the same types of filesystem as on the origin disk
  • We do not generally invoke special options with these so the mkfs process is simple

Note: In some cases (mainly older systems) the origin disk label may be DOS which cannot be used on >2T disks, in which case a GPT label must be used and there is no need to mark partition types.

Filesystem copy

• mkdir /mnt/tmp -p

Once the copy of the partitions is ready, copy everything over at the filesystem level. One rsync will be needed for each origin partition

• mount /dev/sdbX /mnt/tmp # X is the boot partiton
• rsync -axHAXS /boot/ /mnt/tmp
• umount /mnt/tmp

# Next is the master filesystem copy
# IMPORTANT:
# Theoretically this should be done using a liveusb to assure that the source FS is static,
# however if the system is completely idle, all important services are stopped, etc,
# then only /var/log/* is likely to change. YMMV! The last handful of log entries will be lost.

• mount /dev/sdbY /mnt/tmp
• rsync -axHAXS / /mnt/tmp
  • -> be sure that /mnt/tmp is mounted, baby, otherwise rsync will keep going until it crashes into the filesystem recursion limit

Procedure for merging partitions (e.g. removal of separate /var/lib/docker partition). Note that src directory is /src/dir/ with a trailing / and not /src/dir.

• mount /dev/sdbY /mnt/tmp # the new /
• rsync -axHAXS /var/lib/docker/ /mnt/tmp/var/lib/docker

Make new disk bootable

Prepare for chroot:

• umount /mnt/tmp
• mount /dev/sdbY /mnt/tmp
• mount /dev/sdbX /mnt/tmp/boot
• mount --rbind /sys /mnt/tmp/sys
• mount -t proc /proc /mnt/tmp/proc
• mount --rbind /dev /mnt/tmp/dev
• chroot /mnt/tmp

Grab all the block device IDs and drop them in fstab. At this point, the UUIDs of the new disk must be slipped in for the mount points of the old disk in the new filesystem,

• blkid >> /etc/fstab
• vi /etc/fstab

If the system fstab does not already utilize UUID, it should be changed to do so. /dev/sd* are subject to changes depending on drive enumeration order and can render a system unbootable if an expected partition moves.

• grub2-install /dev/sdb
• grub2-mkconfig -o /boot/grub2/grub.cfg
• dracut -f
  • Make absolutely sure that the new disk /boot is mounted before entering chroot!

At this point, check both /boot/grub2/grubenv and any existing grub2 config file to double-check that the root device is the uuid of the new disk's /.

Upon reboot, monitor using ipmi remote control and order it to boot from the new disk and confirm the system comes up successfully. In event of failure, the original disk is available to fallback. Upon success, remove old disk and reboot system one more time to confirm it successfully brings itself up.