Procedure:HostDiskMigration

The host disk migration procedure is to be used in the event that

• a host's disk is failing and it must be moved to a good disk
• expanded host storage on the root disk is needed

WARNING: This procedure is obviously not without risks, backups should be available first!

If everything is done correctly, this process will be read-only from the original disk's perspective, however mistakes can be made and if grub is reinstalled wrong (or the wrong disk gets repartitioned) the results would be gruesome.

New disk preparation

Standard partitions procedure

• Hot install new disk in the host; It is recognized and appears in the /dev/sd* table
• Format the new disk to have the same set of partitions as the old disk
  • If the new disk is larger, feel free to enlarge
  • make sure to mark the /boot partition as bootable
  • mark the partition types as the same
• mkfs the same types of filesystem as on the origin disk
  • We do not generally invoke special options with these so the mkfs process is simple

Note: In some cases (mainly older systems) the origin disk label may be DOS which cannot be used on >2T disks, in which case a GPT label must be used.

LVM preparations procedure

• install new disk
• Copy disk partition table,
  • This will include partitions for EFI, /boot and the big one for LVM

• vgcreate rhelb /dev/newdisk/lvm_partition

  • rhelb is the new volume name, presumably 'ubuntub' would be better for an ubuntu system.
• Use

  lvdisplay</pre to show what's inside the old LVM

* For each existing LV with name FOO, ** <pre>lvcreate -L xxxG -n FOO rhelb

• • mkfs.zzz for the new /dev/mapper/rhelb_FOO

The origin disk's underlying filesystem structure has now been duplicated on the target and copying can proceed as before.

Filesystem copy

• mkdir /mnt/tmp -p

Once the copy of the partitions is ready, copy everything over at the filesystem level. One rsync will be needed for each origin partition

• mount /dev/sdbX /mnt/tmp # X is the boot partiton
• rsync -axHAXS /boot/ /mnt/tmp
• umount /mnt/tmp

# Next is the master filesystem copy
# IMPORTANT:
# Theoretically this should be done using a liveusb to assure that the source FS is static,
# however if the system is completely idle, all important services are stopped, etc,
# then only /var/log/* is likely to change. YMMV! The last handful of log entries will be lost.

• mount /dev/sdbY /mnt/tmp
• rsync -axHAXS / /mnt/tmp
  • -> be sure that /mnt/tmp is mounted, baby, otherwise rsync will keep going until it crashes into the filesystem recursion limit

Procedure for merging partitions (e.g. removal of separate /var/lib/docker partition). Note that src directory is /src/dir/ with a trailing / and not /src/dir.

• mount /dev/sdbY /mnt/tmp # the new /
• rsync -axHAXS /var/lib/docker/ /mnt/tmp/var/lib/docker

Make new disk bootable

Prepare for chroot:

• umount /mnt/tmp
• mount /dev/sdbY /mnt/tmp
• mount /dev/sdbX /mnt/tmp/boot
• mount --rbind /sys /mnt/tmp/sys
• mount -t proc /proc /mnt/tmp/proc
• mount --rbind /dev /mnt/tmp/dev
• chroot /mnt/tmp

Grab all the block device IDs and drop them in fstab. At this point, the UUIDs of the new disk must be slipped in for the mount points of the old disk in the new filesystem,

• blkid >> /etc/fstab
• vi /etc/fstab

If the system fstab does not already utilize UUID, it should be changed to do so. /dev/sd* are subject to changes depending on drive enumeration order and can render a system unbootable if an expected partition moves.

bios boot migration

Reinstall the bootloader on the new drive, regenerate the grub2 configuration and rebuild the initramfs:

• grub2-install /dev/sdb
• grub2-mkconfig -o /boot/grub2/grub.cfg
• RHEL: dracut -f
• Ubuntu: update-initramfs -u -k all
  • Make absolutely sure that the new disk /boot is mounted before entering chroot!

At this point, check both /boot/grub2/grubenv and any existing grub2 config file to double-check that the root device is the uuid of the new disk's /.

uefi boot migration

See also: https://www.linuxbabe.com/command-line/how-to-use-linux-efibootmgr-examples

grub2 will refuse to install. Check /boot/efi/EFI/redhat/ and a grub configuration should be seen. This is the actual grub configuration that is needed.

• cd /boot/efi/EFI/redhat
• cp grub.cfg ~/old_grub.cfg
• grub2-mkconfig -o ./grub.cfg

Now the efi bootloader has to be told to jump here,

• efibootmgr -v

will list all entries it has. Use this to insert the new entry,

• efibootmgr -c -d /dev/FOO -p 1 -L redhat -l \EFI\redhat\grubx64.efi
  • -c create
  • -d device: the drive to be booted from
  • -p 1: the partition (from e.g. gdisk -l /dev/FOO) which is the EFI System Partition
  • -L redhat: The name of the bootloader directory that was copied by the rsync
  • -l The full path (rooted at the EFI partition) to the grub loader grubx64.efi file

Final steps

Upon reboot, monitor using ipmi remote control and order it to boot from the new disk and confirm the system comes up successfully. In event of failure, the original disk is available to fallback. Upon success, remove old disk and reboot system one more time to confirm it successfully brings itself up.