How I recovered my Linux system using a Live USB device

I’ve realized so much over the past couple of weeks.

I created a serious drawback for myself. Having been a sysadmin for years and written a whole lot of articles and 5 books about Linux, I actually ought to have identified higher. Then once more, all of us make errors, which is a vital lesson: You are by no means too skilled to make a mistake.

I am not going to debate the main points of my error. It is sufficient to inform you that it was a mistake and that I ought to have put much more thought into what I used to be doing earlier than I did it. Apart from, the main points aren’t actually the purpose. Expertise cannot prevent from each mistake you are going to make, however it could aid you in restoration. And that is actually what this text is about: Utilizing a Reside USB distribution in addition and enter a restoration mode.

The issue

First, I created the issue, which was basically a nasty configuration for the /and so forth/default/grub file. Subsequent, I used Ansible to distribute the misconfigured file to all my bodily computer systems and run grub2-mkconfig. All 12 of them. Actually, actually quick.

All however two did not boot. They crashed through the very early phases of Linux startup with numerous errors indicating that the /root filesystem couldn’t be situated.

I may use the foundation password to get into “upkeep” mode, however with out /root mounted, it was unimaginable to entry even the only instruments. Booting on to the restoration kernel didn’t work both. The programs had been really damaged.

Restoration mode with Fedora

The one method to resolve this drawback was to discover a method to get into restoration mode. When all else fails, Fedora offers a extremely cool instrument: The identical Reside USB thumb drive used to put in new cases of Fedora.

After setting the BIOS in addition from the Reside USB system, I booted into the Fedora 36 Xfce reside consumer desktop. I opened two terminal classes subsequent to one another on the desktop and switched to root privilege in each.

I ran lsblk in a single for reference. I used the outcomes to establish the / root partition and the boot and efi partitions. I used certainly one of my VMs, as seen beneath. There isn’t a efi partition on this case as a result of this VM doesn’t use UEFI.

The /dev/sda1 partition is definitely identifiable as /boot, and the foundation partition is fairly apparent as properly.

Within the different terminal session, I carried out a sequence of steps to recuperate my programs. The precise quantity group names and system partitions akin to /dev/sda1 will differ to your programs. The instructions proven listed here are particular to my state of affairs.

The target is in addition and get by way of startup utilizing the Reside USB, then mount solely the required filesystems in a picture listing and run the chroot command to run Linux within the chrooted picture listing. This strategy bypasses the broken GRUB (or different) configuration information. Nonetheless, it offers an entire operating system with all the unique filesystems mounted for restoration, each because the supply of the instruments required and the goal of the adjustments to be made.

Listed here are the steps and associated instructions:

1. Create the listing /mnt/sysimage to offer a location for the chroot listing.

2. Mount the foundation partition on /mnt/sysimage:

# mount /dev/mapper/vg01-root /mnt/sysimage

3. Make /mnt/sysimage your working listing:

# cd /mnt/sysimage

4. Mount the /boot and /boot/efi filesystems.

5. Mount the opposite principal filesystems. Filesystems like /dwelling and /tmp usually are not wanted for this process:

6. Mount essential however already mounted filesystems that should be shared between the chrooted system and the unique Reside system, which remains to be on the market and operating:

7. You should definitely do the /dev listing final, or the opposite filesystems will not mount:

# mount --bind /dev dev

8. Chroot the system picture:

# chroot /mnt/sysimage

The system is now prepared for no matter it is advisable to do to recuperate it to a working state. Nonetheless, one time I used to be capable of run my server for a number of days on this state till I may analysis and take a look at actual fixes. I do not actually advocate that, however it may be an choice in a dire emergency when issues simply have to stand up and operating–now!

The answer

The repair was straightforward as soon as I received every system into restoration mode. As a result of my programs now labored simply as if they’d booted efficiently, I merely made the required adjustments to /and so forth/default/grub and /and so forth/fstab and ran the grub2-mkconfig > boot/grub2/grub.cfg command. I used the exit command to exit from chroot after which rebooted the host.

After all, I couldn’t automate the restoration from my mishap. I needed to carry out this complete course of manually on every host—a becoming little bit of karmic retribution for utilizing automation to rapidly and simply propagate my very own errors.

Classes realized

Regardless of their usefulness, I used to hate the “Classes Realized” classes we might have at a few of my sysadmin jobs, nevertheless it does seem that I have to remind myself of some issues. So listed here are my “Classes Realized” from this self-inflicted fiasco.

First, the ten programs that did not boot used a unique quantity group naming scheme, and my new GRUB configuration failed to think about that. I simply ignored the truth that they could presumably be completely different.

• Suppose it by way of utterly.
• Not all programs are alike.
• Check the whole lot.
• Confirm the whole lot.
• By no means make assumptions.

Every thing now works tremendous. Hopefully, I’m just a little bit smarter, too.