Troubleshooting and Rescue
The fundamental rescue theorem: if anything at all boots — a live USB, another partition, the installer's rescue shell — the broken system can be mounted, chrooted into, and repaired. This chapter is the failure playbook: the boot failures, the filesystem damage, the forgotten passwords, and the universal mount-chroot-fix move that solves most of them.
The universal rescue move
# boot a live USB / the installer's rescue mode, then:
mount /dev/sda3 /mnt # the broken system's root
mount --bind /dev /mnt/dev # the kernel interfaces
mount --bind /dev/pts /mnt/dev/pts
mount -t proc proc /mnt/proc
mount -t sysfs sys /mnt/sys
mount -t tmpfs tmpfs /mnt/run
chroot /mnt /bin/bash # you're inside the patient
# fix: reinstall grub, rebuild the kernel, reset a password, edit fstab
exit; umount -R /mnt; rebootThis is chapter 4's pivot in reverse — the same binds, the same chroot — aimed at a broken system instead of a building one. Learn this cold; it's the answer to most of this chapter.
The boot failures — triage by stage
| Where it stops | The layer | First move |
|---|---|---|
grub> or GRUB rescue | Bootloader | ls, set root=, linux, boot by hand |
| Kernel panic — can't mount root | Kernel | The FS/controller as module not built-in — rebuild, or add initramfs |
init: not found | rootfs/init | fstab/root= mismatch; chroot-rescue |
| Emergency/maintenance mode | fstab or fsck | The journal; mount -a to find the bad line |
| Hangs mid-boot | A service | rescue.target/init=/bin/bash |
| Black screen after GRUB | Graphics | nomodeset kernel param |
The rescue boot params — appended at GRUB's e (edit) key:
init=/bin/bash # skip init entirely — a root shell at boot
single # single-user mode — minimal services
systemd.unit=rescue.target # systemd's single-user
nomodeset # kill graphics driver — the display fix
root=/dev/sda3 # override the root=
ro # read-only rootinit=/bin/bash is the deepest door — no init, no login, just bash as PID 1: mount -o remount,rw / and you can fix anything — including resetting a lost root password (passwd).
The failure taxonomy — where it can break
| Stage | Failure | The door |
|---|---|---|
| Firmware | No boot device | Boot menu, another device |
| Bootloader | grub>/grub rescue> | ls, set, linux, boot by hand |
| Kernel load | Kernel panic — no init | init=, root= override at boot line |
| Kernel rootfs | "unable to mount root" | fs/driver built-in question — rebuild or initramfs |
| init | Emergency/maintenance mode | fstab/fsck — the journal |
| Services | A unit hangs boot | rescue/single-user target |
| Login | Forgotten password | init=/bin/bash + passwd |
| Filesystem | Corruption | fsck + backup superblocks |
| Hardware | Dying disk | ddrescue first — image before repair |
| Display | Black screen | nomodeset param |
The GRUB command line — hand-booting
grub> ls # enumerate disks — (hd0) (hd0,msdos1) (hd0,gpt3)
grub> set root=(hd0,gpt3) # point at the partition
grub> linux /vmlinuz-6.x-lfs root=/dev/sda3 ro
grub> initrd /initramfs.img # if there's one
grub> boot # hand it overHand-booting proves the kernel's fine — the rescue is in grub-install/grub.cfg, not the OS.
The filesystem failures
fsck -n /dev/sda3 # diagnose — never mounted
fsck.ext4 -y /dev/sda3 # repair
e2fsck -b 32768 /dev/sda3 # backup superblock
mount -o remount,rw / # read-only → writable in placeFilesystem damage shows as emergency-mode boots or "filesystem check failed". The backup superblocks (mkfs scattered them) rescue corrupted primaries. And dmesg shows I/O errors — a dying disk (SMART says) gets ddrescue imaged before fsck touches it.
The GRUB rescue shell — hand-booting fully
grub> ls # (hd0) (hd0,gpt1) (hd0,gpt2) (hd0,gpt3)
grub> ls (hd0,gpt3)/ # what's on the partition
grub> set root=(hd0,gpt3)
grub> linux /boot/vmlinuz-6.x-lfs root=/dev/sda3 ro
grub> initrd /boot/initrd.img # if present
grub> bootAnd to reinstall the bootloader from a live environment:
mount /dev/sda3 /mnt
mount --bind /dev /mnt/dev; mount -t proc proc /mnt/proc; mount -t sysfs sys /mnt/sys
chroot /mnt
grub-install /dev/sda # BIOS — or --target=x86_64-efi for UEFI
grub-mkconfig -o /boot/grub/grub.cfgThe emergency modes — the layers of the door
| Depth | How | Gets you |
|---|---|---|
| runlevel 1 / rescue.target | telinit 1 / systemctl isolate rescue.target | single-user, minimal services, root shell |
| emergency.target | boot param systemd.unit=emergency.target | barely-up root shell, ro rootfs |
| init=/bin/bash | kernel cmdline | bash as PID 1 — no init at all |
| init=/bin/sh | kernel cmdline | even more minimal |
| rdinit= | initramfs's init override | inside the initramfs |
| Live USB chroot | mount+bind+chroot | the full repair environment |
init=/bin/bash is the deepest door: no init, no login, root is ro → mount -o remount,rw / → fix anything → exec /sbin/init to hand control back.
The forgotten password
# rescue-boot → chroot → passwd root
# or: init=/bin/bash → remount,rw → passwd root → exec /sbin/initNo backdoor needed — physical access to a non-FDE system is root.
Reading the failure — the logs
dmesg -l err,warn # the kernel's complaints
dmesg -T | tail -30 # human timestamps, recent
journalctl -b -p err # this boot's errors (systemd)
journalctl -b -1 # the LAST boot — the crash's record
journalctl -u service # one unit's story
less /var/log/messages # SysV's equivalent
ls -lt /var/log/ # freshest logs firstjournalctl -b -1 — last boot's log — is the single most powerful troubleshooting command on systemd systems: the log of the boot that died.
The repair toolkit — a live USB's contents
A good rescue stick carries: gdisk/fdisk, fsck.*, chroot, grub-install, ddrescue, testdisk, photorec, smartctl, lsblk, mount — Slackware's installer image or a SystemRescue image carries them all. The LFS host VM is itself a rescue host for the LFS partition.
Where rescue goes wrong
| Symptom | Cause |
|---|---|
| chroot can't find /bin/bash | The mount is wrong — ls /mnt first |
| Fixed, still won't boot | Didn't reinstall GRUB after the fix |
| fsck "filesystem is mounted" | It is — unmount or use -n |
| init=/bin/bash shell is read-only | mount -o remount,rw / |
| Changes lost on reboot | Edited the live env, not the mounted system |
| Rescued but data's gone | Hardware — image first, repair second |
Exercises
- Boot a VM to
init=/bin/bash— remount rw, change root's password, boot normally. - Break fstab deliberately; find emergency mode; fix via
mount -a. - Boot to
rescue.target/runlevel 1; inventory what runs. journalctl -b -1— read a previous boot's last moments.- Write the mount+chroot rescue script from memory — the universal move.
Timeline
| Stage | The rescue |
|---|---|
| Bootloader | GRUB CLI hand-boot |
| Kernel | rescue params |
| init | init=/bin/bash |
| fstab | mount -a, emergency mode |
| System | mount+chroot+fix |
| Hardware | ddrescue, then repair |