Diff — Troubleshooting and Rescue

{{infobox

| title = Troubleshooting & Rescue

| chapter = Linux Handbook, chapter 25 — when it breaks: boot failures, rescue mode, chroot surgery

| core | The host boots → mount → chroot → fix — the universal recovery move

| tools | live USB · rescue target · chroot · journalctl/dmesg · fsck

| mindset | Nothing is unrecoverable while *something* boots

| next | The kernel deep dive — config as authorship

}}

**The fundamental rescue theorem**: if anything at all boots — a live USB, another partition, the installer's rescue shell — the broken system can be mounted, chrooted into, and repaired. This chapter is the failure playbook: the boot failures, the filesystem damage, the forgotten passwords, and the universal mount-chroot-fix move that solves most of them.

## The universal rescue move

```sh

# boot a live USB / the installer's rescue mode, then:

mount /dev/sda3 /mnt                  # the broken system's root

mount --bind /dev /mnt/dev            # the kernel interfaces

mount --bind /dev/pts /mnt/dev/pts

mount -t proc proc /mnt/proc

mount -t sysfs sys /mnt/sys

mount -t tmpfs tmpfs /mnt/run

chroot /mnt /bin/bash                 # you're inside the patient

# fix: reinstall grub, rebuild the kernel, reset a password, edit fstab

exit; umount -R /mnt; reboot

```

This is chapter 4's pivot in reverse — the same binds, the same chroot — aimed at a *broken* system instead of a building one. Learn this cold; it's the answer to most of this chapter.

## The boot failures — triage by stage

| Where it stops | The layer | First move |

| --- | --- | --- |

| `grub>` or [[GRUB]] rescue | Bootloader | `ls`, `set root=`, `linux`, `boot` by hand |

| Kernel panic — can't mount root | Kernel | The FS/controller as module not built-in — rebuild, or add initramfs |

| `init: not found` | rootfs/init | fstab/root= mismatch; chroot-rescue |

| Emergency/maintenance mode | fstab or fsck | The journal; `mount -a` to find the bad line |

| Hangs mid-boot | A service | `rescue.target`/`init=/bin/bash` |

| Black screen after GRUB | Graphics | `nomodeset` kernel param |

**The rescue boot params** — appended at GRUB's `e` (edit) key:

```

init=/bin/bash        # skip init entirely — a root shell at boot

single                # single-user mode — minimal services

systemd.unit=rescue.target   # systemd's single-user

nomodeset             # kill graphics driver — the display fix

root=/dev/sda3        # override the root=

ro                    # read-only root

```

`init=/bin/bash` is the deepest door — no init, no login, just bash as PID 1: `mount -o remount,rw /` and you can fix *anything* — including resetting a lost root password (`passwd`).

## The failure taxonomy — where it can break

| Stage | Failure | The door |

| --- | --- | --- |

| Firmware | No boot device | Boot menu, another device |

| Bootloader | `grub>`/`grub rescue>` | `ls`, `set`, `linux`, `boot` by hand |

| Kernel load | Kernel panic — no init | `init=`, `root=` override at boot line |

| Kernel rootfs | "unable to mount root" | fs/driver built-in question — rebuild or initramfs |

| init | Emergency/maintenance mode | fstab/fsck — the journal |

| Services | A unit hangs boot | rescue/single-user target |

| Login | Forgotten password | `init=/bin/bash` + `passwd` |

| Filesystem | Corruption | fsck + backup superblocks |

| Hardware | Dying disk | ddrescue first — image before repair |

| Display | Black screen | `nomodeset` param |

## The GRUB command line — hand-booting

```text

grub> ls                      # enumerate disks — (hd0) (hd0,msdos1) (hd0,gpt3)

grub> set root=(hd0,gpt3)     # point at the partition

grub> linux /vmlinuz-6.x-lfs root=/dev/sda3 ro

grub> initrd /initramfs.img   # if there's one

grub> boot                    # hand it over

```

Hand-booting proves the kernel's fine — the rescue is in `grub-install`/`grub.cfg`, not the OS.

## The filesystem failures

```sh

fsck -n /dev/sda3              # diagnose — never mounted

fsck.ext4 -y /dev/sda3         # repair

e2fsck -b 32768 /dev/sda3      # backup superblock

mount -o remount,rw /          # read-only → writable in place

```

Filesystem damage shows as emergency-mode boots or "filesystem check failed". The backup superblocks (`mkfs` scattered them) rescue corrupted primaries. And `dmesg` shows I/O errors — a *dying* disk (SMART says) gets `ddrescue` imaged before fsck touches it.

## The GRUB rescue shell — hand-booting fully

```text

grub> ls                          # (hd0) (hd0,gpt1) (hd0,gpt2) (hd0,gpt3)

grub> ls (hd0,gpt3)/              # what's on the partition

grub> set root=(hd0,gpt3)

grub> linux /boot/vmlinuz-6.x-lfs root=/dev/sda3 ro

grub> initrd /boot/initrd.img     # if present

grub> boot

```

And to reinstall the bootloader from a live environment:

```sh

mount /dev/sda3 /mnt

mount --bind /dev /mnt/dev; mount -t proc proc /mnt/proc; mount -t sysfs sys /mnt/sys

chroot /mnt

grub-install /dev/sda              # BIOS — or --target=x86_64-efi for UEFI

grub-mkconfig -o /boot/grub/grub.cfg

```

## The emergency modes — the layers of the door

| Depth | How | Gets you |

| --- | --- | --- |

| runlevel 1 / rescue.target | `telinit 1` / `systemctl isolate rescue.target` | single-user, minimal services, root shell |

| init=/bin/bash | kernel cmdline | bash as PID 1 — no init at all |

| init=/bin/sh | kernel cmdline | even more minimal |

| rdinit= | initramfs's init override | inside the initramfs |

| Live USB chroot | mount+bind+chroot | the full repair environment |

`init=/bin/bash` is the deepest door: no init, no login, root is ro → `mount -o remount,rw /` → fix anything → `exec /sbin/init` to hand control back.

## The forgotten password

```sh

# rescue-boot → chroot → passwd root

# or: init=/bin/bash → remount,rw → passwd root → exec /sbin/init

```

No backdoor needed — physical access to a non-FDE system is root.

## Reading the failure — the logs

```sh

dmesg -l err,warn               # the kernel's complaints

dmesg -T | tail -30             # human timestamps, recent

journalctl -b -p err            # this boot's errors (systemd)

journalctl -b -1                # the LAST boot — the crash's record

journalctl -u service           # one unit's story

less /var/log/messages          # SysV's equivalent

ls -lt /var/log/                # freshest logs first

```

`journalctl -b -1` — last boot's log — is the single most powerful troubleshooting command on systemd systems: the log of the boot that *died*.

## The repair toolkit — a live USB's contents

A good rescue stick carries: `gdisk/fdisk`, `fsck.*`, `chroot`, `grub-install`, `ddrescue`, `testdisk`, `photorec`, `smartctl`, `lsblk`, `mount` — Slackware's installer image or a SystemRescue image carries them all. The LFS host VM is itself a rescue host for the LFS partition.

## Where rescue goes wrong

| Symptom | Cause |

| --- | --- |

| chroot can't find /bin/bash | The mount is wrong — `ls /mnt` first |

| Fixed, still won't boot | Didn't reinstall GRUB after the fix |

| fsck "filesystem is mounted" | It is — unmount or use `-n` |

| init=/bin/bash shell is read-only | `mount -o remount,rw /` |

| Changes lost on reboot | Edited the live env, not the mounted system |

| Rescued but data's gone | Hardware — image first, repair second |

## Exercises

1. Boot a VM to `init=/bin/bash` — remount rw, change root's password, boot normally.

2. Break fstab deliberately; find emergency mode; fix via `mount -a`.

3. Boot to `rescue.target`/runlevel 1; inventory what runs.

4. `journalctl -b -1` — read a previous boot's last moments.

5. Write the mount+chroot rescue script from memory — the universal move.

## Timeline

| Stage | The rescue |

| --- | --- |

| Bootloader | GRUB CLI hand-boot |

| Kernel | rescue params |

| init | init=/bin/bash |

| fstab | mount -a, emergency mode |

| System | mount+chroot+fix |

| Hardware | ddrescue, then repair |

## See also

- [[Inside a Linux System]] · [[Disks and Storage]] · [[LFS Kernel and Boot]] · [[Linux]]

[[Category:Linux]]

[[Category:Handbook]]

[[Category:Recovery]]

[[Category:Linux Handbook]]