Debian 13: Recover root access with rescue mode — do it without making it worse

The only rules that matter in rescue mode

Rescue mode is a scalpel, not a chainsaw. The goal is narrow: regain controlled administrative access, then exit cleanly.
Do not “fix everything while you’re here.” That’s how root access incidents become multi-day rebuilds.

Rule 1: Prove what system you’re touching

Rescue environments are full of footguns because they make it easy to mount disks from multiple systems.
Before you change anything, confirm the hostname, disk IDs, and the root filesystem you intend to repair.
If you can’t prove it, you’re guessing. Guessing is expensive.

Rule 2: Default to read-only until you’ve identified the fault

A lot of “root access” failures are not authentication at all. They’re boot-time mount failures, broken initramfs, corrupted PAM modules,
misapplied permissions on /etc/shadow, or an expired disk encryption prompt you never see on headless boots.

Mount read-only first. Gather evidence. Then remount read-write for the one change that addresses the root cause.

Rule 3: Prefer minimal, reversible changes

Editing PAM stacks, sudoers, or SSHD configuration in panic mode is how you lock yourself out twice.
If you must change authentication, use a tested path: set a temporary root password, re-enable a known-good admin user, and schedule a proper fix.

Rule 4: Keep a forensic trail

You will forget what you did. Your teammate will ask. Your future self will curse you.
Take notes: what was broken, what you ran, what file you edited. Even a crude text log in /root/rescue-notes.txt is worth gold.

Joke #1: Rescue mode is like surgery—sterile field, clean instruments, and absolutely no “let’s see what happens if I poke this.”

Interesting facts and context (why Debian behaves this way)

• Single-user roots: Traditional Unix “single-user mode” predates systemd by decades and was meant for local console recovery when multi-user services were broken.
• systemd emergency vs rescue: On systemd systems, “rescue” targets aim for a minimal system; “emergency” is even smaller—often just an initramfs-like shell with fewer mounts.
• Root account can be locked by design: Many Debian installs rely on sudo and may lock root by setting its password hash to a special value. That’s not a bug; it’s a policy choice.
• PAM is a stack, not a switch: Pluggable Authentication Modules were designed to let enterprises swap auth methods without rewriting applications, which also means one broken line can break everything.
• /etc/shadow wasn’t always separate: The split between /etc/passwd and /etc/shadow is a security evolution to hide password hashes from non-root users.
• Initramfs is a tiny OS: Modern Linux boots through an initial RAM filesystem that loads drivers, decrypts disks, activates LVM, and mounts the real root. If that tiny OS is wrong, your “root problem” is upstream.
• GRUB became the default for good reasons: GRUB’s ability to edit kernel parameters at boot is a lifesaver in recovery—and also a reason physical console access matters.
• Mount options can lock you out: A single bad line in /etc/fstab can drop you into emergency mode, even if your passwords are fine.

One quote to keep in your head when the pressure rises: “Hope is not a strategy.” — General Gordon R. Sullivan.

Fast diagnosis playbook (check 1/2/3)

When you’re on a console with a broken system, the fastest route is to classify the failure. Not by vibes. By signals.

Check 1: Are you in an auth failure or a boot/mount failure?

• If you get a login prompt but root/sudo fails: likely account state, PAM, shadow permissions, or sudoers.
• If you never reach a normal login and land in emergency mode: likely fstab, initramfs, LUKS/LVM activation, filesystem errors.
• If the system is up but you can’t get in remotely: likely network, sshd config, host keys, firewall, or key permissions.

Check 2: Is the root filesystem intact and mountable?

Before touching auth, verify you can mount the real root. If you can’t mount root safely, stop and diagnose storage.
Fixing passwords on the wrong mountpoint is a classic self-own.

Check 3: Can you reproduce the failure with logs?

In rescue mode, mount root read-only and inspect /var/log and the journal. If the failure is repeatable and logged, you can fix it precisely.
If it’s not logged, you still can often infer the cause from configuration drift and boot errors.

Ways to enter rescue mode on Debian 13

1) GRUB: edit kernel command line (console access)

If you can reach GRUB, you can usually reach root recovery. Highlight your normal boot entry, press e, and add:
systemd.unit=rescue.target or systemd.unit=emergency.target to the Linux line.

Rescue target tends to bring more of the system up (including some mounts). Emergency target is minimal and is useful when mounts are broken.

2) Debian installer media: “Rescue a broken system”

Boot the installer ISO (or netinst) and choose the rescue option. It can detect installed systems, mount them, and offer a chroot.
This is often the cleanest path on headless servers where you have remote KVM or a virtual console.

3) Cloud/VM: attach a rescue ISO or use provider rescue image

In production, you may not have physical access. Use your hypervisor’s console and boot a rescue ISO, or provider rescue mode.
The same discipline applies: identify disks by stable IDs, mount read-only first, and be careful with encrypted volumes.

Triage map: what you’re actually fixing

“Recover root access” can mean several different problems. Classify it before you act.

A. Root account locked or password unknown

Root may be locked intentionally, or you may have inherited a system where nobody knows the password.
Fix is usually: mount system, chroot, set a new password or re-enable sudo for a known admin user.

B. sudo broken (syntax, permissions, wrong file)

A malformed sudoers entry can break all sudo usage. That looks like “no root access,” but it’s really a config validation failure.
Fix is: validate with visudo (or repair carefully) from a chroot.

C. PAM/auth stack broken

If PAM modules were removed, mismatched, or misconfigured, logins will fail even with correct passwords.
Fix is usually restoring correct packages/config; avoid “disabling PAM” hacks.

D. System drops into emergency mode (fstab, fsck, device not found)

This isn’t an auth problem. It’s a boot dependency problem. Fix the mount, UUID, or filesystem issue first.

E. Encrypted root or LVM not activating

If initramfs can’t unlock LUKS or assemble LVM, your root never appears. Fix is in initramfs, crypttab, lvm metadata, or missing drivers.

F. Remote access gone (sshd, keys, firewall) but local root exists

Sometimes the machine is fine and only remote access is broken. Don’t reset root just because SSH is down.
Fix SSH and network with minimum changes.

Practical tasks (commands, outputs, decisions)

These are the tasks I actually run in rescue scenarios. Each one has: command, what typical output means, and the decision you make.
Commands assume a rescue shell with root privileges. Host prompt is fixed per your requirements.

Task 1: Identify disks and partitions (don’t guess)

cr0x@server:~$ lsblk -o NAME,SIZE,TYPE,FSTYPE,UUID,MOUNTPOINTS
NAME        SIZE TYPE FSTYPE UUID                                 MOUNTPOINTS
sda         480G disk
├─sda1      512M part vfat   7C2A-1F0B                            /boot/efi
├─sda2        2G part ext4   2a9c1f7b-1c77-4e7b-8f4c-2df9c5a2d9c1 /boot
└─sda3    477.5G part crypto_LUKS 3a0b6d9b-... 
  └─cryptroot 477.5G crypt LVM2_member 8h2KqX-...
    ├─vg0-root  80G lvm  ext4   7f6b9e0c-...                      /
    └─vg0-var  200G lvm  xfs    4f0d2c1a-...                      /var

Meaning: You can see whether you’re dealing with plain partitions, LUKS, LVM, and which FS types you’ll need tools for (ext4, xfs, btrfs).

Decision: If you see LUKS/LVM layers, plan to unlock and activate before mounting. If the root FS isn’t obvious, stop and map it before editing anything.

Task 2: Confirm the kernel saw the storage you expect

cr0x@server:~$ dmesg -T | grep -E "sd[a-z]|nvme|dm-|crypt|I/O error" | tail -n 20
[Mon Dec 30 01:12:03 2025] nvme nvme0: pci function 0000:01:00.0
[Mon Dec 30 01:12:03 2025] nvme0n1: p1 p2 p3
[Mon Dec 30 01:12:04 2025] device-mapper: crypt: dm-0: set key succeeded

Meaning: No I/O errors is good. If you see timeouts or read errors, you might be in failing-disk territory, not auth.

Decision: If storage is unstable, minimize writes. Prefer read-only mounts, take an image/snapshot if available, and plan for replacement rather than “fixing configs.”

Task 3: If LUKS is used, unlock explicitly (and verify)

cr0x@server:~$ cryptsetup luksOpen /dev/sda3 cryptroot
Enter passphrase for /dev/sda3:

cr0x@server:~$ cryptsetup status cryptroot
/dev/mapper/cryptroot is active.
  type:    LUKS2
  cipher:  aes-xts-plain64
  keysize: 512 bits
  device:  /dev/sda3
  sector size:  512
  offset:  32768 sectors
  size:    1000204288 sectors
  mode:    read/write

Meaning: The mapper device exists and is active. The mode shows whether it’s writable at this layer.

Decision: If unlock fails, don’t brute-force. Verify you’re on the right device, check keyboard layout issues in the rescue environment, and confirm you have the right passphrase source.

Task 4: Activate LVM volumes (if present)

cr0x@server:~$ vgscan
  Found volume group "vg0" using metadata type lvm2

cr0x@server:~$ vgchange -ay
  2 logical volume(s) in volume group "vg0" now active

Meaning: LVM metadata is readable and logical volumes are active.

Decision: If activation fails, inspect pvs/vgs/lvs and consider whether the PV is missing (detached disk) or filtered by LVM config.

Task 5: Mount root read-only first (always)

cr0x@server:~$ mount -o ro /dev/vg0-root /mnt
cr0x@server:~$ mount | grep " /mnt "
/dev/mapper/vg0-root on /mnt type ext4 (ro,relatime)

Meaning: Root filesystem mounts cleanly in read-only mode.

Decision: If mount fails with “wrong fs type” or “bad superblock,” stop and run filesystem checks from the correct toolchain (next tasks). Don’t start editing /etc if the FS is unhealthy.

Task 6: Mount the supporting filesystems (boot, EFI, var) correctly

cr0x@server:~$ mount -o ro /dev/sda2 /mnt/boot
cr0x@server:~$ mount -o ro /dev/sda1 /mnt/boot/efi
cr0x@server:~$ mount -o ro /dev/vg0-var /mnt/var
cr0x@server:~$ findmnt -R /mnt
TARGET        SOURCE               FSTYPE OPTIONS
/mnt          /dev/mapper/vg0-root ext4   ro,relatime
/mnt/boot     /dev/sda2            ext4   ro,relatime
/mnt/boot/efi /dev/sda1            vfat   ro,relatime
/mnt/var      /dev/mapper/vg0-var  xfs    ro,relatime

Meaning: Your chroot (later) will see the same layout the live system expects.

Decision: If /var is separate and not mounted, journal logs and auth state may look “missing,” leading to wrong conclusions.

Task 7: Check what actually failed last boot (journal)

cr0x@server:~$ journalctl --directory=/mnt/var/log/journal -b -1 -p err..alert --no-pager | tail -n 30
Dec 29 23:58:12 db13 systemd[1]: Failed to mount /data.
Dec 29 23:58:12 db13 systemd[1]: Dependency failed for Local File Systems.
Dec 29 23:58:12 db13 systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.

Meaning: This is a boot failure pattern (fstab mount), not a password failure.

Decision: Fix the mount definition before touching user accounts. Otherwise you’ll “recover root” and still drop into emergency mode.

Task 8: Inspect and validate /etc/fstab safely

cr0x@server:~$ sed -n '1,200p' /mnt/etc/fstab
UUID=7f6b9e0c-...  /      ext4  defaults,errors=remount-ro  0 1
UUID=2a9c1f7b-...  /boot  ext4  defaults                   0 2
UUID=7C2A-1F0B     /boot/efi vfat umask=0077               0 1
UUID=deadbeef-...  /data  ext4  defaults                   0 2

Meaning: That UUID=deadbeef... is suspicious if it doesn’t exist.

Decision: Compare against blkid. If the UUID is stale, either correct it or add nofail,x-systemd.device-timeout=10s for non-critical mounts to let the system boot.

Task 9: Verify UUIDs and labels (ground truth)

cr0x@server:~$ blkid
/dev/sda1: UUID="7C2A-1F0B" TYPE="vfat" PARTUUID="..."
/dev/sda2: UUID="2a9c1f7b-1c77-4e7b-8f4c-2df9c5a2d9c1" TYPE="ext4" PARTUUID="..."
/dev/mapper/vg0-root: UUID="7f6b9e0c-..." TYPE="ext4"
/dev/mapper/vg0-var: UUID="4f0d2c1a-..." TYPE="xfs"

Meaning: If a UUID in fstab isn’t in this list, it won’t mount at boot.

Decision: Fix the incorrect identifier. If the disk is intentionally gone (retired volume), remove/comment the mount or mark as nofail.

Task 10: If filesystem integrity is questionable, check it properly

For ext4, use fsck on an unmounted filesystem. For XFS, you generally use xfs_repair (and it’s picky about being unmounted).

cr0x@server:~$ umount /mnt
cr0x@server:~$ fsck.ext4 -f /dev/vg0-root
e2fsck 1.47.0 (5-Feb-2023)
/dev/vg0-root: clean, 512345/5242880 files, 12345678/20971520 blocks

Meaning: “clean” means the FS is likely not your problem.

Decision: If it reports errors and fixes them, re-mount and re-check logs. If it reports severe corruption, pause and consider imaging/backups before further writes.

Task 11: Prepare for a chroot the right way (bind mounts)

cr0x@server:~$ mount /dev/vg0-root /mnt
cr0x@server:~$ mount /dev/sda2 /mnt/boot
cr0x@server:~$ mount /dev/sda1 /mnt/boot/efi
cr0x@server:~$ mount --bind /dev  /mnt/dev
cr0x@server:~$ mount --bind /proc /mnt/proc
cr0x@server:~$ mount --bind /sys  /mnt/sys
cr0x@server:~$ mount --bind /run  /mnt/run
cr0x@server:~$ chroot /mnt /bin/bash

Meaning: You now operate as if booted into the installed OS, with device nodes and proc/sys available.

Decision: If you’re going to rebuild initramfs, update GRUB, or run package repairs, do it from a correct chroot. Otherwise you risk generating boot artifacts for the wrong environment.

Task 12: Check whether root is locked or expired

cr0x@server:~$ passwd -S root
root L 2025-10-01 0 99999 7 -1

Meaning: The L indicates locked. Other states include P for password set.

Decision: If root is locked but you need it temporarily, unlock and set a password, then later decide whether to re-lock and rely on sudo.

Task 13: Set or reset root password safely

cr0x@server:~$ passwd root
New password:
Retype new password:
passwd: password updated successfully

Meaning: Root authentication should now work (assuming PAM isn’t broken).

Decision: Use a temporary strong password, store it in your incident vault, and rotate it post-incident. Don’t leave a “temporary” password in production for months—time has a way of making temporary permanent.

Task 14: Validate sudoers (don’t hand-edit blind)

cr0x@server:~$ visudo -c
/etc/sudoers: parsed OK
/etc/sudoers.d/ops: parsed OK

Meaning: Syntax is valid. If it fails, it will tell you where.

Decision: If sudoers is broken, fix the indicated file. If you can’t use visudo due to editor issues in rescue, set EDITOR=vi explicitly rather than editing with random tools.

Task 15: Check PAM breakage quickly (missing modules is common)

cr0x@server:~$ grep -R --line-number -E "pam_unix|pam_sss|pam_tally2|pam_faillock" /etc/pam.d | head
/etc/pam.d/common-auth:25:auth    [success=1 default=ignore]      pam_unix.so nullok
/etc/pam.d/sshd:1:@include common-auth

cr0x@server:~$ ldconfig -p | grep pam_unix || true

Meaning: If PAM config references modules not present on disk, logins fail. The absence of expected PAM libraries is a red flag after partial upgrades or overzealous cleanup.

Decision: Reinstall the relevant packages from within chroot if networking/repo access is available; otherwise use local media or fix the package state when the machine boots.

Task 16: Confirm /etc/shadow ownership and permissions

cr0x@server:~$ ls -l /etc/passwd /etc/shadow /etc/group /etc/gshadow
-rw-r--r-- 1 root root   2934 Dec 29 23:01 /etc/passwd
-rw-r----- 1 root shadow 1682 Dec 29 23:01 /etc/shadow
-rw-r--r-- 1 root root   1203 Dec 29 23:01 /etc/group
-rw-r----- 1 root shadow  986 Dec 29 23:01 /etc/gshadow

Meaning: If /etc/shadow is world-readable, that’s a security incident; if it’s not readable by root (yes, it happens via weird ACLs), auth breaks.

Decision: Fix ownership and mode if wrong. If permissions “look” right but auth still fails, check for filesystem-level ACLs or immutable attributes.

Task 17: Check immutable flags and ACL surprises

cr0x@server:~$ lsattr /etc/shadow
---------------------- /etc/shadow
cr0x@server:~$ getfacl -p /etc/shadow | head
# file: /etc/shadow
# owner: root
# group: shadow
user::rw-
group::r--
other::---

Meaning: An immutable bit (i) or strange ACL can prevent edits or change access semantics.

Decision: If immutable is set and you don’t know why, stop. That’s often policy tooling. Remove it only if you understand the blast radius.

Task 18: If SSH is the problem, validate sshd config offline

cr0x@server:~$ sshd -t -f /etc/ssh/sshd_config
cr0x@server:~$ grep -E "^(PermitRootLogin|PasswordAuthentication|PubkeyAuthentication)" /etc/ssh/sshd_config
PermitRootLogin prohibit-password
PasswordAuthentication no
PubkeyAuthentication yes

Meaning: sshd -t exits silently if config is OK; it prints errors otherwise. Root login may be disabled by policy; that’s not “broken.”

Decision: If you need emergency remote access, prefer enabling a known admin user with keys over enabling root password login. If you must flip a switch, do it temporarily and log it.

Task 19: Rebuild initramfs when disk unlocking/LVM activation is failing

cr0x@server:~$ update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-6.12.0-amd64

Meaning: initramfs images are regenerated for installed kernels.

Decision: Do this when boot fails due to missing drivers/modules, broken crypttab hooks, or after you repaired LVM/crypt packages. If you didn’t change anything relevant, don’t churn boot artifacts “just because.”

Task 20: Reinstall or refresh GRUB only when you have evidence

cr0x@server:~$ grub-install /dev/sda
Installing for x86_64-efi platform.
Installation finished. No error reported.
cr0x@server:~$ update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.12.0-amd64
Found initrd image: /boot/initrd.img-6.12.0-amd64
done

Meaning: GRUB is installed and config is regenerated, detecting kernels/initrds.

Decision: Only do this if you had a bootloader failure symptom (missing GRUB, wrong boot entry, after disk replacement). Reinstalling GRUB to “fix root password” is like changing tires to fix the radio.

Task 21: Exit chroot cleanly and unmount in reverse order

cr0x@server:~$ exit
cr0x@server:~$ umount -R /mnt/run
cr0x@server:~$ umount -R /mnt/dev
cr0x@server:~$ umount -R /mnt/proc
cr0x@server:~$ umount -R /mnt/sys
cr0x@server:~$ umount -R /mnt/boot/efi
cr0x@server:~$ umount -R /mnt/boot
cr0x@server:~$ umount -R /mnt/var
cr0x@server:~$ umount -R /mnt

Meaning: No “target is busy” means your mounts are cleanly released.

Decision: If unmount complains, find what’s holding it (lsof or fuser) before rebooting. Dirty unmounts after filesystem repair are how you get “fixed it” into “broke it.”

Joke #2: The good news about rescue mode is you can fix anything. The bad news is you can also fix anything.

Three corporate mini-stories (things that really happen)

Incident 1: The wrong assumption (it was “just a password reset”)

A team inherited a Debian fleet from a vendor. One host stopped accepting SSH for the on-call engineer. The engineer assumed the root password was wrong,
booted into rescue ISO, mounted what looked like the root volume, and reset root.

The server still wouldn’t boot. Still wouldn’t accept logins. Frustration climbed, changes multiplied. Someone then noticed the host had two disks:
a small OS disk and a large data disk, both ext4, both mountable. The rescue workflow had mounted the data disk at /mnt and reset a root password
in a stray chroot that wasn’t even the OS.

The real problem was a stale /etc/fstab entry pointing to a decommissioned iSCSI LUN. systemd waited, then dropped to emergency mode.
Root access was never the blocker; the system wasn’t reaching a usable target.

They recovered by restoring the correct mounts, adding nofail and a short device timeout for non-critical volumes, and rebooted.
Then they rotated credentials properly—because they had, in effect, changed a password on the wrong filesystem and created a compliance headache.

The lesson was not “be careful.” That’s cheap advice. The lesson was to enforce a procedure: always identify root via findmnt/lsblk and confirm
/mnt/etc/os-release matches the expected install before any credential changes.

Incident 2: The optimization that backfired (auth hardening took the whole site down)

Another shop decided to standardize login security. They pushed a change to PAM to enforce stricter password complexity and lockouts.
It looked reasonable in code review: just a couple module lines in /etc/pam.d/common-auth.

A subset of servers failed logins immediately after a routine upgrade and reboot. Console access showed correct passwords being rejected.
Root wasn’t “locked”; it was simply impossible to authenticate because a referenced PAM module didn’t exist on those hosts.

Why the subset? Some servers were built from a slim baseline image where optional PAM modules weren’t installed.
The optimization was “reduce package footprint,” and it succeeded right up until someone assumed the authentication stack was identical everywhere.

Recovery used rescue mode and chroot to reinstall the missing packages and revert PAM config to the known-good baseline.
After the incident, they split PAM changes into two phases: ship packages first, validate module presence, then activate config. Boring. Correct.

Incident 3: The boring but correct practice that saved the day (notes and read-only mounts)

A production database node stopped booting after a kernel update. It landed in emergency mode with minimal output on the remote console.
The primary on-call did something heroic: nothing. They didn’t start editing; they started documenting.

They mounted the root filesystem read-only, pulled the last boot errors from the journal, and found the system was failing to unlock an encrypted volume.
The initramfs image had been generated without the correct cryptsetup integration after a partial upgrade.

Because they mounted read-only first, they avoided journaling and replay writes on a storage stack that was already under suspicion.
They then remounted read-write only to rebuild initramfs and update GRUB configuration, using a clean chroot with bind mounts.

The machine booted cleanly. Postmortem was almost painless because the on-call had a timeline and a list of commands run.
No mystery. No folklore. Just a fix that matched evidence.

Common mistakes: symptom → root cause → fix

1) Symptom: “root password is correct but login fails everywhere”

Root cause: PAM misconfiguration or missing PAM module (after upgrades, cleanup, or manual edits).

Fix: In chroot, validate PAM files under /etc/pam.d. Reinstall missing auth packages, revert to baseline config, then test with local console after reboot.

2) Symptom: Dropped into emergency mode, asked for root password for maintenance

Root cause: A failing mount in /etc/fstab (wrong UUID, missing disk, slow network storage) blocked local-fs.target.

Fix: Mount root and inspect /etc/fstab. Correct UUIDs with blkid, or mark non-critical mounts as nofail with a short timeout.

3) Symptom: sudo says “no valid sudoers sources found” or “parse error”

Root cause: Syntax error or incorrect permissions/ownership in /etc/sudoers or /etc/sudoers.d/*.

Fix: Use visudo -c in chroot. Fix the offending file and ensure correct permissions (typically 0440) and root ownership.

4) Symptom: SSH refused after “quick hardening”

Root cause: sshd_config invalid, host keys missing, or key file permissions wrong under ~/.ssh.

Fix: Validate config with sshd -t. Confirm host keys exist in /etc/ssh. Check authorized_keys permissions and ownership.

5) Symptom: After password reset in rescue, system still denies root login

Root cause: You reset the password on the wrong mounted filesystem, or root is configured to disallow password login (policy), or PAM blocks it.

Fix: Confirm you mounted the correct root (check /etc/os-release, hostname, and disk UUIDs). Confirm policy in PAM/sshd and adjust the correct layer.

6) Symptom: “Authentication token manipulation error” when running passwd

Root cause: Root filesystem mounted read-only, or /etc/shadow permissions/ownership broken, or filesystem is out of space/inodes.

Fix: Remount read-write intentionally, fix /etc/shadow mode/owner/group, and check free space with df -h and df -i.

7) Symptom: System boots, but services fail and sudo hangs

Root cause: Broken DNS or NSS configuration (e.g., pointing at dead LDAP/SSSD) causing lookups to stall; sudo often triggers user/group resolution.

Fix: In rescue/chroot, inspect /etc/nsswitch.conf, SSSD config, and ensure local files auth still works. Consider temporarily disabling remote auth dependency to regain control.

8) Symptom: Boot loop after kernel update, can’t reach login prompt

Root cause: initramfs missing storage drivers, broken crypttab, or incompatible kernel modules.

Fix: Boot an older kernel from GRUB if available. In rescue, chroot and run update-initramfs -u -k all, then update-grub.

Checklists / step-by-step plans

Plan A: You just need root back (no boot/storage issues)

1. Boot rescue target from GRUB or installer rescue.
2. Run lsblk and identify the correct root volume.
3. Mount root read-only at /mnt. Confirm /mnt/etc/os-release matches the host you think it is.
4. Mount /boot, /boot/efi, and /var
5. Bind mount /dev, /proc, /sys, /run, then chroot.
6. Check root state: passwd -S root and chage -l root.
7. Set a temporary strong root password: passwd root.
8. Validate sudoers: visudo -c. Fix if needed.
9. Exit chroot, unmount cleanly, reboot.
10. After login, rotate credentials properly and document what changed.

Plan B: System drops into emergency mode (fstab/mount blockers)

1. Do not reset passwords yet. Treat it as a boot dependency incident.
2. Mount root read-only and read the journal for last boot errors.
3. Inspect /etc/fstab for stale UUIDs, unreachable network mounts, or missing disks.
4. Use blkid to confirm identifiers and correct them.
5. For non-critical mounts, add nofail and x-systemd.device-timeout=10s to avoid blocking boot.
6. Remount root read-write only to apply the minimal change.
7. Reboot and validate boot reaches multi-user target.

Plan C: Encrypted root/LVM not coming up

1. In rescue, confirm the encrypted device is visible (lsblk and dmesg).
2. Unlock with cryptsetup luksOpen; verify status.
3. Activate LVM: vgscan, vgchange -ay.
4. Mount root and chroot properly.
5. Check /etc/crypttab, initramfs hooks, and rebuild initramfs.
6. Update GRUB config if needed, then reboot.

Plan D: SSH is broken but local access exists

1. Validate SSH config with sshd -t (from chroot if needed).
2. Confirm host keys exist and permissions are sane in /etc/ssh.
3. Prefer restoring a known admin user’s key-based access over enabling root password login.
4. Only after access is stable, consider policy adjustments and hardening.

FAQ

1) Should I use rescue.target or emergency.target?

Use rescue.target when you want a minimal system but with more normal services and mounts.
Use emergency.target when mounts are broken or you suspect fstab is what’s dropping you; it reduces dependencies and gets you a shell faster.

2) Is resetting the root password in rescue mode “safe”?

It’s safe only after you’ve proven you mounted the correct root filesystem and the filesystem is healthy.
It’s operationally risky because it changes security posture. Use a temporary password, record it securely, rotate it later, and consider re-locking root if that’s your policy.

3) Why do I get “Authentication token manipulation error” from passwd?

Most often: the filesystem is mounted read-only, disk is full, or /etc/shadow permissions/ownership are wrong.
In rescue mode, it’s common to forget you mounted / as ro. That error is the system telling you it can’t write the credential database.

4) Can I just edit /etc/shadow directly?

You can, but you shouldn’t unless you really know what you’re doing. Use passwd so it handles hashing and file locking properly.
If you must edit, ensure correct permissions and format—one malformed line can break logins for everyone.

5) I fixed sudoers but sudo still fails after reboot. Why?

Check name resolution delays (NSS/SSSD/LDAP) and group lookup. sudo consults user/group membership and can hang if remote identity providers are unreachable.
If necessary, temporarily ensure local /etc/passwd//etc/group auth paths work.

6) When should I run fsck or xfs_repair?

Run them when you have evidence: mount failures, I/O errors, or journal messages indicating corruption.
For ext filesystems, fsck is normal. For XFS, use xfs_repair (often requires unmount). Don’t run repairs on a mounted filesystem unless the tool explicitly supports it.

7) I’m on a remote VM with no console password entry for LUKS. What now?

That’s a design constraint, not a rescue trick. You need a way to provide the passphrase remotely (virtual console), use a keyfile with secure boot-time retrieval,
or redesign the boot unlock workflow. Rescue mode can help you diagnose, but it can’t conjure a keyboard you don’t have.

8) Should I reinstall GRUB as part of root recovery?

Not unless you have a bootloader symptom. GRUB reinstalls are invasive and can be disruptive on complex boot setups.
If the system reaches a login prompt, your bootloader is not your root-access issue.

9) How do I confirm I mounted the right root before changing anything?

Check /mnt/etc/os-release, /mnt/etc/hostname, and compare disk UUIDs with your inventory.
Also verify the expected boot artifacts exist (/mnt/boot) and that findmnt -R /mnt matches your known partitioning scheme.

10) Can I recover by adding init=/bin/bash at boot?

Sometimes, but it’s blunt and bypasses normal initialization. It can also produce confusing states with encrypted disks and systemd expectations.
Prefer systemd.unit=emergency.target or installer rescue, then do a clean chroot-based repair.

Next steps after you’re back in

Regaining root is not the finish line; it’s regaining the ability to make deliberate changes. Do three things before you declare victory:

1. Stabilize access: Ensure at least two independent admin paths (console + key-based SSH for an admin user). Avoid relying on a single password.
2. Fix the actual root cause: If the trigger was fstab, PAM, initramfs, or an identity provider dependency, address that with a change you can justify from evidence.
3. Leave breadcrumbs: Write down what you changed, why, and how to revert. Put it somewhere your team will actually find during the next incident.

The best rescue is the one you only need once. After the incident, invest in predictable partitioning, tested upgrade procedures, and a policy for root/sudo that’s documented, enforced, and boring.
Boring is good. Boring boots.