testsAndMisc/linux_configuration/scripts/boot_recovery/hooks/05-boot-mounted-guard.hook
Krzysztof Rudnicki 8e24ff093f Add boot-repair: offline recovery for a stale EFI System Partition
On 2026-07-30 a kernel upgrade left the machine in systemd emergency mode
on three consecutive boots. The upgrade replaced /usr/lib/modules/<old>
with <new>, but the new kernel never reached the ESP, so the firmware kept
booting the OLD vmlinuz whose module tree had just been deleted. Nothing
modular could load after pivot-root — including vfat, which meant /boot
itself could not be mounted, local-fs.target failed, and systemd dropped to
an emergency shell. pacman reported the transaction as successful.

Root cause: pacman-git with ParallelDownloads > 1 segfaults every forked
alpm hook child, silently skipping 90-mkinitcpio-install and depmod.

boot-repair is self-contained (no repo libs — it must run in the emergency
shell) and fully offline (missing modules are recovered from
/var/cache/pacman/pkg). It repairs by default; --dry-run inspects.

It detects and fixes: read-only root, noauto/nofail fstab drift, orphaned
kernel files shadowing an unmounted ESP, unloadable vfat, unmounted ESP,
missing modules.dep, ESP kernel != installed tree, stale initramfs, and
ParallelDownloads > 1.

Two pacman hooks make the failure impossible rather than merely
recoverable: 05-boot-mounted-guard aborts a kernel transaction when the
ESP is unmounted (AbortOnFail), and 99-boot-autorepair fixes ESP drift
after the transaction, before the machine is ever rebooted.

Safety: a module tree counts as usable only if it has BOTH vmlinuz and
kernel/, so the metadata-only wreckage this bug leaves — which sorts
newer — is never chosen as the target. Files under /boot are deleted only
while it is not a mountpoint AND contains no EFI/ or loader/; deleting
after mounting would destroy the real kernel.

Verified:
- 31/31 fixture tests (--root against throwaway trees, no privileges)
- 11/11 reversible live tests on the real machine: unmounted the real ESP,
  planted shadow files, confirmed they were deleted and the ESP then
  mounted with EFI/, loader/ and kernel 7.1.5-arch1-2 intact
- Gate proven against a real transaction with the ESP unmounted:
  "failed to commit transaction", no packages upgraded, nothing written
- shellcheck + shfmt clean, pre-commit passing

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 12:48:56 +02:00

21 lines
660 B
Plaintext

# Refuse to install a kernel while the ESP is unmounted.
#
# If /boot is not mounted, the kernel image lands in the ROOT filesystem's
# /boot directory instead of the ESP. The firmware keeps booting the previous
# image, whose module tree this same transaction is about to delete — which
# drops the machine into emergency mode on the next boot.
#
# AbortOnFail makes this a gate, not a warning.
[Trigger]
Type = Path
Operation = Install
Operation = Upgrade
Target = usr/lib/modules/*/vmlinuz
[Action]
Description = Verifying the ESP is mounted before installing a kernel...
When = PreTransaction
Exec = /usr/local/sbin/boot-repair --preflight
AbortOnFail