testsAndMisc/docs/superpowers/evidence/boot-repair-esp-recovery-20260730.json
Krzysztof Rudnicki 8e24ff093f Add boot-repair: offline recovery for a stale EFI System Partition
On 2026-07-30 a kernel upgrade left the machine in systemd emergency mode
on three consecutive boots. The upgrade replaced /usr/lib/modules/<old>
with <new>, but the new kernel never reached the ESP, so the firmware kept
booting the OLD vmlinuz whose module tree had just been deleted. Nothing
modular could load after pivot-root — including vfat, which meant /boot
itself could not be mounted, local-fs.target failed, and systemd dropped to
an emergency shell. pacman reported the transaction as successful.

Root cause: pacman-git with ParallelDownloads > 1 segfaults every forked
alpm hook child, silently skipping 90-mkinitcpio-install and depmod.

boot-repair is self-contained (no repo libs — it must run in the emergency
shell) and fully offline (missing modules are recovered from
/var/cache/pacman/pkg). It repairs by default; --dry-run inspects.

It detects and fixes: read-only root, noauto/nofail fstab drift, orphaned
kernel files shadowing an unmounted ESP, unloadable vfat, unmounted ESP,
missing modules.dep, ESP kernel != installed tree, stale initramfs, and
ParallelDownloads > 1.

Two pacman hooks make the failure impossible rather than merely
recoverable: 05-boot-mounted-guard aborts a kernel transaction when the
ESP is unmounted (AbortOnFail), and 99-boot-autorepair fixes ESP drift
after the transaction, before the machine is ever rebooted.

Safety: a module tree counts as usable only if it has BOTH vmlinuz and
kernel/, so the metadata-only wreckage this bug leaves — which sorts
newer — is never chosen as the target. Files under /boot are deleted only
while it is not a mountpoint AND contains no EFI/ or loader/; deleting
after mounting would destroy the real kernel.

Verified:
- 31/31 fixture tests (--root against throwaway trees, no privileges)
- 11/11 reversible live tests on the real machine: unmounted the real ESP,
  planted shadow files, confirmed they were deleted and the ESP then
  mounted with EFI/, loader/ and kernel 7.1.5-arch1-2 intact
- Gate proven against a real transaction with the ESP unmounted:
  "failed to commit transaction", no packages upgraded, nothing written
- shellcheck + shfmt clean, pre-commit passing

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 12:48:56 +02:00

68 lines
5.7 KiB
JSON

{
"intent": "Provide an offline, self-contained recovery tool that repairs a stale EFI System Partition from the systemd emergency shell, and prevent the failure recurring by gating kernel transactions on a mounted ESP.",
"scope": [
"linux_configuration/scripts/boot_recovery/boot-repair",
"linux_configuration/scripts/boot_recovery/install.sh",
"linux_configuration/scripts/boot_recovery/README.md",
"linux_configuration/scripts/boot_recovery/hooks/05-boot-mounted-guard.hook",
"linux_configuration/scripts/boot_recovery/hooks/99-boot-autorepair.hook",
"linux_configuration/scripts/boot_recovery/tests/test_boot_repair.sh",
"linux_configuration/scripts/boot_recovery/tests/live_test.sh",
"Non-goal: no network access at any point; recovery uses only /var/cache/pacman/pkg",
"Non-goal: does not source linux_configuration/scripts/lib/common.sh - the tool must stand alone in an emergency shell"
],
"changes": [
"boot-repair: detects and repairs the 2026-07-30 emergency-mode failure chain - stale ESP kernel, gutted module tree, unloadable vfat, unmounted ESP, missing modules.dep, shadow kernel files written into the root filesystem's /boot, noauto/nofail fstab drift, and ParallelDownloads > 1 (the alpm-hook segfault trigger that was the root cause).",
"Treats a module tree as usable only when BOTH vmlinuz and kernel/ exist, so the metadata-only wreckage the bug leaves behind is never selected as the target kernel.",
"Safety guards on the only destructive path: files under /boot are deleted only while it is not a mountpoint AND contains no EFI/ or loader/. EFI/ and loader/ are never written or removed.",
"Recovers vfat offline by extracting fs/fat + fs/nls from the cached kernel package, since mounting the ESP requires the very module the failed upgrade deleted.",
"05-boot-mounted-guard.hook: PreTransaction gate with AbortOnFail that refuses a kernel install when the ESP is unmounted.",
"99-boot-autorepair.hook: PostTransaction auto-repair that fixes ESP drift before the machine is ever rebooted, exiting non-zero if it cannot.",
"--root flag makes the logic testable against fixture trees and usable from an Arch ISO chroot.",
"install.sh verifies its dependencies, installs to /usr/local/sbin (root filesystem - the ESP is unreachable when this is needed), and runs the installed copy to prove it works."
],
"verification": [
{
"command": "./tests/test_boot_repair.sh",
"result": "pass",
"evidence": "31 passed, 0 failed. Covers healthy system, stale ESP kernel detect+repair, gutted-tree rejection, newest-complete selection, fstab noauto/nofail reset with backup, shadow-file deletion, the refuse-to-delete-a-real-ESP guard, ParallelDownloads fix, missing modules.dep, no-kernel guidance, CLI contract."
},
{
"command": "sudo ./tests/live_test.sh",
"result": "pass",
"evidence": "11 passed, 0 failed on the real machine. Unmounted the real ESP, confirmed the preflight gate opens when mounted and closes when not, confirmed dry-run writes nothing, planted shadow files and confirmed they were deleted and the ESP then mounted with EFI/, loader/ and kernel 7.1.5-arch1-2 intact. EXIT trap remounts the ESP regardless of outcome."
},
{
"command": "sudo pacman -U /var/cache/pacman/pkg/linux-7.1.5.arch1-2-x86_64.pkg.tar.zst (ESP mounted)",
"result": "pass",
"evidence": "(1/3) Verifying the ESP is mounted before installing a kernel... boot-repair: ESP mounted at /boot - proceeding. (6/6) Verifying the ESP received the new kernel... Both hooks fired; post-transaction hook silent, meaning nothing needed repair."
},
{
"command": "sudo umount /boot && sudo pacman -U .../linux-7.1.5.arch1-2-x86_64.pkg.tar.zst",
"result": "pass",
"evidence": "==> boot-repair: ABORTING TRANSACTION / error: failed to commit transaction (failed to run transaction hooks) / Errors occurred, no packages were upgraded. Nothing was written into the unmounted /boot. The gate held."
},
{
"command": "shellcheck boot-repair install.sh tests/*.sh",
"result": "pass",
"evidence": "Clean. SC2034/SC2155/SC2115/SC2001 fixed at source rather than suppressed."
},
{
"command": "shfmt -d -i 2 -ci -sr -s boot-repair install.sh tests/*.sh",
"result": "pass",
"evidence": "Clean, matching the options used by scripts/meta/shell_check.sh."
}
],
"risks": [
"The PreTransaction hook uses AbortOnFail, so a bug in boot-repair --preflight would block all kernel upgrades. Mitigated by the live test proving both the open and closed paths, and by --preflight being a single findmnt check.",
"The 99-boot-autorepair hook is itself forked by alpm, so it is subject to the same ParallelDownloads segfault it compensates for. This is why boot-repair also enforces ParallelDownloads = 1 rather than relying on the hook alone.",
"Shadow-file deletion is destructive. Guarded by requiring /boot to be unmounted AND free of EFI/ and loader/; both guards are covered by fixture and live tests.",
"If the running kernel's package is absent from /var/cache/pacman/pkg, vfat cannot be restored offline. The script reports this and prints Arch ISO chroot recovery steps instead of guessing."
],
"rollback": [
"rm /etc/pacman.d/hooks/05-boot-mounted-guard.hook /etc/pacman.d/hooks/99-boot-autorepair.hook to remove the gates.",
"rm /usr/local/sbin/boot-repair to remove the tool.",
"After rollback validate that a kernel transaction still completes: sudo pacman -U /var/cache/pacman/pkg/linux-*.pkg.tar.zst, then confirm findmnt /boot shows the ESP and the version in /boot/vmlinuz-linux has a matching /usr/lib/modules/<ver>/ directory."
]
}