mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-20 18:12:19 +01:00
On 2026-07-30 a kernel upgrade left the machine in systemd emergency mode on three consecutive boots. The upgrade replaced /usr/lib/modules/<old> with <new>, but the new kernel never reached the ESP, so the firmware kept booting the OLD vmlinuz whose module tree had just been deleted. Nothing modular could load after pivot-root — including vfat, which meant /boot itself could not be mounted, local-fs.target failed, and systemd dropped to an emergency shell. pacman reported the transaction as successful. Root cause: pacman-git with ParallelDownloads > 1 segfaults every forked alpm hook child, silently skipping 90-mkinitcpio-install and depmod. boot-repair is self-contained (no repo libs — it must run in the emergency shell) and fully offline (missing modules are recovered from /var/cache/pacman/pkg). It repairs by default; --dry-run inspects. It detects and fixes: read-only root, noauto/nofail fstab drift, orphaned kernel files shadowing an unmounted ESP, unloadable vfat, unmounted ESP, missing modules.dep, ESP kernel != installed tree, stale initramfs, and ParallelDownloads > 1. Two pacman hooks make the failure impossible rather than merely recoverable: 05-boot-mounted-guard aborts a kernel transaction when the ESP is unmounted (AbortOnFail), and 99-boot-autorepair fixes ESP drift after the transaction, before the machine is ever rebooted. Safety: a module tree counts as usable only if it has BOTH vmlinuz and kernel/, so the metadata-only wreckage this bug leaves — which sorts newer — is never chosen as the target. Files under /boot are deleted only while it is not a mountpoint AND contains no EFI/ or loader/; deleting after mounting would destroy the real kernel. Verified: - 31/31 fixture tests (--root against throwaway trees, no privileges) - 11/11 reversible live tests on the real machine: unmounted the real ESP, planted shadow files, confirmed they were deleted and the ESP then mounted with EFI/, loader/ and kernel 7.1.5-arch1-2 intact - Gate proven against a real transaction with the ESP unmounted: "failed to commit transaction", no packages upgraded, nothing written - shellcheck + shfmt clean, pre-commit passing Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
68 lines
5.7 KiB
JSON
68 lines
5.7 KiB
JSON
{
|
|
"intent": "Provide an offline, self-contained recovery tool that repairs a stale EFI System Partition from the systemd emergency shell, and prevent the failure recurring by gating kernel transactions on a mounted ESP.",
|
|
"scope": [
|
|
"linux_configuration/scripts/boot_recovery/boot-repair",
|
|
"linux_configuration/scripts/boot_recovery/install.sh",
|
|
"linux_configuration/scripts/boot_recovery/README.md",
|
|
"linux_configuration/scripts/boot_recovery/hooks/05-boot-mounted-guard.hook",
|
|
"linux_configuration/scripts/boot_recovery/hooks/99-boot-autorepair.hook",
|
|
"linux_configuration/scripts/boot_recovery/tests/test_boot_repair.sh",
|
|
"linux_configuration/scripts/boot_recovery/tests/live_test.sh",
|
|
"Non-goal: no network access at any point; recovery uses only /var/cache/pacman/pkg",
|
|
"Non-goal: does not source linux_configuration/scripts/lib/common.sh - the tool must stand alone in an emergency shell"
|
|
],
|
|
"changes": [
|
|
"boot-repair: detects and repairs the 2026-07-30 emergency-mode failure chain - stale ESP kernel, gutted module tree, unloadable vfat, unmounted ESP, missing modules.dep, shadow kernel files written into the root filesystem's /boot, noauto/nofail fstab drift, and ParallelDownloads > 1 (the alpm-hook segfault trigger that was the root cause).",
|
|
"Treats a module tree as usable only when BOTH vmlinuz and kernel/ exist, so the metadata-only wreckage the bug leaves behind is never selected as the target kernel.",
|
|
"Safety guards on the only destructive path: files under /boot are deleted only while it is not a mountpoint AND contains no EFI/ or loader/. EFI/ and loader/ are never written or removed.",
|
|
"Recovers vfat offline by extracting fs/fat + fs/nls from the cached kernel package, since mounting the ESP requires the very module the failed upgrade deleted.",
|
|
"05-boot-mounted-guard.hook: PreTransaction gate with AbortOnFail that refuses a kernel install when the ESP is unmounted.",
|
|
"99-boot-autorepair.hook: PostTransaction auto-repair that fixes ESP drift before the machine is ever rebooted, exiting non-zero if it cannot.",
|
|
"--root flag makes the logic testable against fixture trees and usable from an Arch ISO chroot.",
|
|
"install.sh verifies its dependencies, installs to /usr/local/sbin (root filesystem - the ESP is unreachable when this is needed), and runs the installed copy to prove it works."
|
|
],
|
|
"verification": [
|
|
{
|
|
"command": "./tests/test_boot_repair.sh",
|
|
"result": "pass",
|
|
"evidence": "31 passed, 0 failed. Covers healthy system, stale ESP kernel detect+repair, gutted-tree rejection, newest-complete selection, fstab noauto/nofail reset with backup, shadow-file deletion, the refuse-to-delete-a-real-ESP guard, ParallelDownloads fix, missing modules.dep, no-kernel guidance, CLI contract."
|
|
},
|
|
{
|
|
"command": "sudo ./tests/live_test.sh",
|
|
"result": "pass",
|
|
"evidence": "11 passed, 0 failed on the real machine. Unmounted the real ESP, confirmed the preflight gate opens when mounted and closes when not, confirmed dry-run writes nothing, planted shadow files and confirmed they were deleted and the ESP then mounted with EFI/, loader/ and kernel 7.1.5-arch1-2 intact. EXIT trap remounts the ESP regardless of outcome."
|
|
},
|
|
{
|
|
"command": "sudo pacman -U /var/cache/pacman/pkg/linux-7.1.5.arch1-2-x86_64.pkg.tar.zst (ESP mounted)",
|
|
"result": "pass",
|
|
"evidence": "(1/3) Verifying the ESP is mounted before installing a kernel... boot-repair: ESP mounted at /boot - proceeding. (6/6) Verifying the ESP received the new kernel... Both hooks fired; post-transaction hook silent, meaning nothing needed repair."
|
|
},
|
|
{
|
|
"command": "sudo umount /boot && sudo pacman -U .../linux-7.1.5.arch1-2-x86_64.pkg.tar.zst",
|
|
"result": "pass",
|
|
"evidence": "==> boot-repair: ABORTING TRANSACTION / error: failed to commit transaction (failed to run transaction hooks) / Errors occurred, no packages were upgraded. Nothing was written into the unmounted /boot. The gate held."
|
|
},
|
|
{
|
|
"command": "shellcheck boot-repair install.sh tests/*.sh",
|
|
"result": "pass",
|
|
"evidence": "Clean. SC2034/SC2155/SC2115/SC2001 fixed at source rather than suppressed."
|
|
},
|
|
{
|
|
"command": "shfmt -d -i 2 -ci -sr -s boot-repair install.sh tests/*.sh",
|
|
"result": "pass",
|
|
"evidence": "Clean, matching the options used by scripts/meta/shell_check.sh."
|
|
}
|
|
],
|
|
"risks": [
|
|
"The PreTransaction hook uses AbortOnFail, so a bug in boot-repair --preflight would block all kernel upgrades. Mitigated by the live test proving both the open and closed paths, and by --preflight being a single findmnt check.",
|
|
"The 99-boot-autorepair hook is itself forked by alpm, so it is subject to the same ParallelDownloads segfault it compensates for. This is why boot-repair also enforces ParallelDownloads = 1 rather than relying on the hook alone.",
|
|
"Shadow-file deletion is destructive. Guarded by requiring /boot to be unmounted AND free of EFI/ and loader/; both guards are covered by fixture and live tests.",
|
|
"If the running kernel's package is absent from /var/cache/pacman/pkg, vfat cannot be restored offline. The script reports this and prints Arch ISO chroot recovery steps instead of guessing."
|
|
],
|
|
"rollback": [
|
|
"rm /etc/pacman.d/hooks/05-boot-mounted-guard.hook /etc/pacman.d/hooks/99-boot-autorepair.hook to remove the gates.",
|
|
"rm /usr/local/sbin/boot-repair to remove the tool.",
|
|
"After rollback validate that a kernel transaction still completes: sudo pacman -U /var/cache/pacman/pkg/linux-*.pkg.tar.zst, then confirm findmnt /boot shows the ESP and the version in /boot/vmlinuz-linux has a matching /usr/lib/modules/<ver>/ directory."
|
|
]
|
|
}
|