testsAndMisc/linux_configuration/boot_recovery
Krzysztof kuhy Rudnicki 0fb3e3eee2 Cap extensionless scripts, and split the one that was hiding
check_file_length.sh required a dot in the filename, so every
extensionless executable was silently uncapped. is_source_file() now
reads the first two bytes and accepts a shebang, which catches real
scripts while still skipping data files like i3/config.

That immediately surfaced boot_recovery/boot-repair at 756 lines --
three times the largest capped shell file in the repo, uncapped purely
because its name has no dot in it. Split along its own section markers
into lib/{discovery,repair_fs,repair_kernel,modes}.sh, leaving the entry
script at 218.

The report block stays inline on purpose. It is top-level code with exit
calls rather than function definitions, and sourcing it changed control
flow: 30 of 42 tests failed. That is recorded in the lib headers so the
next reader does not retry it.

No vmbox pass for this one, deliberately. boot-repair is UEFI/ESP tooling
and the guest is BIOS/GRUB, so a sandbox green would be actively
misleading -- the vmbox skill says so itself. The evidence is the fixture
suite, which drives the real script against synthetic root trees: 42
passed before, 42 passed after.

verify_shell_split.sh reports 31 of 32 functions byte-identical. The one
difference is clean_shadow_files, where quoting $ESP_MOUNTPOINT fixed a
genuine SC2086 that only became visible once the function left the
monolith -- shellcheck could no longer see the readonly declaration and
stopped assuming the expansion was safe.

lib/tests/run_all.sh delegates to that same suite rather than standing up
a second, weaker harness: these libraries need the entry script's strict
mode and globals, so testing them in isolation would test a fiction.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJrqKERAo5QUeEHWS5Asgp
2026-08-23 19:13:40 +02:00
..
hooks Dissolve linux_configuration/scripts/ and single_use/ 2026-08-23 18:53:30 +02:00
lib Cap extensionless scripts, and split the one that was hiding 2026-08-23 19:13:40 +02:00
tests Dissolve linux_configuration/scripts/ and single_use/ 2026-08-23 18:53:30 +02:00
boot-repair Cap extensionless scripts, and split the one that was hiding 2026-08-23 19:13:40 +02:00
install.sh Dissolve linux_configuration/scripts/ and single_use/ 2026-08-23 18:53:30 +02:00
README.md Dissolve linux_configuration/scripts/ and single_use/ 2026-08-23 18:53:30 +02:00

boot-repair

Offline recovery for a stale EFI System Partition — the failure that dropped this laptop into emergency mode on 2026-07-30.

The failure it fixes

A kernel upgrade replaced /usr/lib/modules/<old> with <new>, but the new kernel image never reached the ESP. The firmware kept booting the old vmlinuz, whose module tree no longer existed, so nothing modular could load after pivot-root. vfat was among the casualties — which meant /boot itself could not be mounted, local-fs.target failed, and systemd dropped to emergency mode. pacman reported the upgrade as successful.

Root cause on this machine: pacman-git with ParallelDownloads > 1 makes every forked alpm hook child segfault (SIGSEGV in libc), silently skipping 90-mkinitcpio-install and depmod.

Install

sudo ./install.sh

Installs:

Path Purpose
/usr/local/sbin/boot-repair the repair tool
/etc/pacman.d/hooks/05-boot-mounted-guard.hook pre-transaction gate
/etc/pacman.d/hooks/99-boot-autorepair.hook post-transaction auto-repair

It lives on the root filesystem by necessity: the ESP is unreachable by definition when this is needed.

Use

sudo boot-repair              # repair now (default)
sudo boot-repair --dry-run    # report only, change nothing

From the systemd emergency shell the root filesystem is already mounted, so:

/usr/local/sbin/boot-repair

No network is used at any point. Missing kernel modules are recovered from /var/cache/pacman/pkg.

Options

Flag Meaning
--dry-run Report problems, change nothing
--preflight Exit non-zero unless the ESP is mounted (used by the gate hook)
--auto Quiet repair for the post-transaction hook
--root DIR Operate on another root (fixtures, or an Arch ISO chroot)
--esp DEVICE Override ESP auto-detection

Exit status: 0 consistent/repaired, 1 problems remain, 2 usage error.

What it checks and repairs

  1. Root filesystem read-only after a failed boot → remount rw
  2. noauto/nofail on /boot in fstab → reset to defaults (backed up). nofail turns a loud emergency drop into a silent boot of a stale kernel, which is strictly worse.
  3. Orphaned kernel files written into the unmounted /boot → delete
  4. vfat unloadable → restore fs/fat + fs/nls from the cached kernel package, depmod, modprobe
  5. ESP not mounted → mount it
  6. Missing modules.dep → depmod
  7. ESP kernel ≠ newest complete module tree → install the kernel
  8. initramfs ≠ that kernel → mkinitcpio -P
  9. ParallelDownloads > 1 → set to 1 (the root cause)

A module tree counts as usable only if it has both vmlinuz and kernel/. The wreckage this bug leaves behind is a directory holding only modules.* metadata, and a naive "newest directory" pick would choose it.

Safety invariants

  • Files under /boot are deleted only while it is not a mountpoint and contains no EFI/ or loader/. Deleting after mounting would destroy the real kernel.
  • EFI/ and loader/ are never written or removed.
  • The running kernel's module tree is never deleted.
  • No network, ever.

Tests

./tests/test_boot_repair.sh     # 31 fixture tests, no root needed
sudo ./tests/live_test.sh       # 11 live tests, reversible

live_test.sh refuses to start unless the system is already consistent, and an EXIT trap remounts the ESP however it ends. It covers what fixtures cannot: the pre-transaction gate, unmounting the real ESP, and deleting shadow files off the real root filesystem while proving the ESP survives.

Verified behaviour

  • 31/31 fixture tests, 11/11 live tests, shellcheck clean
  • Gate proven to abort a real kernel transaction with the ESP unmounted: error: failed to commit transaction, no packages upgraded, nothing written into the unmounted /boot

Limitation

If the running kernel's package is not in /var/cache/pacman/pkg, step 4 is impossible offline. The script says so and prints the Arch-ISO chroot recovery commands rather than guessing.