testsAndMisc/linux_configuration/scripts/boot_recovery
Krzysztof Rudnicki ed1a224ce7 Format last three markdown files; cap billsplit's Android build memory
Brings the repo to prettier-clean across all 525 tracked md/json/yaml files.
The markdown edits are cosmetic: escaping `L*` so it stops rendering as
emphasis, normalising `*em*` to `_em_`, and aligning tables.

billsplit/android/gradle.properties shipped from the Flutter template asking
for `-Xmx8G -XX:MaxMetaspaceSize=4G` — more than this machine has. Lowered to
768m/512m and, more importantly, cut the number of JVMs: Kotlin compiles
in-process, no persistent daemon, no parallel execution, one worker.

Also repairs a regression an earlier prettier pass introduced. That pass
pretty-printed test/fixtures/sample_receipt.json, but
`accepts bare JPK payload and bare JWS token` reads the fixture as raw TEXT
with a regex matching `"data":"..."` with no space after the colon. The
inserted space made the match null and the null-check operator threw.
Comparing the parsed JSON for equality had passed and proved nothing about
that raw-text contract.

The fixture is restored to its original minified bytes — a real e-paragon
arrives as single-line JSON — and test/fixtures/ is now in .prettierignore so
captured payloads stay byte-faithful. The regex also tolerates whitespace now
and asserts the match is non-null with a reason, so the same change would fail
readably instead of crashing. Verified against both the minified and the
pretty-printed fixture.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 13:25:48 +02:00
..
hooks Add boot-repair: offline recovery for a stale EFI System Partition 2026-07-30 12:48:56 +02:00
tests Add boot-repair: offline recovery for a stale EFI System Partition 2026-07-30 12:48:56 +02:00
boot-repair Add boot-repair: offline recovery for a stale EFI System Partition 2026-07-30 12:48:56 +02:00
install.sh Add boot-repair: offline recovery for a stale EFI System Partition 2026-07-30 12:48:56 +02:00
README.md Format last three markdown files; cap billsplit's Android build memory 2026-08-03 13:25:48 +02:00

boot-repair

Offline recovery for a stale EFI System Partition — the failure that dropped this laptop into emergency mode on 2026-07-30.

The failure it fixes

A kernel upgrade replaced /usr/lib/modules/<old> with <new>, but the new kernel image never reached the ESP. The firmware kept booting the old vmlinuz, whose module tree no longer existed, so nothing modular could load after pivot-root. vfat was among the casualties — which meant /boot itself could not be mounted, local-fs.target failed, and systemd dropped to emergency mode. pacman reported the upgrade as successful.

Root cause on this machine: pacman-git with ParallelDownloads > 1 makes every forked alpm hook child segfault (SIGSEGV in libc), silently skipping 90-mkinitcpio-install and depmod.

Install

sudo ./install.sh

Installs:

Path Purpose
/usr/local/sbin/boot-repair the repair tool
/etc/pacman.d/hooks/05-boot-mounted-guard.hook pre-transaction gate
/etc/pacman.d/hooks/99-boot-autorepair.hook post-transaction auto-repair

It lives on the root filesystem by necessity: the ESP is unreachable by definition when this is needed.

Use

sudo boot-repair              # repair now (default)
sudo boot-repair --dry-run    # report only, change nothing

From the systemd emergency shell the root filesystem is already mounted, so:

/usr/local/sbin/boot-repair

No network is used at any point. Missing kernel modules are recovered from /var/cache/pacman/pkg.

Options

Flag Meaning
--dry-run Report problems, change nothing
--preflight Exit non-zero unless the ESP is mounted (used by the gate hook)
--auto Quiet repair for the post-transaction hook
--root DIR Operate on another root (fixtures, or an Arch ISO chroot)
--esp DEVICE Override ESP auto-detection

Exit status: 0 consistent/repaired, 1 problems remain, 2 usage error.

What it checks and repairs

  1. Root filesystem read-only after a failed boot → remount rw
  2. noauto/nofail on /boot in fstab → reset to defaults (backed up). nofail turns a loud emergency drop into a silent boot of a stale kernel, which is strictly worse.
  3. Orphaned kernel files written into the unmounted /boot → delete
  4. vfat unloadable → restore fs/fat + fs/nls from the cached kernel package, depmod, modprobe
  5. ESP not mounted → mount it
  6. Missing modules.dep → depmod
  7. ESP kernel ≠ newest complete module tree → install the kernel
  8. initramfs ≠ that kernel → mkinitcpio -P
  9. ParallelDownloads > 1 → set to 1 (the root cause)

A module tree counts as usable only if it has both vmlinuz and kernel/. The wreckage this bug leaves behind is a directory holding only modules.* metadata, and a naive "newest directory" pick would choose it.

Safety invariants

  • Files under /boot are deleted only while it is not a mountpoint and contains no EFI/ or loader/. Deleting after mounting would destroy the real kernel.
  • EFI/ and loader/ are never written or removed.
  • The running kernel's module tree is never deleted.
  • No network, ever.

Tests

./tests/test_boot_repair.sh     # 31 fixture tests, no root needed
sudo ./tests/live_test.sh       # 11 live tests, reversible

live_test.sh refuses to start unless the system is already consistent, and an EXIT trap remounts the ESP however it ends. It covers what fixtures cannot: the pre-transaction gate, unmounting the real ESP, and deleting shadow files off the real root filesystem while proving the ESP survives.

Verified behaviour

  • 31/31 fixture tests, 11/11 live tests, shellcheck clean
  • Gate proven to abort a real kernel transaction with the ESP unmounted: error: failed to commit transaction, no packages upgraded, nothing written into the unmounted /boot

Limitation

If the running kernel's package is not in /var/cache/pacman/pkg, step 4 is impossible offline. The script says so and prints the Arch-ISO chroot recovery commands rather than guessing.