testsAndMisc/linux_configuration/boot_recovery/README.md
Krzysztof kuhy Rudnicki 4f2b0ccdf7 Dissolve linux_configuration/scripts/ and single_use/
Two levels of pure taxonomy carrying no information, and a 2-directory
cap leaves no room for them. Categories now sit directly under
linux_configuration/: features, fixes, utils, misc, fresh-install,
installers, diagnostics, gaming, boot_recovery, periodic_background, lib,
meta, tests. A script's own lib/ and tests/ are exempt from the cap, so
those stay put.

Depth violations 248 -> 139; roughly 40 of the remainder are inside
digital_wellbeing and hosts, deferred until the guard-lib repoint.

The moves are the easy half. What breaks is every path that counted
levels, and three classes of it had to be found by resolving paths rather
than reading diffs:

  - 52 relative `source` lines for lib/common.sh and lib/android.sh. The
    correct prefix differs per file: 40 were ../../ and 12 were ../../../,
    so a blanket replace would have fixed one set and broken the other.
  - REPO_ROOT in 7 fixes/lib/tests harnesses and setup_syncyomi_guard.sh,
    each walking up four levels from a directory now two levels shallower.
    This caused a real regression -- the fixes suite went red with
    "log_warn: command not found" -- caught by running it and comparing
    against the pre-move baseline worktree.
  - conftest.py's sys.path entries, which took the pytest gate down with
    14 collection errors, and the `# shellcheck source=` directives, which
    took shellcheck down with SC1091.

16 symlinks in $HOME broke, including the entire i3blocks bar. All
repointed, and cpu_monitor/memory/disk/time/volume were run to confirm
they emit real output rather than merely resolving.

Verified: 1798 tests pass at 100% coverage, 8 of 9 lib/tests suites pass
(the ninth needs the coverage jail, identically at baseline), pre-commit
is clean, and the two untracked secret configs are still ignored after
their gitignore paths changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJrqKERAo5QUeEHWS5Asgp
2026-08-23 18:53:30 +02:00

116 lines
4.5 KiB
Markdown

# boot-repair
Offline recovery for a stale EFI System Partition — the failure that dropped
this laptop into emergency mode on 2026-07-30.
## The failure it fixes
A kernel upgrade replaced `/usr/lib/modules/<old>` with `<new>`, but the new
kernel image never reached the ESP. The firmware kept booting the **old**
`vmlinuz`, whose module tree no longer existed, so nothing modular could load
after pivot-root. `vfat` was among the casualties — which meant `/boot` itself
could not be mounted, `local-fs.target` failed, and systemd dropped to
emergency mode. pacman reported the upgrade as successful.
**Root cause on this machine:** `pacman-git` with `ParallelDownloads > 1` makes
every forked alpm hook child segfault (SIGSEGV in libc), silently skipping
`90-mkinitcpio-install` and `depmod`.
## Install
```bash
sudo ./install.sh
```
Installs:
| Path | Purpose |
| ------------------------------------------------ | ---------------------------- |
| `/usr/local/sbin/boot-repair` | the repair tool |
| `/etc/pacman.d/hooks/05-boot-mounted-guard.hook` | pre-transaction **gate** |
| `/etc/pacman.d/hooks/99-boot-autorepair.hook` | post-transaction auto-repair |
It lives on the **root** filesystem by necessity: the ESP is unreachable by
definition when this is needed.
## Use
```bash
sudo boot-repair # repair now (default)
sudo boot-repair --dry-run # report only, change nothing
```
From the systemd emergency shell the root filesystem is already mounted, so:
```
/usr/local/sbin/boot-repair
```
No network is used at any point. Missing kernel modules are recovered from
`/var/cache/pacman/pkg`.
### Options
| Flag | Meaning |
| -------------- | --------------------------------------------------------------- |
| `--dry-run` | Report problems, change nothing |
| `--preflight` | Exit non-zero unless the ESP is mounted (used by the gate hook) |
| `--auto` | Quiet repair for the post-transaction hook |
| `--root DIR` | Operate on another root (fixtures, or an Arch ISO chroot) |
| `--esp DEVICE` | Override ESP auto-detection |
Exit status: `0` consistent/repaired, `1` problems remain, `2` usage error.
## What it checks and repairs
1. Root filesystem read-only after a failed boot → remount rw
2. `noauto`/`nofail` on `/boot` in fstab → reset to `defaults` (backed up).
`nofail` turns a loud emergency drop into a _silent_ boot of a stale kernel,
which is strictly worse.
3. Orphaned kernel files written into the unmounted `/boot` → delete
4. `vfat` unloadable → restore `fs/fat` + `fs/nls` from the cached kernel
package, `depmod`, `modprobe`
5. ESP not mounted → mount it
6. Missing `modules.dep` → `depmod`
7. ESP kernel ≠ newest complete module tree → install the kernel
8. initramfs ≠ that kernel → `mkinitcpio -P`
9. `ParallelDownloads > 1` → set to `1` (the root cause)
A module tree counts as usable only if it has **both** `vmlinuz` and `kernel/`.
The wreckage this bug leaves behind is a directory holding only `modules.*`
metadata, and a naive "newest directory" pick would choose it.
## Safety invariants
- Files under `/boot` are deleted **only** while it is not a mountpoint **and**
contains no `EFI/` or `loader/`. Deleting after mounting would destroy the
real kernel.
- `EFI/` and `loader/` are never written or removed.
- The running kernel's module tree is never deleted.
- No network, ever.
## Tests
```bash
./tests/test_boot_repair.sh # 31 fixture tests, no root needed
sudo ./tests/live_test.sh # 11 live tests, reversible
```
`live_test.sh` refuses to start unless the system is already consistent, and an
EXIT trap remounts the ESP however it ends. It covers what fixtures cannot:
the pre-transaction gate, unmounting the real ESP, and deleting shadow files
off the real root filesystem while proving the ESP survives.
### Verified behaviour
- 31/31 fixture tests, 11/11 live tests, `shellcheck` clean
- Gate proven to abort a real kernel transaction with the ESP unmounted:
`error: failed to commit transaction`, _no packages upgraded_, nothing
written into the unmounted `/boot`
## Limitation
If the running kernel's package is not in `/var/cache/pacman/pkg`, step 4 is
impossible offline. The script says so and prints the Arch-ISO chroot recovery
commands rather than guessing.