mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-15 06:30:51 +01:00
Krzysztof Rudnicki
8e24ff093f
On 2026-07-30 a kernel upgrade left the machine in systemd emergency mode on three consecutive boots. The upgrade replaced /usr/lib/modules/<old> with <new>, but the new kernel never reached the ESP, so the firmware kept booting the OLD vmlinuz whose module tree had just been deleted. Nothing modular could load after pivot-root — including vfat, which meant /boot itself could not be mounted, local-fs.target failed, and systemd dropped to an emergency shell. pacman reported the transaction as successful. Root cause: pacman-git with ParallelDownloads > 1 segfaults every forked alpm hook child, silently skipping 90-mkinitcpio-install and depmod. boot-repair is self-contained (no repo libs — it must run in the emergency shell) and fully offline (missing modules are recovered from /var/cache/pacman/pkg). It repairs by default; --dry-run inspects. It detects and fixes: read-only root, noauto/nofail fstab drift, orphaned kernel files shadowing an unmounted ESP, unloadable vfat, unmounted ESP, missing modules.dep, ESP kernel != installed tree, stale initramfs, and ParallelDownloads > 1. Two pacman hooks make the failure impossible rather than merely recoverable: 05-boot-mounted-guard aborts a kernel transaction when the ESP is unmounted (AbortOnFail), and 99-boot-autorepair fixes ESP drift after the transaction, before the machine is ever rebooted. Safety: a module tree counts as usable only if it has BOTH vmlinuz and kernel/, so the metadata-only wreckage this bug leaves — which sorts newer — is never chosen as the target. Files under /boot are deleted only while it is not a mountpoint AND contains no EFI/ or loader/; deleting after mounting would destroy the real kernel. Verified: - 31/31 fixture tests (--root against throwaway trees, no privileges) - 11/11 reversible live tests on the real machine: unmounted the real ESP, planted shadow files, confirmed they were deleted and the ESP then mounted with EFI/, loader/ and kernel 7.1.5-arch1-2 intact - Gate proven against a real transaction with the ESP unmounted: "failed to commit transaction", no packages upgraded, nothing written - shellcheck + shfmt clean, pre-commit passing Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
116 lines
4.1 KiB
Markdown
116 lines
4.1 KiB
Markdown
# boot-repair
|
|
|
|
Offline recovery for a stale EFI System Partition — the failure that dropped
|
|
this laptop into emergency mode on 2026-07-30.
|
|
|
|
## The failure it fixes
|
|
|
|
A kernel upgrade replaced `/usr/lib/modules/<old>` with `<new>`, but the new
|
|
kernel image never reached the ESP. The firmware kept booting the **old**
|
|
`vmlinuz`, whose module tree no longer existed, so nothing modular could load
|
|
after pivot-root. `vfat` was among the casualties — which meant `/boot` itself
|
|
could not be mounted, `local-fs.target` failed, and systemd dropped to
|
|
emergency mode. pacman reported the upgrade as successful.
|
|
|
|
**Root cause on this machine:** `pacman-git` with `ParallelDownloads > 1` makes
|
|
every forked alpm hook child segfault (SIGSEGV in libc), silently skipping
|
|
`90-mkinitcpio-install` and `depmod`.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
sudo ./install.sh
|
|
```
|
|
|
|
Installs:
|
|
|
|
| Path | Purpose |
|
|
|---|---|
|
|
| `/usr/local/sbin/boot-repair` | the repair tool |
|
|
| `/etc/pacman.d/hooks/05-boot-mounted-guard.hook` | pre-transaction **gate** |
|
|
| `/etc/pacman.d/hooks/99-boot-autorepair.hook` | post-transaction auto-repair |
|
|
|
|
It lives on the **root** filesystem by necessity: the ESP is unreachable by
|
|
definition when this is needed.
|
|
|
|
## Use
|
|
|
|
```bash
|
|
sudo boot-repair # repair now (default)
|
|
sudo boot-repair --dry-run # report only, change nothing
|
|
```
|
|
|
|
From the systemd emergency shell the root filesystem is already mounted, so:
|
|
|
|
```
|
|
/usr/local/sbin/boot-repair
|
|
```
|
|
|
|
No network is used at any point. Missing kernel modules are recovered from
|
|
`/var/cache/pacman/pkg`.
|
|
|
|
### Options
|
|
|
|
| Flag | Meaning |
|
|
|---|---|
|
|
| `--dry-run` | Report problems, change nothing |
|
|
| `--preflight` | Exit non-zero unless the ESP is mounted (used by the gate hook) |
|
|
| `--auto` | Quiet repair for the post-transaction hook |
|
|
| `--root DIR` | Operate on another root (fixtures, or an Arch ISO chroot) |
|
|
| `--esp DEVICE` | Override ESP auto-detection |
|
|
|
|
Exit status: `0` consistent/repaired, `1` problems remain, `2` usage error.
|
|
|
|
## What it checks and repairs
|
|
|
|
1. Root filesystem read-only after a failed boot → remount rw
|
|
2. `noauto`/`nofail` on `/boot` in fstab → reset to `defaults` (backed up).
|
|
`nofail` turns a loud emergency drop into a *silent* boot of a stale kernel,
|
|
which is strictly worse.
|
|
3. Orphaned kernel files written into the unmounted `/boot` → delete
|
|
4. `vfat` unloadable → restore `fs/fat` + `fs/nls` from the cached kernel
|
|
package, `depmod`, `modprobe`
|
|
5. ESP not mounted → mount it
|
|
6. Missing `modules.dep` → `depmod`
|
|
7. ESP kernel ≠ newest complete module tree → install the kernel
|
|
8. initramfs ≠ that kernel → `mkinitcpio -P`
|
|
9. `ParallelDownloads > 1` → set to `1` (the root cause)
|
|
|
|
A module tree counts as usable only if it has **both** `vmlinuz` and `kernel/`.
|
|
The wreckage this bug leaves behind is a directory holding only `modules.*`
|
|
metadata, and a naive "newest directory" pick would choose it.
|
|
|
|
## Safety invariants
|
|
|
|
- Files under `/boot` are deleted **only** while it is not a mountpoint **and**
|
|
contains no `EFI/` or `loader/`. Deleting after mounting would destroy the
|
|
real kernel.
|
|
- `EFI/` and `loader/` are never written or removed.
|
|
- The running kernel's module tree is never deleted.
|
|
- No network, ever.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
./tests/test_boot_repair.sh # 31 fixture tests, no root needed
|
|
sudo ./tests/live_test.sh # 11 live tests, reversible
|
|
```
|
|
|
|
`live_test.sh` refuses to start unless the system is already consistent, and an
|
|
EXIT trap remounts the ESP however it ends. It covers what fixtures cannot:
|
|
the pre-transaction gate, unmounting the real ESP, and deleting shadow files
|
|
off the real root filesystem while proving the ESP survives.
|
|
|
|
### Verified behaviour
|
|
|
|
- 31/31 fixture tests, 11/11 live tests, `shellcheck` clean
|
|
- Gate proven to abort a real kernel transaction with the ESP unmounted:
|
|
`error: failed to commit transaction`, *no packages upgraded*, nothing
|
|
written into the unmounted `/boot`
|
|
|
|
## Limitation
|
|
|
|
If the running kernel's package is not in `/var/cache/pacman/pkg`, step 4 is
|
|
impossible offline. The script says so and prints the Arch-ISO chroot recovery
|
|
commands rather than guessing.
|