mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 19:27:03 +01:00
Krzysztof Rudnicki
79829ec8a2
pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned
transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on
retry. This investigates it and repairs the real defects found on the way.
The stall is NOT fixed, and deliberately so: no verified defect was found, so
nothing pacman-side was changed.
- Not guard-lib. /etc/guard-lib/targets was empty, so that hook script
provably exits in milliseconds. The failures share POSITION (first hook of
the transaction), not identity: one stall hit the hosts hook back when that
one ran first.
- Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB
of memory pressure) produced zero stalls. Against the historical 4-in-56
rate that is ~0.9% likely, so the trigger is conditional on something a
synthetic load does not recreate. Memory pressure is untested at the load
that actually occurred - not refuted.
So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and
its --watch mode passively captures /proc/<pid>/{stack,wchan,syscall}, the
process tree, meminfo and PSI from the next REAL stall. No pacman-side fix
should ship before that dump exists.
Three real defects were found and fixed:
1. The deployed pacman wrapper was 19916 B against a 27510 B source, never
sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used.
It hid because check_pacman_wrapper() only tested that files EXIST. Both
installers now write a drift manifest covering source AND installed hashes;
the checker replays it with sha256sum -c and fails closed when it cannot
verify. Manifest paths are absolute - relative ones resolve to nothing under
systemd's cwd=/ and would have looped reinstalling forever.
2. The guard-lib migration (66c4698) was never finished here: guard-lib was
installed with zero registered targets while the legacy hosts-guard layer did
all the work, so every transaction ran two PreTransaction hooks and the first
one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts,
nsswitch and resolved, retires the legacy hooks and units, and has a
--rollback that was actually exercised. Plugins install to a stable path
because guardctl records absolute plugin paths. Side effect: one hook pair
instead of two, transactions ~5-6s -> ~2-3s.
3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh
makes the overlap impossible: one system-wide flock shared between root and
the user (a per-user lock would give root's pacman and the user's builds
separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only
- read-only queries stay instant - plus makepkg_capped and the coverage hook.
Always fails open.
Two bugs surfaced only by running --rollback, both of which briefly left
/etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while
legacy watchers only fire on PathModified, and `((i > 20)) && break` left the
loop body's status at 1, which under set -e aborted the rollback silently - but
only when the loop actually ran, so --dry-run looked healthy.
Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not
exist here). Not wired: Flutter/Gradle builds, which have no single entry point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
21 lines
2.7 KiB
JSON
21 lines
2.7 KiB
JSON
{
|
|
"title": "Diagnose the pacman first-hook stall; fix wrapper drift, finish the guard-lib migration, serialise heavy jobs",
|
|
"objective": "pacman intermittently freezes at its first PreTransaction hook (5 abandoned transactions since 2026-07-29, stalls of 4 minutes to overnight). Establish what is actually failing rather than assuming, then repair the real deployment defects found along the way. Success means: the stall is either root-caused or instrumented so the next real occurrence is captured; the deployed pacman wrapper matches its source and drift can no longer hide; there is one guard system for /etc/hosts instead of two half-installed ones; and concurrent heavy jobs on this 7.6 GB machine are prevented by a mechanism rather than by a human remembering a rule.",
|
|
"acceptance_criteria": [
|
|
"The guard-lib hook is proven not to be the cause (its target list is empty, so the script body cannot execute)",
|
|
"A repro harness drives real pacman transactions and captures /proc/<pid>/stack, wchan, syscall, process tree, meminfo and PSI on a stall",
|
|
"A passive --watch mode captures the same evidence from a real (not synthetic) stall",
|
|
"Deployed pacman_wrapper matches the repo source, and check_and_enable_services.sh detects content drift in both directions (source moved on / deployed tampered) and fails closed when unverifiable",
|
|
"hosts, nsswitch and resolved are guardctl file-guard instances; the legacy pacman hooks and units are retired; /etc/hosts keeps chattr +i, its read-only bind mount, its exact contents and its blocking behaviour",
|
|
"The migration has a --rollback that restores the legacy layer INCLUDING the immutable flag, verified by running it",
|
|
"A heavy-job lock serialises pacman transactions, makepkg builds and the coverage hook, names the current holder when it makes something wait, and always fails open"
|
|
],
|
|
"out_of_scope": [
|
|
"Shipping a pacman-side fix for the stall itself: it could not be reproduced, so there is no verified defect to fix yet",
|
|
"The shutdown-schedule guard instance: /etc/shutdown-schedule.conf does not exist on this machine, so registering it would mean installing the midnight-shutdown feature rather than finishing this migration",
|
|
"Wiring the lock into Flutter/Gradle builds: the repo has no single build entry point to wrap; the lock is available as a CLI for those",
|
|
"Anything on the desktop machine, which is not reachable from this session"
|
|
],
|
|
"verifier": "sudo diagnose_pacman_hook_stall.sh --runs N [--with-load]; sudo migrate_hosts_guard_to_guard_lib.sh --status/--rollback; sudo check_and_enable_services.sh --status with deliberately induced drift; timed heavy_job_lock.sh contention tests; pre-commit run --files <changed>"
|
|
}
|