mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 19:27:03 +01:00
Krzysztof Rudnicki
79829ec8a2
pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned
transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on
retry. This investigates it and repairs the real defects found on the way.
The stall is NOT fixed, and deliberately so: no verified defect was found, so
nothing pacman-side was changed.
- Not guard-lib. /etc/guard-lib/targets was empty, so that hook script
provably exits in milliseconds. The failures share POSITION (first hook of
the transaction), not identity: one stall hit the hosts hook back when that
one ran first.
- Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB
of memory pressure) produced zero stalls. Against the historical 4-in-56
rate that is ~0.9% likely, so the trigger is conditional on something a
synthetic load does not recreate. Memory pressure is untested at the load
that actually occurred - not refuted.
So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and
its --watch mode passively captures /proc/<pid>/{stack,wchan,syscall}, the
process tree, meminfo and PSI from the next REAL stall. No pacman-side fix
should ship before that dump exists.
Three real defects were found and fixed:
1. The deployed pacman wrapper was 19916 B against a 27510 B source, never
sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used.
It hid because check_pacman_wrapper() only tested that files EXIST. Both
installers now write a drift manifest covering source AND installed hashes;
the checker replays it with sha256sum -c and fails closed when it cannot
verify. Manifest paths are absolute - relative ones resolve to nothing under
systemd's cwd=/ and would have looped reinstalling forever.
2. The guard-lib migration (66c4698) was never finished here: guard-lib was
installed with zero registered targets while the legacy hosts-guard layer did
all the work, so every transaction ran two PreTransaction hooks and the first
one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts,
nsswitch and resolved, retires the legacy hooks and units, and has a
--rollback that was actually exercised. Plugins install to a stable path
because guardctl records absolute plugin paths. Side effect: one hook pair
instead of two, transactions ~5-6s -> ~2-3s.
3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh
makes the overlap impossible: one system-wide flock shared between root and
the user (a per-user lock would give root's pacman and the user's builds
separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only
- read-only queries stay instant - plus makepkg_capped and the coverage hook.
Always fails open.
Two bugs surfaced only by running --rollback, both of which briefly left
/etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while
legacy watchers only fire on PathModified, and `((i > 20)) && break` left the
loop body's status at 1, which under set -e aborted the rollback silently - but
only when the loop actually ran, so --dry-run looked healthy.
Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not
exist here). Not wired: Flutter/Gradle builds, which have no single entry point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
67 lines
6.3 KiB
JSON
67 lines
6.3 KiB
JSON
{
|
|
"intent": "Investigate pacman freezing at its first PreTransaction hook, and repair the deployment defects the investigation exposed: a stale pacman wrapper that no check could see, a half-finished guard-lib migration, and unserialised heavy jobs on a 7.6 GB machine.",
|
|
"scope": [
|
|
"linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh (new)",
|
|
"linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh (new)",
|
|
"linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh (new)",
|
|
"meta/scripts/run_with_heavy_lock.sh (new)",
|
|
"check_and_enable_services.sh, install_pacman_wrapper.sh, install_makepkg_wrapper.sh, pacman_wrapper.sh, makepkg_capped.sh, .pre-commit-config.yaml",
|
|
"NON-GOAL: a pacman-side fix for the stall - it could not be reproduced, so there is no verified defect to fix",
|
|
"NON-GOAL: the shutdown-schedule guard instance (/etc/shutdown-schedule.conf absent on this machine)"
|
|
],
|
|
"changes": [
|
|
"Established from /var/log/pacman.log that the stall hit 5 transactions since 2026-07-29, always at the FIRST PreTransaction hook (4/56 guard-lib runs, plus 1/142 hosts-hook runs from when THAT hook was first). What the failures share is position, not identity.",
|
|
"Proved the guard-lib hook is not the cause: /etc/guard-lib/targets is empty since install, so the loop body never executes and the script reaches exit 0 in milliseconds.",
|
|
"New diagnose_pacman_hook_stall.sh: drives real transactions (reinstalling the 0-file base-devel meta-package from cache), with a watchdog that dumps /proc/<pid>/{stack,wchan,syscall,status}, the process tree, meminfo and PSI when a live pacman sits on a hook line; --with-load applies memory pressure; --watch passively captures a real stall.",
|
|
"Redeployed the pacman wrapper: 19916 B -> 27515 B. The old copy had its own inline stale-lock handler but never sourced the shared pacman_lock_lib.sh that makepkg_wrapper already used.",
|
|
"Both installers now write a drift manifest (absolute paths) covering source AND installed hashes; check_and_enable_services.sh replays it with sha256sum -c and fails closed when the manifest is missing.",
|
|
"New migrate_hosts_guard_to_guard_lib.sh: registers hosts/nsswitch/resolved as guardctl file-guard instances, installs plugins to a stable /usr/local/share path (guardctl records absolute plugin paths), retires the legacy hooks and units, and provides --rollback/--status/--dry-run.",
|
|
"New heavy_job_lock.sh: one system-wide flock shared between root and the user, serialising pacman transactions, makepkg builds and the coverage hook; names the current holder; always fails open."
|
|
],
|
|
"verification": [
|
|
{
|
|
"command": "sudo diagnose_pacman_hook_stall.sh --runs 40 (idle)",
|
|
"result": "pass",
|
|
"evidence": "40 transactions, 0 stalls, 3/5/8 s min/median/max"
|
|
},
|
|
{
|
|
"command": "sudo diagnose_pacman_hook_stall.sh --runs 25 --with-load",
|
|
"result": "pass",
|
|
"evidence": "2578 MB hog, MemAvailable -> 1657 MB; 25 transactions, 0 stalls, 4/4/8 s. NOT a reproduction: 65 clean runs against a historical 7% rate is ~0.9% likely, so the trigger is conditional on something this synthetic load does not recreate. Memory pressure is neither confirmed nor refuted."
|
|
},
|
|
{
|
|
"command": "sudo check_and_enable_services.sh --status, across five induced states",
|
|
"result": "pass",
|
|
"evidence": "clean -> no drift; repo source edited -> detected; deployed file tampered -> detected; restored -> clean; manifest removed -> 'deployed version unverifiable' (fails closed)"
|
|
},
|
|
{
|
|
"command": "sudo migrate_hosts_guard_to_guard_lib.sh (then --rollback, then migrate again)",
|
|
"result": "pass",
|
|
"evidence": "3 instances registered; /etc/hosts byte-identical (171441 lines), chattr +i, ro bind mount intact, www.youtube.com -> 0.0.0.0, root write refused with EPERM; only ONE PreTransaction hook now and transactions dropped from ~5-6 s to ~3-4 s; rollback restores hooks, units, +i and the bind mount"
|
|
},
|
|
{
|
|
"command": "heavy_job_lock.sh contention tests (timed)",
|
|
"result": "pass",
|
|
"evidence": "--fail-fast returns 75 naming the holder; waiting mode started job B 0.01 s after job A ended; unusable lock path runs the command anyway (fail open); with the wrapper deployed, `pacman -Q` took 0 s while a real -U transaction waited 18 s behind a held lock"
|
|
},
|
|
{
|
|
"command": "pre-commit run --files <10 changed files>",
|
|
"result": "pass",
|
|
"evidence": "shellcheck, codespell, secrets, binary-block, polling-anti-pattern all Passed; only the evidence/contract hooks failed, which this artifact and its contract resolve"
|
|
}
|
|
],
|
|
"risks": [
|
|
"The stall is NOT fixed - it is only instrumented. --watch must capture a real occurrence before any pacman-side change is justified.",
|
|
"The drift manifest currently records paths inside the git worktree this work was done in. Both installers must be re-run from ~/testsAndMisc after merging, or the check will report permanent drift once the worktree is removed.",
|
|
"Retiring the legacy hosts-guard layer means /etc/hosts protection now depends entirely on guard-lib; --rollback is the escape hatch and has been exercised.",
|
|
"Two bugs were found ONLY by running --rollback: fg_uninstall clears chattr +i on its way out (legacy watchers only fire on PathModified), and a `((i > 20)) && break` left the loop body's status at 1, which under set -e silently aborted the rollback - but only when the loop actually ran, so --dry-run looked healthy. Both fixed; both had briefly left /etc/hosts unprotected.",
|
|
"A pacman transaction can now wait up to 30 minutes behind a build before failing open. Read-only queries are deliberately exempt."
|
|
],
|
|
"rollback": [
|
|
"guard-lib migration: sudo migrate_hosts_guard_to_guard_lib.sh --rollback, then confirm lsattr /etc/hosts shows 'i' and findmnt shows the ro bind mount",
|
|
"pacman wrapper: sudo ln -sfn /usr/bin/pacman.orig /usr/bin/pacman bypasses the wrapper entirely; pre-change copies are in the session scratchpad",
|
|
"heavy-job lock: rm /etc/tmpfiles.d/heavy-job-lock.conf /run/heavy-job.lock - every call site fails open when the lock is absent",
|
|
"After any rollback, validate: pacman -Q works, a real -U transaction completes, getent hosts www.youtube.com still returns 0.0.0.0, and /etc/hosts is still 171441 lines"
|
|
]
|
|
}
|