mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 20:27:02 +01:00
Krzysztof Rudnicki
79829ec8a2
pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned
transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on
retry. This investigates it and repairs the real defects found on the way.
The stall is NOT fixed, and deliberately so: no verified defect was found, so
nothing pacman-side was changed.
- Not guard-lib. /etc/guard-lib/targets was empty, so that hook script
provably exits in milliseconds. The failures share POSITION (first hook of
the transaction), not identity: one stall hit the hosts hook back when that
one ran first.
- Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB
of memory pressure) produced zero stalls. Against the historical 4-in-56
rate that is ~0.9% likely, so the trigger is conditional on something a
synthetic load does not recreate. Memory pressure is untested at the load
that actually occurred - not refuted.
So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and
its --watch mode passively captures /proc/<pid>/{stack,wchan,syscall}, the
process tree, meminfo and PSI from the next REAL stall. No pacman-side fix
should ship before that dump exists.
Three real defects were found and fixed:
1. The deployed pacman wrapper was 19916 B against a 27510 B source, never
sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used.
It hid because check_pacman_wrapper() only tested that files EXIST. Both
installers now write a drift manifest covering source AND installed hashes;
the checker replays it with sha256sum -c and fails closed when it cannot
verify. Manifest paths are absolute - relative ones resolve to nothing under
systemd's cwd=/ and would have looped reinstalling forever.
2. The guard-lib migration (66c4698) was never finished here: guard-lib was
installed with zero registered targets while the legacy hosts-guard layer did
all the work, so every transaction ran two PreTransaction hooks and the first
one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts,
nsswitch and resolved, retires the legacy hooks and units, and has a
--rollback that was actually exercised. Plugins install to a stable path
because guardctl records absolute plugin paths. Side effect: one hook pair
instead of two, transactions ~5-6s -> ~2-3s.
3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh
makes the overlap impossible: one system-wide flock shared between root and
the user (a per-user lock would give root's pacman and the user's builds
separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only
- read-only queries stay instant - plus makepkg_capped and the coverage hook.
Always fails open.
Two bugs surfaced only by running --rollback, both of which briefly left
/etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while
legacy watchers only fire on PathModified, and `((i > 20)) && break` left the
loop body's status at 1, which under set -e aborted the rollback silently - but
only when the loop actually ran, so --dry-run looked healthy.
Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not
exist here). Not wired: Flutter/Gradle builds, which have no single entry point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
32 lines
856 B
Bash
Executable File
32 lines
856 B
Bash
Executable File
#!/bin/bash
|
|
# Run a command under this machine's heavy-job lock, if it has one.
|
|
#
|
|
# Front-end for .pre-commit-config.yaml so the config stays portable: the lock
|
|
# lives at a machine-specific absolute path (installed by
|
|
# install_pacman_wrapper.sh), and hardcoding that into a tracked config would
|
|
# break the hook on any clone that has not installed it. Missing lock => run the
|
|
# command unserialised, exactly as before.
|
|
#
|
|
# Usage: run_with_heavy_lock.sh <lock-name> <command> [args...]
|
|
|
|
set -uo pipefail
|
|
|
|
readonly HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh"
|
|
|
|
if [[ $# -lt 2 ]]; then
|
|
echo "usage: $(basename "$0") <lock-name> <command> [args...]" >&2
|
|
exit 2
|
|
fi
|
|
|
|
lock_name="$1"
|
|
shift
|
|
|
|
if [[ -r $HEAVY_JOB_LOCK_LIB ]]; then
|
|
# shellcheck source=/dev/null
|
|
source "$HEAVY_JOB_LOCK_LIB"
|
|
with_heavy_lock "$lock_name" -- "$@"
|
|
exit $?
|
|
fi
|
|
|
|
exec "$@"
|