testsAndMisc/linux_configuration/scripts/periodic_background
Krzysztof Rudnicki 79829ec8a2 Diagnose the pacman first-hook stall; fix the drift that hid a stale wrapper
pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned
transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on
retry. This investigates it and repairs the real defects found on the way.

The stall is NOT fixed, and deliberately so: no verified defect was found, so
nothing pacman-side was changed.

  - Not guard-lib. /etc/guard-lib/targets was empty, so that hook script
    provably exits in milliseconds. The failures share POSITION (first hook of
    the transaction), not identity: one stall hit the hosts hook back when that
    one ran first.
  - Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB
    of memory pressure) produced zero stalls. Against the historical 4-in-56
    rate that is ~0.9% likely, so the trigger is conditional on something a
    synthetic load does not recreate. Memory pressure is untested at the load
    that actually occurred - not refuted.

So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and
its --watch mode passively captures /proc/<pid>/{stack,wchan,syscall}, the
process tree, meminfo and PSI from the next REAL stall. No pacman-side fix
should ship before that dump exists.

Three real defects were found and fixed:

1. The deployed pacman wrapper was 19916 B against a 27510 B source, never
   sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used.
   It hid because check_pacman_wrapper() only tested that files EXIST. Both
   installers now write a drift manifest covering source AND installed hashes;
   the checker replays it with sha256sum -c and fails closed when it cannot
   verify. Manifest paths are absolute - relative ones resolve to nothing under
   systemd's cwd=/ and would have looped reinstalling forever.

2. The guard-lib migration (66c4698) was never finished here: guard-lib was
   installed with zero registered targets while the legacy hosts-guard layer did
   all the work, so every transaction ran two PreTransaction hooks and the first
   one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts,
   nsswitch and resolved, retires the legacy hooks and units, and has a
   --rollback that was actually exercised. Plugins install to a stable path
   because guardctl records absolute plugin paths. Side effect: one hook pair
   instead of two, transactions ~5-6s -> ~2-3s.

3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh
   makes the overlap impossible: one system-wide flock shared between root and
   the user (a per-user lock would give root's pacman and the user's builds
   separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only
   - read-only queries stay instant - plus makepkg_capped and the coverage hook.
   Always fails open.

Two bugs surfaced only by running --rollback, both of which briefly left
/etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while
legacy watchers only fire on PathModified, and `((i > 20)) && break` left the
loop body's status at 1, which under set -e aborted the rollback silently - but
only when the loop actually ran, so --dry-run looked healthy.

Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not
exist here). Not wired: Flutter/Gradle builds, which have no single entry point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-03 15:15:34 +02:00
..
digital_wellbeing Diagnose the pacman first-hook stall; fix the drift that hid a stale wrapper 2026-08-03 15:15:34 +02:00
hosts fix(hosts-guard): stop setup aborting silently, so the guards actually enable 2026-07-17 08:25:51 +02:00
i3-configuration fix(i3): repair the dead idle-inhibitor autostart and make it a singleton 2026-07-25 19:57:57 +02:00
system-maintenance fix(maintenance): make repair failures loud instead of silently exiting 0 2026-07-17 08:17:51 +02:00
utils Diagnose the pacman first-hook stall; fix the drift that hid a stale wrapper 2026-08-03 15:15:34 +02:00
check_and_enable_services.sh Diagnose the pacman first-hook stall; fix the drift that hid a stale wrapper 2026-08-03 15:15:34 +02:00
setup_periodic_system.sh fix(pacman): wrap makepkg so an orphaned db.lck can't hang makepkg -i 2026-07-17 07:46:31 +02:00