pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned
transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on
retry. This investigates it and repairs the real defects found on the way.
The stall is NOT fixed, and deliberately so: no verified defect was found, so
nothing pacman-side was changed.
- Not guard-lib. /etc/guard-lib/targets was empty, so that hook script
provably exits in milliseconds. The failures share POSITION (first hook of
the transaction), not identity: one stall hit the hosts hook back when that
one ran first.
- Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB
of memory pressure) produced zero stalls. Against the historical 4-in-56
rate that is ~0.9% likely, so the trigger is conditional on something a
synthetic load does not recreate. Memory pressure is untested at the load
that actually occurred - not refuted.
So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and
its --watch mode passively captures /proc/<pid>/{stack,wchan,syscall}, the
process tree, meminfo and PSI from the next REAL stall. No pacman-side fix
should ship before that dump exists.
Three real defects were found and fixed:
1. The deployed pacman wrapper was 19916 B against a 27510 B source, never
sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used.
It hid because check_pacman_wrapper() only tested that files EXIST. Both
installers now write a drift manifest covering source AND installed hashes;
the checker replays it with sha256sum -c and fails closed when it cannot
verify. Manifest paths are absolute - relative ones resolve to nothing under
systemd's cwd=/ and would have looped reinstalling forever.
2. The guard-lib migration (66c4698) was never finished here: guard-lib was
installed with zero registered targets while the legacy hosts-guard layer did
all the work, so every transaction ran two PreTransaction hooks and the first
one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts,
nsswitch and resolved, retires the legacy hooks and units, and has a
--rollback that was actually exercised. Plugins install to a stable path
because guardctl records absolute plugin paths. Side effect: one hook pair
instead of two, transactions ~5-6s -> ~2-3s.
3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh
makes the overlap impossible: one system-wide flock shared between root and
the user (a per-user lock would give root's pacman and the user's builds
separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only
- read-only queries stay instant - plus makepkg_capped and the coverage hook.
Always fails open.
Two bugs surfaced only by running --rollback, both of which briefly left
/etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while
legacy watchers only fire on PathModified, and `((i > 20)) && break` left the
loop body's status at 1, which under set -e aborted the rollback silently - but
only when the loop actually ran, so --dry-run looked healthy.
Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not
exist here). Not wired: Flutter/Gradle builds, which have no single entry point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Steam/browser mutual-exclusion daemon was unusable in its current state.
Removed all code, install tooling, and every reference across
install_core_system.sh, check_and_enable_services.sh, the security
hardening test suite, and CLAUDE.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three of the four remaining items were scripts hardcoding assumptions that
stopped being true — not real drift.
hosts (phantom warning): check_hosts looked for 10-unlock-etc-hosts.hook /
90-relock-etc-hosts.hook, gone since the guard migrated to guard-lib's generic
unlock-all/relock-all hooks. Match what pacman_wrapper.sh already treats as
authoritative (legacy pair still accepted).
focus_mode (never installable): install_focus_mode_daemon.sh calls
`systemctl --user`, but the checker invokes it via `sudo -u <user>` from root —
sudo changes the UID but grants no session, so XDG_RUNTIME_DIR /
DBUS_SESSION_BUS_ADDRESS were unset and it died at daemon-reload on every
automated run. Derive them from the effective UID. Now enabled+active.
compulsive_blocker (discord never limited): REAL_BINARIES hardcoded
/opt/discord/Discord — another discord package's layout. This one ships only
/usr/bin/discord, a bootstrap that downloads the app into ~/.config/discord/
and execs ~/.config/discord/Discord (a version symlink its updater maintains),
so the `[[ -x $real_binary ]]` gate failed forever. Exec the preserved LAUNCHER
(/usr/bin/discord.orig) so Discord's bootstrap/self-update still runs. Two
supporting fixes: the gate skips -x when the exec target IS the wrapper's own
.orig (the wrap creates it — chicken-and-egg), and PROCESS_MATCH is added
because the launcher exec()s away, so pgrep -f on '.orig' matches nothing and
auto-close/kill/running-state would have silently no-opped.
vbox_hosts: skipped -> n/a with a reason. VirtualBox is absent AND blocked by
pacman policy ("can bypass /etc/hosts restrictions"), so the check is not
applicable, not ducked. Installing it to satisfy a hosts check would open the
hole that check exists to close.
Also removed thesis_tracker (unused; unit not-found, script never installed).
Verified: every service ok, vbox_hosts n/a, zero errors, zero warnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JsCs1GUyiinA7bg68YvAh1
check_and_enable_services.sh was lying about two things and tracking a third
the author no longer uses.
1. workout_locker: paths still pointed at python_pkg/screen_locker/, which
stopped existing when screen-locker was extracted to its own repo
(~/screen-locker). The service was installed, enabled and active the whole
time, but the checker reported a red error and its "fix" silently did
nothing. Now resolves ~/screen-locker via SUDO_USER + getent (the script
re-execs under sudo, so $HOME is /root).
2. User-service queries could never work: `sudo -u <user> systemctl --user`
has no XDG_RUNTIME_DIR/DBUS_SESSION_BUS_ADDRESS, so it always failed with
"Failed to connect to user scope bus" and reported EVERY user service as
not-enabled. Added user_systemctl() using
`systemctl --user --machine=<user>@.host` (systemd's own suggested fix) and
rerouted all 6 call sites (focus-mode, thorium, workout-locker).
3. Removed thesis_tracker entirely — not used; unit not-found, script never
installed. Help list renumbered.
focus_mode and guest_mode_removal keep reporting error: those are GENUINE
(focus-mode.service is not-found; the guest-mode policies are absent).
Verified: ground truth (as kuhy) workout-locker.service = enabled+active.
Before: checker said "not enabled". After: workout_locker ok, all 3 checks ✓.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JsCs1GUyiinA7bg68YvAh1
Both tools swallowed failures. execute_with_log appended one line to a log
nobody reads and returned 0, so the systemd oneshot reported SUCCESS.
check_and_enable_services printed one err line among ~100 and exited 0.
That is why the 2026-05-15 reorg silently killed the pacman-wrapper and
hosts self-repair for five months with no visible symptom: check_hosts
detected the disabled hosts guards on every single run, could not find
setup_hosts_guard.sh at the stale $CONFIG_DIR/hosts path, and still exited 0.
Driver — any failure now surfaces four ways:
1. journal at ERROR priority (logger -p user.err)
2. the systemd unit FAILS (exit 1) -> systemctl --failed / red timer
3. a critical desktop notification that never auto-dismisses
4. /var/lib/periodic-system-maintenance/LAST_RUN_FAILED breadcrumb
It also collects ALL failures rather than aborting on the first.
check_and_enable_services — a missing repair script means this tool cannot
do its job, so all 12 "script not found" sites now route through
err_missing_script(): journal ERROR priority, a terminal banner, and exit 2.
Healthy runs still exit 0 and clear the breadcrumb.
Verified: driver fed nonexistent paths -> exit 1, all 3 failures on stderr,
breadcrumb written, 3 journal ERROR entries. Live regenerate + real run ->
rc=0, all steps ok, no breadcrumb. Checker --status -> rc=0, no banner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JsCs1GUyiinA7bg68YvAh1
Vendored makepkg's run_pacman() waits on /var/lib/pacman/db.lck existence
with no timeout BEFORE invoking pacman, so an orphaned lock hangs
`makepkg -i` forever ("Pacman is currently in use, please wait...") and
the pacman wrapper's stale-lock cleanup never gets a turn. Wrap makepkg
(mirroring the pacman-wrapper pattern) to clear an orphaned lock up front.
- Extract stale-lock logic into shared pacman_lock_lib.sh; pacman_wrapper
sources it after integrity verification and it is in the SHA256 manifest
- makepkg_wrapper: pre-flight clear for -i/--install; bypass on FAKEROOTKEY
and non-install; FAIL OPEN if the lib is missing (never brick makepkg)
- Cross-user pacman_process_running() pgrep guard: unprivileged makepkg can
detect a root pacman its own fuser cannot see (avoids removing a live lock)
- Upgrade survival: rewrap hook (Install/Upgrade only) + check_makepkg_wrapper
drift verifier + both wrappers wired into the hourly periodic driver
- Fix reorg path bugs: stale ~/linux-configuration periodic path,
$CONFIG_DIR/hosts -> scripts/periodic_background/hosts, test WRAPPER_DIR;
refresh .orig only when target isn't already our symlink (no exec loop)
- Tests: test_makepkg_wrapper.sh (18) incl. hook Operation validity; add
lib-integrity coverage to test_pacman_wrapper_security.sh (also path-fixed)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JsCs1GUyiinA7bg68YvAh1
- Move all linux_configuration scripts into two semantic categories:
- single_use/: scripts run once manually (fresh install, fixes, setup)
- periodic_background/: scripts run by systemd timers or daemons
- Preserve existing subdirectory structure within each category
- Fix lib/common.sh source paths for new directory depths
- Fix CONFIG_DIR depth in setup_periodic_system.sh and check_and_enable_services.sh
- Update all references in tests, fresh-install/main.sh, nix modules, and docs
- Fix check_polling_antipatterns.sh false positives (||, regex |, case patterns, jq strings)
- Fix pre-existing mypy exclusion path and type annotations for moved tools/ directory
- Rewrite check_polling_antipatterns.sh using awk (no bash regex loops); add require_serial: true
