Diagnose the pacman first-hook stall; fix the drift that hid a stale wrapper

pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on retry. This investigates it and repairs the real defects found on the way. The stall is NOT fixed, and deliberately so: no verified defect was found, so nothing pacman-side was changed. - Not guard-lib. /etc/guard-lib/targets was empty, so that hook script provably exits in milliseconds. The failures share POSITION (first hook of the transaction), not identity: one stall hit the hosts hook back when that one ran first. - Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB of memory pressure) produced zero stalls. Against the historical 4-in-56 rate that is ~0.9% likely, so the trigger is conditional on something a synthetic load does not recreate. Memory pressure is untested at the load that actually occurred - not refuted. So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and its --watch mode passively captures /proc/<pid>/{stack,wchan,syscall}, the process tree, meminfo and PSI from the next REAL stall. No pacman-side fix should ship before that dump exists. Three real defects were found and fixed: 1. The deployed pacman wrapper was 19916 B against a 27510 B source, never sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used. It hid because check_pacman_wrapper() only tested that files EXIST. Both installers now write a drift manifest covering source AND installed hashes; the checker replays it with sha256sum -c and fails closed when it cannot verify. Manifest paths are absolute - relative ones resolve to nothing under systemd's cwd=/ and would have looped reinstalling forever. 2. The guard-lib migration (66c4698) was never finished here: guard-lib was installed with zero registered targets while the legacy hosts-guard layer did all the work, so every transaction ran two PreTransaction hooks and the first one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts, nsswitch and resolved, retires the legacy hooks and units, and has a --rollback that was actually exercised. Plugins install to a stable path because guardctl records absolute plugin paths. Side effect: one hook pair instead of two, transactions ~5-6s -> ~2-3s. 3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh makes the overlap impossible: one system-wide flock shared between root and the user (a per-user lock would give root's pacman and the user's builds separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only - read-only queries stay instant - plus makepkg_capped and the coverage hook. Always fails open. Two bugs surfaced only by running --rollback, both of which briefly left /etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while legacy watchers only fire on PathModified, and `((i > 20)) && break` left the loop body's status at 1, which under set -e aborted the rollback silently - but only when the loop actually ran, so --dry-run looked healthy. Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not exist here). Not wired: Flutter/Gradle builds, which have no single entry point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-11-10 18:07:06 +01:00 · 2026-08-03 15:14:16 +02:00 · 2026-08-03 15:14:16 +02:00 · 79829ec8a2
commit 79829ec8a2
parent 4d9f486176
12 changed files with 1571 additions and 109 deletions
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@ -240,7 +240,10 @@ repos:
    hooks:
      - id: pytest-coverage
        name: pytest with coverage enforcement
-        entry: python meta/scripts/pytest_changed_packages.py
+        # Serialised against builds and pacman transactions on this 7.6 GB
+        # machine via the heavy-job lock. run_with_heavy_lock.sh is a plain
+        # passthrough on any clone that has not installed one.
+        entry: bash meta/scripts/run_with_heavy_lock.sh pytest python meta/scripts/pytest_changed_packages.py
        language: system
        types: [python]
        pass_filenames: true
--- a/docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json
+++ b/docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json
@ -0,0 +1,20 @@
+{
+  "title": "Diagnose the pacman first-hook stall; fix wrapper drift, finish the guard-lib migration, serialise heavy jobs",
+  "objective": "pacman intermittently freezes at its first PreTransaction hook (5 abandoned transactions since 2026-07-29, stalls of 4 minutes to overnight). Establish what is actually failing rather than assuming, then repair the real deployment defects found along the way. Success means: the stall is either root-caused or instrumented so the next real occurrence is captured; the deployed pacman wrapper matches its source and drift can no longer hide; there is one guard system for /etc/hosts instead of two half-installed ones; and concurrent heavy jobs on this 7.6 GB machine are prevented by a mechanism rather than by a human remembering a rule.",
+  "acceptance_criteria": [
+    "The guard-lib hook is proven not to be the cause (its target list is empty, so the script body cannot execute)",
+    "A repro harness drives real pacman transactions and captures /proc/<pid>/stack, wchan, syscall, process tree, meminfo and PSI on a stall",
+    "A passive --watch mode captures the same evidence from a real (not synthetic) stall",
+    "Deployed pacman_wrapper matches the repo source, and check_and_enable_services.sh detects content drift in both directions (source moved on / deployed tampered) and fails closed when unverifiable",
+    "hosts, nsswitch and resolved are guardctl file-guard instances; the legacy pacman hooks and units are retired; /etc/hosts keeps chattr +i, its read-only bind mount, its exact contents and its blocking behaviour",
+    "The migration has a --rollback that restores the legacy layer INCLUDING the immutable flag, verified by running it",
+    "A heavy-job lock serialises pacman transactions, makepkg builds and the coverage hook, names the current holder when it makes something wait, and always fails open"
+  ],
+  "out_of_scope": [
+    "Shipping a pacman-side fix for the stall itself: it could not be reproduced, so there is no verified defect to fix yet",
+    "The shutdown-schedule guard instance: /etc/shutdown-schedule.conf does not exist on this machine, so registering it would mean installing the midnight-shutdown feature rather than finishing this migration",
+    "Wiring the lock into Flutter/Gradle builds: the repo has no single build entry point to wrap; the lock is available as a CLI for those",
+    "Anything on the desktop machine, which is not reachable from this session"
+  ],
+  "verifier": "sudo diagnose_pacman_hook_stall.sh --runs N [--with-load]; sudo migrate_hosts_guard_to_guard_lib.sh --status/--rollback; sudo check_and_enable_services.sh --status with deliberately induced drift; timed heavy_job_lock.sh contention tests; pre-commit run --files <changed>"
+}
--- a/docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json
+++ b/docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json
@ -0,0 +1,66 @@
+{
+  "intent": "Investigate pacman freezing at its first PreTransaction hook, and repair the deployment defects the investigation exposed: a stale pacman wrapper that no check could see, a half-finished guard-lib migration, and unserialised heavy jobs on a 7.6 GB machine.",
+  "scope": [
+    "linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh (new)",
+    "linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh (new)",
+    "linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh (new)",
+    "meta/scripts/run_with_heavy_lock.sh (new)",
+    "check_and_enable_services.sh, install_pacman_wrapper.sh, install_makepkg_wrapper.sh, pacman_wrapper.sh, makepkg_capped.sh, .pre-commit-config.yaml",
+    "NON-GOAL: a pacman-side fix for the stall - it could not be reproduced, so there is no verified defect to fix",
+    "NON-GOAL: the shutdown-schedule guard instance (/etc/shutdown-schedule.conf absent on this machine)"
+  ],
+  "changes": [
+    "Established from /var/log/pacman.log that the stall hit 5 transactions since 2026-07-29, always at the FIRST PreTransaction hook (4/56 guard-lib runs, plus 1/142 hosts-hook runs from when THAT hook was first). What the failures share is position, not identity.",
+    "Proved the guard-lib hook is not the cause: /etc/guard-lib/targets is empty since install, so the loop body never executes and the script reaches exit 0 in milliseconds.",
+    "New diagnose_pacman_hook_stall.sh: drives real transactions (reinstalling the 0-file base-devel meta-package from cache), with a watchdog that dumps /proc/<pid>/{stack,wchan,syscall,status}, the process tree, meminfo and PSI when a live pacman sits on a hook line; --with-load applies memory pressure; --watch passively captures a real stall.",
+    "Redeployed the pacman wrapper: 19916 B -> 27515 B. The old copy had its own inline stale-lock handler but never sourced the shared pacman_lock_lib.sh that makepkg_wrapper already used.",
+    "Both installers now write a drift manifest (absolute paths) covering source AND installed hashes; check_and_enable_services.sh replays it with sha256sum -c and fails closed when the manifest is missing.",
+    "New migrate_hosts_guard_to_guard_lib.sh: registers hosts/nsswitch/resolved as guardctl file-guard instances, installs plugins to a stable /usr/local/share path (guardctl records absolute plugin paths), retires the legacy hooks and units, and provides --rollback/--status/--dry-run.",
+    "New heavy_job_lock.sh: one system-wide flock shared between root and the user, serialising pacman transactions, makepkg builds and the coverage hook; names the current holder; always fails open."
+  ],
+  "verification": [
+    {
+      "command": "sudo diagnose_pacman_hook_stall.sh --runs 40  (idle)",
+      "result": "pass",
+      "evidence": "40 transactions, 0 stalls, 3/5/8 s min/median/max"
+    },
+    {
+      "command": "sudo diagnose_pacman_hook_stall.sh --runs 25 --with-load",
+      "result": "pass",
+      "evidence": "2578 MB hog, MemAvailable -> 1657 MB; 25 transactions, 0 stalls, 4/4/8 s. NOT a reproduction: 65 clean runs against a historical 7% rate is ~0.9% likely, so the trigger is conditional on something this synthetic load does not recreate. Memory pressure is neither confirmed nor refuted."
+    },
+    {
+      "command": "sudo check_and_enable_services.sh --status, across five induced states",
+      "result": "pass",
+      "evidence": "clean -> no drift; repo source edited -> detected; deployed file tampered -> detected; restored -> clean; manifest removed -> 'deployed version unverifiable' (fails closed)"
+    },
+    {
+      "command": "sudo migrate_hosts_guard_to_guard_lib.sh  (then --rollback, then migrate again)",
+      "result": "pass",
+      "evidence": "3 instances registered; /etc/hosts byte-identical (171441 lines), chattr +i, ro bind mount intact, www.youtube.com -> 0.0.0.0, root write refused with EPERM; only ONE PreTransaction hook now and transactions dropped from ~5-6 s to ~3-4 s; rollback restores hooks, units, +i and the bind mount"
+    },
+    {
+      "command": "heavy_job_lock.sh contention tests (timed)",
+      "result": "pass",
+      "evidence": "--fail-fast returns 75 naming the holder; waiting mode started job B 0.01 s after job A ended; unusable lock path runs the command anyway (fail open); with the wrapper deployed, `pacman -Q` took 0 s while a real -U transaction waited 18 s behind a held lock"
+    },
+    {
+      "command": "pre-commit run --files <10 changed files>",
+      "result": "pass",
+      "evidence": "shellcheck, codespell, secrets, binary-block, polling-anti-pattern all Passed; only the evidence/contract hooks failed, which this artifact and its contract resolve"
+    }
+  ],
+  "risks": [
+    "The stall is NOT fixed - it is only instrumented. --watch must capture a real occurrence before any pacman-side change is justified.",
+    "The drift manifest currently records paths inside the git worktree this work was done in. Both installers must be re-run from ~/testsAndMisc after merging, or the check will report permanent drift once the worktree is removed.",
+    "Retiring the legacy hosts-guard layer means /etc/hosts protection now depends entirely on guard-lib; --rollback is the escape hatch and has been exercised.",
+    "Two bugs were found ONLY by running --rollback: fg_uninstall clears chattr +i on its way out (legacy watchers only fire on PathModified), and a `((i > 20)) && break` left the loop body's status at 1, which under set -e silently aborted the rollback - but only when the loop actually ran, so --dry-run looked healthy. Both fixed; both had briefly left /etc/hosts unprotected.",
+    "A pacman transaction can now wait up to 30 minutes behind a build before failing open. Read-only queries are deliberately exempt."
+  ],
+  "rollback": [
+    "guard-lib migration: sudo migrate_hosts_guard_to_guard_lib.sh --rollback, then confirm lsattr /etc/hosts shows 'i' and findmnt shows the ro bind mount",
+    "pacman wrapper: sudo ln -sfn /usr/bin/pacman.orig /usr/bin/pacman bypasses the wrapper entirely; pre-change copies are in the session scratchpad",
+    "heavy-job lock: rm /etc/tmpfiles.d/heavy-job-lock.conf /run/heavy-job.lock - every call site fails open when the lock is absent",
+    "After any rollback, validate: pacman -Q works, a real -U transaction completes, getent hosts www.youtube.com still returns 0.0.0.0, and /etc/hosts is still 171441 lines"
+  ]
+}
--- a/linux_configuration/scripts/periodic_background/check_and_enable_services.sh
+++ b/linux_configuration/scripts/periodic_background/check_and_enable_services.sh
@ -32,6 +32,9 @@ CONFIG_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")"
 # Script paths
 PACMAN_WRAPPER_INSTALL="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh"
 MAKEPKG_WRAPPER_INSTALL="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh"
+# Drift manifests written by the two installers above (see deployment_drift).
+PACMAN_WRAPPER_MANIFEST="/var/lib/pacman-wrapper/source.sha256"
+MAKEPKG_WRAPPER_MANIFEST="/var/lib/pacman-wrapper/makepkg-source.sha256"
 MIDNIGHT_SHUTDOWN_SCRIPT="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/setup_midnight_shutdown.sh"
 STARTUP_MONITOR_SCRIPT="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/setup_pc_startup_monitor.sh"
 PERIODIC_SYSTEM_SCRIPT="$CONFIG_DIR/scripts/periodic_background/setup_periodic_system.sh"
@ -109,6 +112,28 @@ hosts_pacman_hooks_installed() {
 			[[ -f /etc/pacman.d/hooks/90-relock-etc-hosts.hook ]]; }
 }

+# Replay a drift manifest written at install time. The manifest holds plain
+# sha256sum lines covering BOTH the repo sources an installer copied from and
+# the installed copies it produced, so one `sha256sum -c` answers both "has the
+# repo moved on?" and "did someone edit the deployed file?".
+#
+# This exists because an existence check is not a deployment check. On
+# 2026-08-03 /usr/local/bin/pacman_wrapper was 19916 B against a 27510 B source
+# — missing the policy integrity manifest, the pacman_lock_lib stale-lock fix
+# and the guard-lib fallbacks — while every check here reported "ok" hourly,
+# because the file was present and /usr/bin/pacman was a symlink. Both of those
+# facts stayed true the entire time the wrapper was a week out of date.
+#
+# Return: 0 verified, 1 drift detected, 2 no manifest (install predates this).
+# Callers must treat 2 as actionable rather than a pass: "cannot verify" is
+# precisely the state that hid the stale wrapper.
+deployment_drift() { # <manifest-path>
+	local manifest="$1"
+	[[ -f $manifest ]] || return 2
+	sha256sum -c --status "$manifest" 2>/dev/null || return 1
+	return 0
+}
+
 run() {
 	if [[ $DRY_RUN -eq 1 ]]; then
 		echo -e "${YELLOW}DRY-RUN:${NC} $*"
@ -280,6 +305,22 @@ check_pacman_wrapper() {
 		fi
 	done

+	# Content drift — the check everything above misses. A correct symlink and a
+	# present file say nothing about WHICH version is deployed.
+	local drift_rc=0
+	deployment_drift "$PACMAN_WRAPPER_MANIFEST" || drift_rc=$?
+	case $drift_rc in
+	0) msg "Wrapper matches the source it was installed from (no drift)" ;;
+	1)
+		issues+=("Deployed pacman wrapper differs from its install manifest (stale or tampered)")
+		status="error"
+		;;
+	2)
+		issues+=("No drift manifest at $PACMAN_WRAPPER_MANIFEST — deployed version unverifiable")
+		status="error"
+		;;
+	esac
+
 	# Report and fix
 	if [[ $status == "error" ]]; then
 		for issue in "${issues[@]}"; do
@ -292,8 +333,10 @@ check_pacman_wrapper() {
 			if [[ -f $PACMAN_WRAPPER_INSTALL ]]; then
 				run bash "$PACMAN_WRAPPER_INSTALL"
 				((FIXES_APPLIED++)) || true
-				# Re-verify after fix
-				if [[ $DRY_RUN -eq 0 ]] && [[ -L /usr/bin/pacman ]] && [[ -f /usr/bin/pacman.orig ]] && [[ -f /usr/local/bin/pacman_wrapper ]]; then
+				# Re-verify after fix. Content is part of "fixed": a reinstall
+				# that leaves the manifest failing has repaired nothing.
+				if [[ $DRY_RUN -eq 0 ]] && [[ -L /usr/bin/pacman ]] && [[ -f /usr/bin/pacman.orig ]] &&
+					[[ -f /usr/local/bin/pacman_wrapper ]] && deployment_drift "$PACMAN_WRAPPER_MANIFEST"; then
 					status="ok"
 				fi
 			else
@ -360,6 +403,21 @@ check_makepkg_wrapper() {
 		status="error"
 	fi

+	# Content drift (see check_pacman_wrapper for why existence is not enough)
+	local drift_rc=0
+	deployment_drift "$MAKEPKG_WRAPPER_MANIFEST" || drift_rc=$?
+	case $drift_rc in
+	0) msg "Makepkg wrapper matches the source it was installed from (no drift)" ;;
+	1)
+		issues+=("Deployed makepkg wrapper differs from its install manifest (stale or tampered)")
+		status="error"
+		;;
+	2)
+		issues+=("No drift manifest at $MAKEPKG_WRAPPER_MANIFEST — deployed version unverifiable")
+		status="error"
+		;;
+	esac
+
 	# Report and fix
 	if [[ $status == "error" ]]; then
 		for issue in "${issues[@]}"; do
--- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh
+++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh
@ -33,6 +33,7 @@ REWRAP_SOURCE="$SRC_DIR/rewrap_pkg_managers.sh"
 REWRAP_DEST="${INSTALL_DIR}/rewrap_pkg_managers.sh"
 HOOK_SOURCE="$SRC_DIR/96-restore-pkg-wrappers.hook"
 HOOK_DEST="/etc/pacman.d/hooks/96-restore-pkg-wrappers.hook"
+SOURCE_MANIFEST="/var/lib/pacman-wrapper/makepkg-source.sha256"

 # Copy a file, unlocking an immutable destination first (the shared lib may be
 # chattr +i from install_pacman_wrapper.sh's integrity protection).
@ -83,5 +84,26 @@ deploy_file "$HOOK_SOURCE" "$HOOK_DEST" 644
 echo -e "${BLUE}Creating symbolic link /usr/bin/makepkg -> ${WRAPPER_DEST}...${NC}"
 ln -sf "$WRAPPER_DEST" /usr/bin/makepkg

+# Drift manifest — see deployment_drift() in check_and_enable_services.sh. Same
+# rationale as the pacman wrapper's: "the file exists" is not "the right version
+# is deployed", and only the second one is worth re-checking every hour.
+mkdir -p "$(dirname "$SOURCE_MANIFEST")"
+{
+	sha256sum "$WRAPPER_SOURCE" "$LOCK_LIB_SOURCE" "$REWRAP_SOURCE" "$HOOK_SOURCE" || {
+		echo -e "${RED}Failed to checksum makepkg wrapper sources${NC}" >&2
+		exit 1
+	}
+	sha256sum "$WRAPPER_DEST" "$REWRAP_DEST" "$HOOK_DEST" || {
+		echo -e "${RED}Failed to checksum installed makepkg wrapper${NC}" >&2
+		exit 1
+	}
+} >"$SOURCE_MANIFEST"
+
+if [[ ! -s $SOURCE_MANIFEST ]]; then
+	echo -e "${RED}Error: drift manifest was not created or is empty${NC}" >&2
+	exit 1
+fi
+chmod 644 "$SOURCE_MANIFEST"
+
 echo -e "${GREEN}makepkg wrapper installed.${NC} Original preserved at ${CYAN}/usr/bin/makepkg.orig${NC}"
 echo -e "Upgrade-survival hook installed at ${CYAN}${HOOK_DEST}${NC}"
--- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh
+++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh
@ -5,9 +5,9 @@ set -euo pipefail

 # Auto-sudo functionality
 if [ "$EUID" -ne 0 ]; then
-  echo "Executing with sudo..."
-  sudo "$0" "$@"
-  exit $?
+	echo "Executing with sudo..."
+	sudo "$0" "$@"
+	exit $?
 fi

 # Colors
@ -27,6 +27,7 @@ WHITELIST_SOURCE="$(dirname "$0")/pacman_whitelist.txt"
 GREYLIST_SOURCE="$(dirname "$0")/pacman_greylist.txt"
 MAKEPKG_CAPPED_SOURCE="$(dirname "$0")/makepkg_capped.sh"
 MKPKG_SOURCE="$(dirname "$0")/mkpkg.sh"
+HEAVY_LOCK_SOURCE="$(dirname "$0")/../../utils/heavy_job_lock.sh"
 INSTALL_DIR="/usr/local/bin"
 WRAPPER_DEST="${INSTALL_DIR}/pacman_wrapper"
 LOCK_LIB_DEST="${INSTALL_DIR}/pacman_lock_lib.sh"
@ -36,8 +37,10 @@ WHITELIST_DEST="${INSTALL_DIR}/pacman_whitelist.txt"
 GREYLIST_DEST="${INSTALL_DIR}/pacman_greylist.txt"
 MAKEPKG_CAPPED_DEST="${INSTALL_DIR}/makepkg_capped"
 MKPKG_DEST="${INSTALL_DIR}/mkpkg"
+HEAVY_LOCK_DEST="${INSTALL_DIR}/heavy_job_lock.sh"
 INTEGRITY_DIR="/var/lib/pacman-wrapper"
 INTEGRITY_FILE="${INTEGRITY_DIR}/policy.sha256"
+SOURCE_MANIFEST="${INTEGRITY_DIR}/source.sha256"
 LEECHBLOCK_INSTALLER_SOURCE="$(dirname "$0")/../install_leechblock.sh"
 LEECHBLOCK_DEFAULTS_SOURCE="$(dirname "$0")/../leechblock_defaults.json"
 LEECHBLOCK_SEEDER_SOURCE="$(dirname "$0")/../seed_leechblock_storage.js"
@ -53,63 +56,63 @@ VBOX_ENFORCE_DEST="${VBOX_INSTALL_DIR}/enforce_vbox_hosts.sh"
 declare -a RELock_FILES=()

 is_immutable_file() {
-  local file_path="$1"
-  [[ -e "$file_path" ]] || return 1
-  [[ $(lsattr -d "$file_path" 2>/dev/null | awk '{print $1}') == *i* ]]
+	local file_path="$1"
+	[[ -e "$file_path" ]] || return 1
+	[[ $(lsattr -d "$file_path" 2>/dev/null | awk '{print $1}') == *i* ]]
 }

 unlock_immutable_file_if_needed() {
-  local file_path="$1"
-  if ! command -v chattr >/dev/null 2>&1; then
-    return 0
-  fi
-  if is_immutable_file "$file_path"; then
-    chattr -i "$file_path"
-    RELock_FILES+=("$file_path")
-  fi
+	local file_path="$1"
+	if ! command -v chattr >/dev/null 2>&1; then
+		return 0
+	fi
+	if is_immutable_file "$file_path"; then
+		chattr -i "$file_path"
+		RELock_FILES+=("$file_path")
+	fi
 }

 relock_files_on_exit() {
-  if ! command -v chattr >/dev/null 2>&1; then
-    return
-  fi
-  for file_path in "${RELock_FILES[@]}"; do
-    [[ -e "$file_path" ]] || continue
-    chattr +i "$file_path" 2>/dev/null || true
-  done
+	if ! command -v chattr >/dev/null 2>&1; then
+		return
+	fi
+	for file_path in "${RELock_FILES[@]}"; do
+		[[ -e "$file_path" ]] || continue
+		chattr +i "$file_path" 2>/dev/null || true
+	done
 }

 copy_managed_file() {
-  local source_file="$1"
-  local dest_file="$2"
-  local required="$3"
-  local label="$4"
+	local source_file="$1"
+	local dest_file="$2"
+	local required="$3"
+	local label="$4"

-  if [[ ! -f "$source_file" ]]; then
-    if [[ "$required" == "required" ]]; then
-      echo -e "${RED}Error:${NC} Missing required ${label} at ${source_file}" >&2
-      exit 1
-    fi
-    echo -e "${YELLOW}Warning:${NC} Missing ${label} at ${source_file}" >&2
-    return
-  fi
+	if [[ ! -f "$source_file" ]]; then
+		if [[ "$required" == "required" ]]; then
+			echo -e "${RED}Error:${NC} Missing required ${label} at ${source_file}" >&2
+			exit 1
+		fi
+		echo -e "${YELLOW}Warning:${NC} Missing ${label} at ${source_file}" >&2
+		return
+	fi

-  unlock_immutable_file_if_needed "$dest_file"
-  cp "$source_file" "$dest_file"
+	unlock_immutable_file_if_needed "$dest_file"
+	cp "$source_file" "$dest_file"
 }

 trap relock_files_on_exit EXIT

 # Check if script is run as root
 if [ "$EUID" -ne 0 ]; then
-  echo -e "${RED}Please run as root${NC}"
-  exit 1
+	echo -e "${RED}Please run as root${NC}"
+	exit 1
 fi

 # Check if the wrapper script exists
 if [ ! -f "$WRAPPER_SOURCE" ]; then
-  echo -e "${RED}Error: Wrapper script not found at ${WRAPPER_SOURCE}${NC}"
-  exit 1
+	echo -e "${RED}Error: Wrapper script not found at ${WRAPPER_SOURCE}${NC}"
+	exit 1
 fi

 echo -e "${CYAN}Installing pacman wrapper...${NC}"
@ -128,7 +131,12 @@ copy_managed_file "$MAKEPKG_CAPPED_SOURCE" "$MAKEPKG_CAPPED_DEST" required "make
 chmod +x "$MAKEPKG_CAPPED_DEST"
 copy_managed_file "$MKPKG_SOURCE" "$MKPKG_DEST" required "mkpkg helper"
 chmod +x "$MKPKG_DEST"
-chmod 644 "$WORDS_DEST" "$BLOCKED_DEST" "$WHITELIST_DEST" "$GREYLIST_DEST" 2> /dev/null || true
+# The heavy-job lock is sourced by pacman_wrapper.sh and makepkg_capped.sh from
+# a fixed absolute path, so it has to land next to them rather than be read out
+# of a repo checkout that may move.
+copy_managed_file "$HEAVY_LOCK_SOURCE" "$HEAVY_LOCK_DEST" required "heavy-job lock library"
+chmod 755 "$HEAVY_LOCK_DEST"
+chmod 644 "$WORDS_DEST" "$BLOCKED_DEST" "$WHITELIST_DEST" "$GREYLIST_DEST" 2>/dev/null || true

 # Automatically use symbolic link installation method
 echo -e "${YELLOW}Installing using symbolic link method...${NC}"
@ -138,8 +146,8 @@ echo -e "${YELLOW}Installing using symbolic link method...${NC}"
 # NEVER when it is already our symlink — copying the symlink's target would put
 # the wrapper into pacman.orig and cause an exec loop.
 if [ ! -L /usr/bin/pacman ]; then
-  echo -e "${BLUE}Backing up original pacman to /usr/bin/pacman.orig...${NC}"
-  cp -f /usr/bin/pacman /usr/bin/pacman.orig
+	echo -e "${BLUE}Backing up original pacman to /usr/bin/pacman.orig...${NC}"
+	cp -f /usr/bin/pacman /usr/bin/pacman.orig
 fi

 # Update the PACMAN_BIN variable in the wrapper to point to the original
@ -160,81 +168,135 @@ missing_files=()
 [[ ! -f "$LOCK_LIB_DEST" ]] && missing_files+=("$LOCK_LIB_DEST")

 if [[ ${#missing_files[@]} -gt 0 ]]; then
-  echo -e "${RED}Error: Critical policy files are missing:${NC}"
-  printf '%s\n' "${missing_files[@]}" >&2
-  echo -e "${RED}Installation incomplete. Cannot create integrity file.${NC}"
-  exit 1
+	echo -e "${RED}Error: Critical policy files are missing:${NC}"
+	printf '%s\n' "${missing_files[@]}" >&2
+	echo -e "${RED}Installation incomplete. Cannot create integrity file.${NC}"
+	exit 1
 fi

 {
-  sha256sum "$BLOCKED_DEST" || { echo -e "${RED}Failed to checksum blocked list${NC}" >&2; exit 1; }
-  sha256sum "$GREYLIST_DEST" || { echo -e "${RED}Failed to checksum greylist${NC}" >&2; exit 1; }
-  # The shared stale-lock library is executed (sourced) by the wrapper, so it is
-  # integrity-checked too: pacman_wrapper.sh sources it only AFTER
-  # verify_policy_integrity passes, so a tampered lib is rejected before it runs.
-  sha256sum "$LOCK_LIB_DEST" || { echo -e "${RED}Failed to checksum lock library${NC}" >&2; exit 1; }
-  # Whitelist is optional
-  if [[ -f "$WHITELIST_DEST" ]]; then
-    sha256sum "$WHITELIST_DEST" || { echo -e "${RED}Failed to checksum whitelist${NC}" >&2; exit 1; }
-  fi
-} > "$INTEGRITY_FILE"
+	sha256sum "$BLOCKED_DEST" || {
+		echo -e "${RED}Failed to checksum blocked list${NC}" >&2
+		exit 1
+	}
+	sha256sum "$GREYLIST_DEST" || {
+		echo -e "${RED}Failed to checksum greylist${NC}" >&2
+		exit 1
+	}
+	# The shared stale-lock library is executed (sourced) by the wrapper, so it is
+	# integrity-checked too: pacman_wrapper.sh sources it only AFTER
+	# verify_policy_integrity passes, so a tampered lib is rejected before it runs.
+	sha256sum "$LOCK_LIB_DEST" || {
+		echo -e "${RED}Failed to checksum lock library${NC}" >&2
+		exit 1
+	}
+	# Whitelist is optional
+	if [[ -f "$WHITELIST_DEST" ]]; then
+		sha256sum "$WHITELIST_DEST" || {
+			echo -e "${RED}Failed to checksum whitelist${NC}" >&2
+			exit 1
+		}
+	fi
+} >"$INTEGRITY_FILE"

 # Verify integrity file was created and has content
 if [[ ! -s "$INTEGRITY_FILE" ]]; then
-  echo -e "${RED}Error: Integrity file was not created or is empty${NC}"
-  exit 1
+	echo -e "${RED}Error: Integrity file was not created or is empty${NC}"
+	exit 1
 fi

 # Make integrity file immutable
 chmod 400 "$INTEGRITY_FILE"
-if command -v chattr > /dev/null 2>&1; then
-  chattr +i "$INTEGRITY_FILE" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make integrity file immutable${NC}"
+if command -v chattr >/dev/null 2>&1; then
+	chattr +i "$INTEGRITY_FILE" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make integrity file immutable${NC}"
 fi

+# Record a drift manifest: the hash of every SOURCE file we installed from, and
+# of the installed copies we produced. check_and_enable_services.sh replays it
+# with `sha256sum -c`, which answers both questions in one command:
+#   - a source line fails => the repo moved on, this deployment is stale
+#   - a dest line fails   => someone edited the installed copy directly
+# Neither was detectable before: the checker only tested that files EXIST, which
+# is why a wrapper 7.6 KB behind the repo (no integrity manifest, no
+# pacman_lock_lib, no guard-lib fallbacks) ran unnoticed for a week while the
+# hourly maintenance timer kept reporting "ok".
+# Deliberately NOT chattr +i (unlike the policy integrity file above): this is a
+# drift record, not a security boundary, and it must be rewritable each install.
+echo -e "${BLUE}Recording deployment drift manifest...${NC}"
+# ABSOLUTE paths only. The SOURCE_* vars above are built from `dirname "$0"`, so
+# they are relative whenever the installer is invoked by a relative path — and
+# check_and_enable_services.sh replays this manifest from systemd with cwd=/,
+# where relative entries resolve to nothing. sha256sum -c would report those as
+# failures, the checker would read that as drift, and it would reinstall on
+# every single hourly run forever.
+manifest_sources=()
+for src in "$WRAPPER_SOURCE" "$LOCK_LIB_SOURCE" "$BLOCKED_SOURCE" "$GREYLIST_SOURCE" \
+	"$MAKEPKG_CAPPED_SOURCE" "$MKPKG_SOURCE" "$WHITELIST_SOURCE"; do
+	[[ -f "$src" ]] || continue # whitelist is optional
+	manifest_sources+=("$(readlink -f "$src")")
+done
+
+{
+	sha256sum "${manifest_sources[@]}" || {
+		echo -e "${RED}Failed to checksum wrapper sources${NC}" >&2
+		exit 1
+	}
+	sha256sum "$LOCK_LIB_DEST" || {
+		echo -e "${RED}Failed to checksum installed lock lib${NC}" >&2
+		exit 1
+	}
+} >"$SOURCE_MANIFEST"
+
+if [[ ! -s "$SOURCE_MANIFEST" ]]; then
+	echo -e "${RED}Error: drift manifest was not created or is empty${NC}" >&2
+	exit 1
+fi
+chmod 644 "$SOURCE_MANIFEST"
+
 # Make policy files immutable to prevent easy tampering
 echo -e "${BLUE}Protecting policy files from modification...${NC}"
-if command -v chattr > /dev/null 2>&1; then
-  chattr +i "$BLOCKED_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make blocked list immutable${NC}"
-  chattr +i "$GREYLIST_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make greylist immutable${NC}"
-  chattr +i "$LOCK_LIB_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make lock library immutable${NC}"
-  # Note: whitelist is intentionally left modifiable for user convenience
+if command -v chattr >/dev/null 2>&1; then
+	chattr +i "$BLOCKED_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make blocked list immutable${NC}"
+	chattr +i "$GREYLIST_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make greylist immutable${NC}"
+	chattr +i "$LOCK_LIB_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make lock library immutable${NC}"
+	# Note: whitelist is intentionally left modifiable for user convenience
 else
-  echo -e "${YELLOW}Warning: chattr not available, policy files will not be immutable${NC}"
+	echo -e "${YELLOW}Warning: chattr not available, policy files will not be immutable${NC}"
 fi

 # Install LeechBlock installer and defaults if available
 mkdir -p "$LEECHBLOCK_INSTALL_DIR"
 if [ -f "$LEECHBLOCK_INSTALLER_SOURCE" ]; then
-  echo -e "${BLUE}Installing LeechBlock installer to ${LEECHBLOCK_INSTALLER_DEST}...${NC}"
-  cp "$LEECHBLOCK_INSTALLER_SOURCE" "$LEECHBLOCK_INSTALLER_DEST"
-  chmod +x "$LEECHBLOCK_INSTALLER_DEST"
-  echo -e "${GREEN}LeechBlock installer deployed to ${LEECHBLOCK_INSTALLER_DEST}${NC}"
+	echo -e "${BLUE}Installing LeechBlock installer to ${LEECHBLOCK_INSTALLER_DEST}...${NC}"
+	cp "$LEECHBLOCK_INSTALLER_SOURCE" "$LEECHBLOCK_INSTALLER_DEST"
+	chmod +x "$LEECHBLOCK_INSTALLER_DEST"
+	echo -e "${GREEN}LeechBlock installer deployed to ${LEECHBLOCK_INSTALLER_DEST}${NC}"
 else
-  echo -e "${YELLOW}LeechBlock installer not found at ${LEECHBLOCK_INSTALLER_SOURCE}, skipping...${NC}"
+	echo -e "${YELLOW}LeechBlock installer not found at ${LEECHBLOCK_INSTALLER_SOURCE}, skipping...${NC}"
 fi
 if [ -f "$LEECHBLOCK_DEFAULTS_SOURCE" ]; then
-  cp "$LEECHBLOCK_DEFAULTS_SOURCE" "$LEECHBLOCK_DEFAULTS_DEST"
-  echo -e "${GREEN}LeechBlock defaults deployed to ${LEECHBLOCK_DEFAULTS_DEST}${NC}"
+	cp "$LEECHBLOCK_DEFAULTS_SOURCE" "$LEECHBLOCK_DEFAULTS_DEST"
+	echo -e "${GREEN}LeechBlock defaults deployed to ${LEECHBLOCK_DEFAULTS_DEST}${NC}"
 fi
 if [ -f "$LEECHBLOCK_SEEDER_SOURCE" ]; then
-  cp "$LEECHBLOCK_SEEDER_SOURCE" "$LEECHBLOCK_SEEDER_DEST"
-  echo -e "${GREEN}LeechBlock seeder deployed to ${LEECHBLOCK_SEEDER_DEST}${NC}"
+	cp "$LEECHBLOCK_SEEDER_SOURCE" "$LEECHBLOCK_SEEDER_DEST"
+	echo -e "${GREEN}LeechBlock seeder deployed to ${LEECHBLOCK_SEEDER_DEST}${NC}"
 fi
 if [ -f "$LEECHBLOCK_PKG_SOURCE" ]; then
-  cp "$LEECHBLOCK_PKG_SOURCE" "${LEECHBLOCK_INSTALL_DIR}/package.json"
-  echo -e "${BLUE}Installing Node.js deps in ${LEECHBLOCK_INSTALL_DIR}...${NC}"
-  npm install --prefix "$LEECHBLOCK_INSTALL_DIR" 2>&1 | grep -v '^npm warn' || true
+	cp "$LEECHBLOCK_PKG_SOURCE" "${LEECHBLOCK_INSTALL_DIR}/package.json"
+	echo -e "${BLUE}Installing Node.js deps in ${LEECHBLOCK_INSTALL_DIR}...${NC}"
+	npm install --prefix "$LEECHBLOCK_INSTALL_DIR" 2>&1 | grep -v '^npm warn' || true
 fi

 # Install VirtualBox enforcement script if available
 if [ -f "$VBOX_ENFORCE_SOURCE" ]; then
-  echo -e "${BLUE}Installing VirtualBox hosts enforcement script...${NC}"
-  mkdir -p "$VBOX_INSTALL_DIR"
-  cp "$VBOX_ENFORCE_SOURCE" "$VBOX_ENFORCE_DEST"
-  chmod +x "$VBOX_ENFORCE_DEST"
-  echo -e "${GREEN}VirtualBox enforcement script installed to ${VBOX_ENFORCE_DEST}${NC}"
+	echo -e "${BLUE}Installing VirtualBox hosts enforcement script...${NC}"
+	mkdir -p "$VBOX_INSTALL_DIR"
+	cp "$VBOX_ENFORCE_SOURCE" "$VBOX_ENFORCE_DEST"
+	chmod +x "$VBOX_ENFORCE_DEST"
+	echo -e "${GREEN}VirtualBox enforcement script installed to ${VBOX_ENFORCE_DEST}${NC}"
 else
-  echo -e "${YELLOW}VirtualBox enforcement script not found, skipping...${NC}"
+	echo -e "${YELLOW}VirtualBox enforcement script not found, skipping...${NC}"
 fi

 # Create symbolic link
@ -243,12 +305,12 @@ ln -sf "$WRAPPER_DEST" /usr/bin/pacman
 echo -e "${GREEN}Installation complete!${NC}"
 echo -e "Pacman is now wrapped. The original pacman is available at ${CYAN}/usr/bin/pacman.orig${NC}"
 if [ -f "$MAKEPKG_CAPPED_DEST" ]; then
-  echo -e "Run constrained package builds with: ${CYAN}pacman --makepkg-capped <args>${NC}"
+	echo -e "Run constrained package builds with: ${CYAN}pacman --makepkg-capped <args>${NC}"
 fi
 if [ -f "$MKPKG_DEST" ]; then
-  echo -e "Shortcut available: ${CYAN}mkpkg <args>${NC}"
+	echo -e "Shortcut available: ${CYAN}mkpkg <args>${NC}"
 fi
 echo -e "${CYAN}Policy files are now protected with immutable attributes.${NC}"
 if [ -f "$VBOX_ENFORCE_DEST" ]; then
-  echo -e "${CYAN}VirtualBox VMs will automatically be configured to use host's /etc/hosts.${NC}"
+	echo -e "${CYAN}VirtualBox VMs will automatically be configured to use host's /etc/hosts.${NC}"
 fi
--- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/makepkg_capped.sh
+++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/makepkg_capped.sh
@ -11,33 +11,48 @@ IONICE_CLASS="${MAKEPKG_IONICE_CLASS:-2}"
 IONICE_LEVEL="${MAKEPKG_IONICE_LEVEL:-7}"

 if ! command -v makepkg >/dev/null 2>&1; then
-  echo "makepkg_capped: makepkg not found in PATH" >&2
-  exit 127
+	echo "makepkg_capped: makepkg not found in PATH" >&2
+	exit 127
+fi
+
+# Serialise against other heavy jobs (Gradle/Flutter builds, coverage runs,
+# pacman transactions) on this 7.6 GB machine. Re-exec ourselves under the lock
+# instead of wrapping each branch below, so all three fallback paths are covered
+# by one guard. HEAVY_JOB_LOCK_HELD keeps the re-exec from recursing.
+# Note the cgroup caps below are NOT a substitute: MemoryMax defaults to 12G on
+# a machine with 7.6G, so they bound one build's shape, not the machine's total.
+HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh"
+if [[ -r $HEAVY_JOB_LOCK_LIB && -z ${HEAVY_JOB_LOCK_HELD:-} ]]; then
+	# shellcheck source=/dev/null
+	source "$HEAVY_JOB_LOCK_LIB"
+	export HEAVY_JOB_LOCK_HELD=1
+	with_heavy_lock makepkg -- "$0" "$@"
+	exit $?
 fi

 run_makepkg_fallback() {
-  exec ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \
-    nice -n "$NICE_LEVEL" \
-    makepkg "$@"
+	exec ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \
+		nice -n "$NICE_LEVEL" \
+		makepkg "$@"
 }

 if ! command -v systemd-run >/dev/null 2>&1; then
-  echo "makepkg_capped: systemd-run unavailable, falling back to local limits" >&2
-  run_makepkg_fallback "$@"
+	echo "makepkg_capped: systemd-run unavailable, falling back to local limits" >&2
+	run_makepkg_fallback "$@"
 fi

 # Keep builds in the user manager scope. If this fails (no user manager),
 # gracefully fall back to ionice+nice only.
 if ! systemd-run --user --scope --quiet true 2>/dev/null; then
-  echo "makepkg_capped: user systemd scope unavailable, falling back" >&2
-  run_makepkg_fallback "$@"
+	echo "makepkg_capped: user systemd scope unavailable, falling back" >&2
+	run_makepkg_fallback "$@"
 fi

 exec systemd-run --user --scope --quiet --same-dir --collect \
-  -p "CPUQuota=${CPU_QUOTA}" \
-  -p "MemoryMax=${MEMORY_MAX}" \
-  -p "MemorySwapMax=0" \
-  -p "TasksMax=${TASKS_MAX}" \
-  ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \
-  nice -n "$NICE_LEVEL" \
-  makepkg "$@"
+	-p "CPUQuota=${CPU_QUOTA}" \
+	-p "MemoryMax=${MEMORY_MAX}" \
+	-p "MemorySwapMax=0" \
+	-p "TasksMax=${TASKS_MAX}" \
+	ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \
+	nice -n "$NICE_LEVEL" \
+	makepkg "$@"
--- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/pacman_wrapper.sh
+++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/pacman_wrapper.sh
@ -12,6 +12,10 @@ BOLD='\033[1m'
 NC='\033[0m' # No Color

 PACMAN_BIN="/usr/bin/pacman"
+# Installed alongside the wrapper by install_pacman_wrapper.sh. Absent on a
+# machine that has not run it yet, in which case transactions simply are not
+# serialised (see the guarded source below).
+HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh"
 MAKEPKG_CAPPED_BIN="/usr/local/bin/makepkg_capped"

 declare -a BLOCKED_KEYWORDS_LIST=()
@ -87,7 +91,7 @@ load_policy_lists() {
 				continue
 			fi
 			target_array+=("${line,,}")
-		done < "$file_path"
+		done <"$file_path"
 	}

 	if [[ -f $blocked_file ]]; then
@ -741,7 +745,19 @@ if should_use_wrapper_guard_lib_fallback "$@"; then
 	manual_guard_lib_fallback=1
 fi

-"$PACMAN_BIN" "$@"
+# Serialise real TRANSACTIONS against builds and coverage runs on this 7.6 GB
+# box (see utils/heavy_job_lock.sh). Deliberately scoped by needs_unlock, i.e.
+# -S/-U/-R only: read-only queries like `pacman -Q` take no db lock and run
+# constantly from background services here, so making those wait behind a
+# 20-minute Gradle build would be worse than the contention it avoids.
+# with_heavy_lock always fails open, so this can never brick package management.
+if [[ -r $HEAVY_JOB_LOCK_LIB ]] && needs_unlock "$@"; then
+	# shellcheck source=/dev/null
+	source "$HEAVY_JOB_LOCK_LIB"
+	with_heavy_lock pacman -- "$PACMAN_BIN" "$@"
+else
+	"$PACMAN_BIN" "$@"
+fi
 exit_code=$?

 if [[ $manual_guard_lib_fallback -eq 1 ]]; then
--- a/linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh
+++ b/linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh
@ -0,0 +1,232 @@
+#!/bin/bash
+
+# ============================================================================
+# heavy_job_lock.sh - one heavy job at a time on this machine
+#
+# This box has 7.6 GB. On 2026-08-03 a Gradle/Flutter APK build, a full
+# `pytest --cov` run and Chromium overlapped; an earlier overlap the same week
+# got a whole session OOM-killed (exit 137/143). The rule "one heavy job at a
+# time" was written down in ~/.claude/memories/mistakes.md - but a rule that
+# only works when a human remembers it is deployment hygiene, not a fix. This
+# makes the overlap impossible instead.
+#
+# Usage as a CLI:
+#     heavy_job_lock.sh --name gradle -- flutter build apk --debug
+#     heavy_job_lock.sh --name pytest --fail-fast -- pytest --cov
+#
+# Usage as a library:
+#     source /path/to/heavy_job_lock.sh
+#     with_heavy_lock gradle -- ./gradlew assembleDebug
+#
+# Setup (once, as root - /run is tmpfs so the lock needs recreating on boot):
+#     sudo heavy_job_lock.sh --install
+#
+# Design notes:
+#   - ONE system-wide lock shared between root and the user. pacman runs as
+#     root while builds run as the user, so a per-user /run/user/$UID lock
+#     would give them separate locks and no mutual exclusion at all.
+#   - The lock file is mode 0666 (created by root via tmpfiles.d) precisely so
+#     an unprivileged build can take the same lock a root pacman takes.
+#   - FAIL OPEN, always. A lock this central must never be able to brick
+#     package management or builds: if the lock file cannot be created, or the
+#     wait times out, we warn and run the command anyway. Blocking work forever
+#     is a worse failure than the memory contention we are avoiding.
+# ============================================================================
+
+HEAVY_JOB_LOCK_FILE="${HEAVY_JOB_LOCK_FILE:-/run/heavy-job.lock}"
+# Long enough for a real Gradle build to finish ahead of us, short enough that
+# a wedged holder cannot stall a machine indefinitely.
+HEAVY_JOB_LOCK_TIMEOUT="${HEAVY_JOB_LOCK_TIMEOUT:-1800}"
+HEAVY_JOB_TMPFILES_CONF="/etc/tmpfiles.d/heavy-job-lock.conf"
+
+_hjl_warn() { printf '[heavy-job-lock] %s\n' "$*" >&2; }
+
+# Who holds it right now? Best-effort: the holder writes "<name> <pid> <date>"
+# after acquiring, so a refusal can name the culprit instead of just failing.
+_hjl_holder() {
+	local holder=""
+	[[ -r $HEAVY_JOB_LOCK_FILE ]] && read -r holder <"$HEAVY_JOB_LOCK_FILE" 2>/dev/null
+	printf '%s' "${holder:-unknown}"
+}
+
+# Create the lock file world-writable. Only meaningful as root; unprivileged
+# callers just discover they cannot and fail open.
+_hjl_ensure_lock_file() {
+	[[ -e $HEAVY_JOB_LOCK_FILE ]] && return 0
+	if [[ $EUID -eq 0 ]]; then
+		install -m 0666 /dev/null "$HEAVY_JOB_LOCK_FILE" 2>/dev/null && return 0
+	else
+		# Braces around the redirect: a failing redirection is reported by the
+		# shell itself, before the command runs, so `cmd 2>/dev/null` does not
+		# suppress it. The group does.
+		{ : >"$HEAVY_JOB_LOCK_FILE"; } 2>/dev/null &&
+			chmod 0666 "$HEAVY_JOB_LOCK_FILE" 2>/dev/null && return 0
+	fi
+	return 1
+}
+
+# with_heavy_lock <name> [--fail-fast] -- <command...>
+with_heavy_lock() {
+	local name="$1"
+	shift
+	local fail_fast=0
+
+	while [[ $# -gt 0 ]]; do
+		case "$1" in
+		--fail-fast)
+			fail_fast=1
+			shift
+			;;
+		--)
+			shift
+			break
+			;;
+		*) break ;;
+		esac
+	done
+
+	if [[ $# -eq 0 ]]; then
+		_hjl_warn "with_heavy_lock: no command given"
+		return 2
+	fi
+
+	if ! _hjl_ensure_lock_file; then
+		_hjl_warn "cannot create $HEAVY_JOB_LOCK_FILE - running '$name' UNSERIALISED"
+		_hjl_warn "run 'sudo $0 --install' once to enable serialisation"
+		"$@"
+		return $?
+	fi
+
+	local lock_fd
+	# `<>` (read-write), NOT `>`. Redirecting with `>` truncates the file at
+	# open time, so a second caller wiped the holder's "<name> <pid> <date>"
+	# line before it could read it — every refusal said "held by unknown".
+	exec {lock_fd}<>"$HEAVY_JOB_LOCK_FILE" || {
+		_hjl_warn "cannot open lock - running '$name' unserialised"
+		"$@"
+		return $?
+	}
+
+	local acquired=0
+	if ((fail_fast == 1)); then
+		if flock --nonblock "$lock_fd"; then
+			acquired=1
+		else
+			_hjl_warn "refusing to start '$name': held by $(_hjl_holder)"
+			exec {lock_fd}>&-
+			return 75 # EX_TEMPFAIL
+		fi
+	else
+		local holder
+		holder="$(_hjl_holder)"
+		if ! flock --nonblock "$lock_fd"; then
+			_hjl_warn "waiting for '$holder' to finish before starting '$name' (timeout ${HEAVY_JOB_LOCK_TIMEOUT}s)"
+		fi
+		if flock --timeout "$HEAVY_JOB_LOCK_TIMEOUT" "$lock_fd"; then
+			acquired=1
+		else
+			# Fail open: see the header. A stuck holder must not permanently
+			# block builds or package management.
+			_hjl_warn "timed out after ${HEAVY_JOB_LOCK_TIMEOUT}s waiting for $(_hjl_holder)"
+			_hjl_warn "proceeding UNSERIALISED with '$name'"
+		fi
+	fi
+
+	if ((acquired == 1)); then
+		printf '%s %s %s\n' "$name" "$$" "$(date -Is 2>/dev/null || echo unknown)" \
+			>"$HEAVY_JOB_LOCK_FILE" 2>/dev/null || true
+	fi
+
+	local rc=0
+	"$@" || rc=$?
+
+	# Closing the fd releases the lock; do it explicitly so the holder line is
+	# not left implying we still hold it.
+	exec {lock_fd}>&-
+	return "$rc"
+}
+
+# ----------------------------------------------------------------------------
+# CLI (skipped when this file is sourced as a library)
+# ----------------------------------------------------------------------------
+_hjl_install() {
+	if [[ $EUID -ne 0 ]]; then
+		_hjl_warn "--install must run as root"
+		return 1
+	fi
+	# /run is tmpfs: without a tmpfiles.d entry the lock vanishes every boot and
+	# the first unprivileged caller silently falls back to unserialised.
+	printf 'f %s 0666 root root -\n' "$HEAVY_JOB_LOCK_FILE" >"$HEAVY_JOB_TMPFILES_CONF"
+	systemd-tmpfiles --create "$HEAVY_JOB_TMPFILES_CONF" 2>/dev/null || true
+	_hjl_ensure_lock_file || {
+		_hjl_warn "failed to create $HEAVY_JOB_LOCK_FILE"
+		return 1
+	}
+	printf 'installed %s and %s\n' "$HEAVY_JOB_TMPFILES_CONF" "$HEAVY_JOB_LOCK_FILE"
+}
+
+_hjl_usage() {
+	cat <<EOF
+Usage: $(basename "$0") --name <label> [--fail-fast] -- <command...>
+       $(basename "$0") --install     # once, as root
+       $(basename "$0") --status
+
+Serialises heavy jobs (builds, coverage runs, pacman transactions) so only one
+runs at a time on this memory-constrained machine. Always fails open.
+
+Options:
+  --name <label>   Name recorded as the lock holder (shown to whoever waits)
+  --fail-fast      Refuse immediately if held, instead of waiting
+  --install        Create the lock file + tmpfiles.d entry (root)
+  --status         Show the current holder
+EOF
+}
+
+# ${BASH_SOURCE[0]} != $0 means we were sourced - define functions, run nothing.
+if [[ ${BASH_SOURCE[0]} == "${0}" ]]; then
+	set -uo pipefail
+
+	_name="heavy-job"
+	_fail_fast=()
+	while [[ $# -gt 0 ]]; do
+		case "$1" in
+		--name)
+			_name="$2"
+			shift 2
+			;;
+		--fail-fast)
+			_fail_fast=(--fail-fast)
+			shift
+			;;
+		--install)
+			_hjl_install
+			exit $?
+			;;
+		--status)
+			printf 'lock file: %s\n' "$HEAVY_JOB_LOCK_FILE"
+			printf 'holder:    %s\n' "$(_hjl_holder)"
+			exit 0
+			;;
+		-h | --help)
+			_hjl_usage
+			exit 0
+			;;
+		--)
+			shift
+			break
+			;;
+		*)
+			_hjl_warn "unknown option: $1"
+			exit 2
+			;;
+		esac
+	done
+
+	if [[ $# -eq 0 ]]; then
+		_hjl_usage
+		exit 2
+	fi
+
+	with_heavy_lock "$_name" "${_fail_fast[@]}" -- "$@"
+	exit $?
+fi
--- a/linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh
+++ b/linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh
@ -0,0 +1,493 @@
+#!/bin/bash
+
+# ============================================================================
+# diagnose_pacman_hook_stall.sh
+#
+# Reproduces and instruments the intermittent pacman freeze that happens at the
+# FIRST PreTransaction hook of a transaction:
+#
+#     :: Running pre-transaction hooks...
+#     (1/2) guard-lib: unlocking protected files before pacman transaction
+#     <hangs for minutes>
+#
+# /var/log/pacman.log shows the hook's "running '...hook'..." line and then
+# nothing at all - the next entry is the user's retry. Five such transactions
+# were abandoned between 2026-07-29 and 2026-08-03. The hook script itself is
+# provably a no-op on this machine (/etc/guard-lib/targets is empty, so its
+# loop body never executes), which means the stall is pacman forking/exec'ing
+# the hook, not the hook doing work.
+#
+# This script drives real transactions in a loop and, when one stalls, captures
+# the kernel-side evidence needed to tell "blocked in a syscall" from "the box
+# could not spawn a process": /proc/<pid>/stack, wchan, syscall, the process
+# tree, and memory/PSI counters.
+#
+# The repro target is a 0-file meta-package (base-devel), reinstalled from the
+# local cache: a real transaction that fires every hook but writes no files and
+# needs no network.
+# ============================================================================
+
+set -euo pipefail
+
+SCRIPT_NAME="$(basename "$0")"
+readonly SCRIPT_NAME
+readonly PACMAN_BIN="/usr/bin/pacman.orig"
+readonly PACMAN_LOG="/var/log/pacman.log"
+readonly CACHE_DIR="/var/cache/pacman/pkg"
+
+# Defaults
+RUNS=40
+PACKAGE="base-devel"
+STALL_TIMEOUT=20 # seconds with no new pacman.log line => declare a stall
+HARD_TIMEOUT=120 # seconds before we give up and kill the stalled transaction
+OUT_DIR="/var/log/pacman-hook-stall"
+WITH_LOAD=0
+WATCH_MODE=0
+LOAD_FLOOR_MB=800     # never push MemAvailable below this
+LOAD_MIN_FREE_MB=1500 # refuse to start --with-load below this much available
+
+HOG_FILE=""
+LOAD_PID=""
+PACMAN_PID=""
+STALLS=0
+RUN_INDEX=0
+LAST_ELAPSED=0
+
+usage() {
+	cat <<EOF
+Usage: $SCRIPT_NAME [OPTIONS]
+
+Drives repeated pacman transactions and captures diagnostics when one stalls
+at its first PreTransaction hook.
+
+Options:
+  -n, --runs N          Number of transactions to drive (default: $RUNS)
+  -p, --package NAME    Package to reinstall from cache (default: $PACKAGE)
+  -t, --timeout S       Seconds of pacman.log silence => stall (default: $STALL_TIMEOUT)
+      --hard-timeout S  Seconds before killing a stalled run (default: $HARD_TIMEOUT)
+      --with-load       Also apply memory pressure (the hypothesis under test).
+                        Run this with nothing else going - it deliberately
+                        breaks the one-heavy-job-at-a-time rule.
+      --watch           Passive mode: drive nothing, just watch pacman.log and
+                        dump diagnostics when someone else's transaction stalls
+                        on a hook. Intended to run as a systemd service.
+  -o, --out DIR         Where to write stall dumps (default: $OUT_DIR)
+  -h, --help            Show this help
+
+Exit status: 0 if the loop completed (with or without stalls), non-zero on a
+setup failure. The stall count is reported in the summary.
+EOF
+	exit 0
+}
+
+cleanup() {
+	local rc=$?
+	stop_load
+	if [[ -n "$PACMAN_PID" ]] && kill -0 "$PACMAN_PID" 2>/dev/null; then
+		echo "Cleaning up: killing in-flight pacman ($PACMAN_PID)" >&2
+		kill_tree "$PACMAN_PID"
+	fi
+	exit "$rc"
+}
+
+trap cleanup EXIT INT TERM
+
+# ----------------------------------------------------------------------------
+# Preconditions
+# ----------------------------------------------------------------------------
+
+require_root() {
+	if [[ $EUID -ne 0 ]]; then
+		exec sudo -E bash "$0" "$@"
+	fi
+}
+
+validate_requirements() {
+	local tool
+	for tool in "$PACMAN_BIN" ps awk sed journalctl; do
+		command -v "$tool" >/dev/null 2>&1 || [[ -x "$tool" ]] || {
+			echo "Error: required tool '$tool' not found" >&2
+			exit 1
+		}
+	done
+
+	[[ -r "$PACMAN_LOG" ]] || {
+		echo "Error: cannot read $PACMAN_LOG" >&2
+		exit 1
+	}
+
+	# Refuse to run alongside another *transaction*: a collision would look
+	# exactly like the stall we are hunting. db.lck is the authoritative signal
+	# - pacman holds it for the whole transaction. Deliberately NOT a pgrep for
+	# "pacman": read-only queries (pacman -Qi) take no lock, are harmless, and
+	# run constantly here from background services.
+	if [[ -e /var/lib/pacman/db.lck ]]; then
+		echo "Error: /var/lib/pacman/db.lck exists - another transaction is in" >&2
+		echo "flight (or a stale lock remains). Resolve it before running." >&2
+		exit 1
+	fi
+}
+
+# Resolve the cached package file for the installed version of $PACKAGE.
+resolve_package_file() {
+	local version
+	version="$("$PACMAN_BIN" -Q "$PACKAGE" 2>/dev/null | awk '{print $2}')"
+	[[ -n "$version" ]] || {
+		echo "Error: package '$PACKAGE' is not installed" >&2
+		exit 1
+	}
+
+	local candidate
+	for candidate in "$CACHE_DIR/$PACKAGE-$version"-*.pkg.tar.zst; do
+		[[ -e "$candidate" ]] || continue
+		printf '%s\n' "$candidate"
+		return 0
+	done
+
+	echo "Error: no cached package for $PACKAGE-$version in $CACHE_DIR" >&2
+	echo "Hint: run '$PACMAN_BIN -Sw $PACKAGE' first" >&2
+	exit 1
+}
+
+# ----------------------------------------------------------------------------
+# Load generator (tmpfs allocation = real anonymous-ish memory, no extra deps)
+# ----------------------------------------------------------------------------
+
+mem_available_mb() {
+	local kb=0 key value rest
+	while read -r key value rest; do
+		if [[ $key == "MemAvailable:" ]]; then
+			kb="$value"
+			break
+		fi
+	done </proc/meminfo
+	printf '%d\n' $((kb / 1024))
+}
+
+start_load() {
+	((WITH_LOAD == 1)) || return 0
+
+	local avail
+	avail="$(mem_available_mb)"
+	if ((avail < LOAD_MIN_FREE_MB)); then
+		echo "Error: only ${avail} MB available; --with-load needs >= ${LOAD_MIN_FREE_MB} MB" >&2
+		echo "Close Chromium / builds first." >&2
+		exit 1
+	fi
+
+	local target=$((avail - LOAD_FLOOR_MB))
+	HOG_FILE="$(mktemp /dev/shm/hook-stall-hog.XXXXXX)"
+	echo "Applying memory pressure: allocating ${target} MB (floor ${LOAD_FLOOR_MB} MB available)"
+	dd if=/dev/zero of="$HOG_FILE" bs=1M count="$target" status=none 2>/dev/null || true
+	echo "MemAvailable now: $(mem_available_mb) MB"
+}
+
+stop_load() {
+	if [[ -n "$LOAD_PID" ]] && kill -0 "$LOAD_PID" 2>/dev/null; then
+		kill "$LOAD_PID" 2>/dev/null || true
+	fi
+	if [[ -n "$HOG_FILE" && -e "$HOG_FILE" ]]; then
+		rm -f "$HOG_FILE"
+		HOG_FILE=""
+	fi
+}
+
+# ----------------------------------------------------------------------------
+# Diagnostics capture
+# ----------------------------------------------------------------------------
+
+# Print every descendant PID of $1, including $1 itself.
+descendant_pids() {
+	local root="$1"
+	local pids=("$root")
+	local i=0
+	while ((i < ${#pids[@]})); do
+		local parent="${pids[i]}"
+		local child
+		while read -r child; do
+			[[ -n "$child" ]] && pids+=("$child")
+		done < <(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ')
+		i=$((i + 1))
+	done
+	printf '%s\n' "${pids[@]}"
+}
+
+kill_tree() {
+	local root="$1"
+	local pid
+	while read -r pid; do
+		kill -9 "$pid" 2>/dev/null || true
+	done < <(descendant_pids "$root" | tac)
+}
+
+capture_stall() {
+	local root_pid="$1"
+	local run="$2"
+	local stamp
+	printf -v stamp '%(%Y%m%d-%H%M%S)T' -1
+
+	# Classify before capturing. The stall we are hunting leaves pacman.log
+	# ending on a hook's "running '...hook'..." line; anything else stalled
+	# somewhere unrelated and must not be counted as a reproduction.
+	local kind="other"
+	if tail -1 "$PACMAN_LOG" 2>/dev/null | grep -q "running '.*\.hook'"; then
+		kind="hook"
+	fi
+
+	local dir="$OUT_DIR/${stamp}-run${run}-${kind}"
+	mkdir -p "$dir"
+
+	echo "  !! STALL ($kind) - capturing diagnostics to $dir"
+	tail -1 "$PACMAN_LOG" >"$dir/last-log-line.txt" 2>&1 || true
+
+	ps -eo pid,ppid,stat,wchan:32,etime,args --forest >"$dir/ps-forest.txt" 2>&1 || true
+
+	local pid
+	while read -r pid; do
+		[[ -d "/proc/$pid" ]] || continue
+		{
+			echo "=== pid $pid ==="
+			echo "--- cmdline ---"
+			tr '\0' ' ' <"/proc/$pid/cmdline" 2>/dev/null || true
+			echo
+			echo "--- wchan ---"
+			cat "/proc/$pid/wchan" 2>/dev/null || true
+			echo
+			echo "--- syscall ---"
+			cat "/proc/$pid/syscall" 2>/dev/null || true
+			echo "--- stack ---"
+			cat "/proc/$pid/stack" 2>/dev/null || echo "(unavailable - needs CONFIG_STACKTRACE)"
+			echo "--- status ---"
+			grep -E "^(Name|State|Threads|VmRSS|SigBlk|SigIgn|SigCgt)" "/proc/$pid/status" 2>/dev/null || true
+			echo "--- smaps_rollup (Pss) ---"
+			grep -E "^Pss:" "/proc/$pid/smaps_rollup" 2>/dev/null || true
+			echo
+		} >>"$dir/procs.txt" 2>&1
+	done < <(descendant_pids "$root_pid")
+
+	cp /proc/meminfo "$dir/meminfo.txt" 2>/dev/null || true
+	{
+		echo "--- pressure/memory ---"
+		cat /proc/pressure/memory 2>/dev/null || true
+		echo "--- pressure/io ---"
+		cat /proc/pressure/io 2>/dev/null || true
+		echo "--- pressure/cpu ---"
+		cat /proc/pressure/cpu 2>/dev/null || true
+	} >"$dir/pressure.txt" 2>&1
+
+	journalctl -k -n 50 --no-pager >"$dir/dmesg-tail.txt" 2>&1 || true
+	tail -30 "$PACMAN_LOG" >"$dir/pacman-log-tail.txt" 2>&1 || true
+
+	STALLS=$((STALLS + 1))
+}
+
+# ----------------------------------------------------------------------------
+# The repro loop
+# ----------------------------------------------------------------------------
+
+log_size() { stat -c %s "$PACMAN_LOG" 2>/dev/null || echo 0; }
+
+# Run one transaction, watching pacman.log for silence. Sets LAST_ELAPSED and,
+# when it hangs, bumps STALLS via capture_stall. Deliberately NOT called in a
+# command substitution: both counters and PACMAN_PID (which the EXIT trap needs)
+# must live in this shell, not a subshell.
+run_one() {
+	local pkg_file="$1"
+	local run="$2"
+
+	local started
+	printf -v started '%(%s)T' -1
+
+	"$PACMAN_BIN" -U --noconfirm "$pkg_file" >/dev/null 2>&1 &
+	PACMAN_PID=$!
+
+	local last_size last_change now captured=0
+	last_size="$(log_size)"
+	printf -v last_change '%(%s)T' -1
+
+	while kill -0 "$PACMAN_PID" 2>/dev/null; do
+		sleep 1
+		local size
+		size="$(log_size)"
+		printf -v now '%(%s)T' -1
+
+		if [[ $size != "$last_size" ]]; then
+			last_size="$size"
+			last_change="$now"
+		elif ((captured == 0 && now - last_change >= STALL_TIMEOUT)); then
+			capture_stall "$PACMAN_PID" "$run"
+			captured=1
+		fi
+
+		if ((now - started >= HARD_TIMEOUT)); then
+			echo "  !! hard timeout (${HARD_TIMEOUT}s) - killing run $run"
+			((captured == 0)) && capture_stall "$PACMAN_PID" "$run"
+			kill_tree "$PACMAN_PID"
+			break
+		fi
+	done
+
+	wait "$PACMAN_PID" 2>/dev/null || true
+	PACMAN_PID=""
+
+	local ended
+	printf -v ended '%(%s)T' -1
+	LAST_ELAPSED=$((ended - started))
+}
+
+# Passive mode: watch OTHER people's transactions instead of driving our own.
+#
+# This exists because the active repro loop could not reproduce the stall: 65
+# transactions (40 idle + 25 under 2.6 GB of memory pressure) ran clean, against
+# a historical rate of 4 in 56. At a uniform 7% the odds of 65 clean runs are
+# under 1%, so the trigger is conditional on something a synthetic loop does not
+# recreate - which means the evidence has to be captured from a real stall, when
+# it next happens, rather than manufactured.
+#
+# Runs as a systemd service; costs one stat(2) per second and nothing else.
+watch_forever() {
+	echo "Watching $PACMAN_LOG for hook stalls (>= ${STALL_TIMEOUT}s silent)"
+	echo "Dumps: $OUT_DIR"
+
+	local last_size last_change now armed=0 seq=0
+	last_size="$(log_size)"
+	printf -v last_change '%(%s)T' -1
+
+	while true; do
+		sleep 1
+		local size
+		size="$(log_size)"
+		printf -v now '%(%s)T' -1
+
+		if [[ $size != "$last_size" ]]; then
+			last_size="$size"
+			last_change="$now"
+			armed=0
+			continue
+		fi
+
+		# Silence alone is normal - no transaction is running most of the time.
+		# Only a LIVE pacman sitting on a hook line is the signature we want.
+		((armed == 1)) && continue
+		((now - last_change >= STALL_TIMEOUT)) || continue
+
+		local pacman_pid
+		pacman_pid="$(pgrep -x pacman.orig | head -1 || true)"
+		[[ -n $pacman_pid ]] || continue
+
+		seq=$((seq + 1))
+		capture_stall "$pacman_pid" "watch${seq}"
+		logger -t pacman-hook-stall "captured a stalled transaction (pid $pacman_pid)"
+		armed=1
+	done
+}
+
+main() {
+	validate_requirements
+
+	if ((WATCH_MODE == 1)); then
+		mkdir -p "$OUT_DIR"
+		watch_forever
+		return
+	fi
+
+	local pkg_file
+	pkg_file="$(resolve_package_file)"
+
+	mkdir -p "$OUT_DIR"
+
+	echo "============================================================================"
+	echo "pacman hook-stall diagnosis"
+	echo "============================================================================"
+	echo "  package     : $PACKAGE ($pkg_file)"
+	echo "  runs        : $RUNS"
+	echo "  stall after : ${STALL_TIMEOUT}s of pacman.log silence"
+	echo "  hard timeout: ${HARD_TIMEOUT}s"
+	local load_desc="idle"
+	((WITH_LOAD == 1)) && load_desc="memory pressure ON"
+	echo "  load        : $load_desc"
+	echo "  dumps       : $OUT_DIR"
+	echo
+
+	start_load
+
+	local durations=()
+	while ((RUN_INDEX < RUNS)); do
+		RUN_INDEX=$((RUN_INDEX + 1))
+		run_one "$pkg_file" "$RUN_INDEX"
+		durations+=("$LAST_ELAPSED")
+		local marker=""
+		((LAST_ELAPSED >= STALL_TIMEOUT)) && marker="   <-- slow"
+		printf '  run %2d/%d: %3ds%s\n' "$RUN_INDEX" "$RUNS" "$LAST_ELAPSED" "$marker"
+	done
+
+	stop_load
+
+	echo
+	echo "============================================================================"
+	echo "Summary"
+	echo "============================================================================"
+	printf '  transactions : %d\n' "$RUNS"
+	printf '  stalls       : %d\n' "$STALLS"
+	printf '  durations    : %s\n' "${durations[*]}"
+	printf '%s\n' "${durations[@]}" | sort -n | awk '
+        { d[NR] = $1 }
+        END {
+            if (NR == 0) exit
+            printf "  min/median/max: %ds / %ds / %ds\n", d[1], d[int((NR+1)/2)], d[NR]
+        }'
+	if ((STALLS > 0)); then
+		echo
+		echo "  Captured dumps:"
+		find "$OUT_DIR" -mindepth 1 -maxdepth 1 -type d -printf '    %p\n' \
+			2>/dev/null | sort | tail -"$STALLS"
+	fi
+	echo "============================================================================"
+}
+
+# Escalate BEFORE parsing: after the parse loop "$@" has been shifted away, so
+# re-exec'ing under sudo there would silently drop every flag.
+require_root "$@"
+
+# Parse arguments
+while [[ $# -gt 0 ]]; do
+	case $1 in
+	-n | --runs)
+		RUNS="$2"
+		shift 2
+		;;
+	-p | --package)
+		PACKAGE="$2"
+		shift 2
+		;;
+	-t | --timeout)
+		STALL_TIMEOUT="$2"
+		shift 2
+		;;
+	--hard-timeout)
+		HARD_TIMEOUT="$2"
+		shift 2
+		;;
+	--with-load)
+		WITH_LOAD=1
+		shift
+		;;
+	--watch)
+		WATCH_MODE=1
+		shift
+		;;
+	-o | --out)
+		OUT_DIR="$2"
+		shift 2
+		;;
+	-h | --help)
+		usage
+		;;
+	*)
+		echo "Unknown option: $1" >&2
+		exit 1
+		;;
+	esac
+done
+
+main
--- a/linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh
+++ b/linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh
@ -0,0 +1,444 @@
+#!/bin/bash
+
+# ============================================================================
+# migrate_hosts_guard_to_guard_lib.sh
+#
+# Completes the guard-lib migration that commit 66c4698 started but never
+# finished on this machine.
+#
+# State this repairs: guard-lib is installed (guardctl, lib/, systemd templates,
+# and BOTH generic pacman hooks) with /etc/guard-lib/targets completely empty,
+# while the pre-guard-lib hosts-guard implementation still does all the real
+# work through its own per-file pacman hooks and systemd units. So every pacman
+# transaction runs two PreTransaction hooks where one would do, and the
+# guard-lib pair - the one that runs FIRST, and the one present for all four
+# recorded hook stalls - does literally nothing.
+#
+# After this runs there is one guard system: guardctl file-guard instances for
+# hosts, nsswitch and resolved, driven by guard-lib's generic unlock-all /
+# relock-all hooks. The legacy hooks and units are retired.
+#
+# NOT migrated here: the "shutdown-schedule" instance that
+# setup_midnight_shutdown.sh registers. Its target /etc/shutdown-schedule.conf
+# does not exist on this machine, so registering it would mean installing a
+# different feature, not finishing this migration. Run that script if you want
+# the midnight-shutdown guard.
+#
+# Every step is idempotent, and --rollback restores the legacy layer.
+# ============================================================================
+
+set -euo pipefail
+
+SCRIPT_NAME="$(basename "$0")"
+readonly SCRIPT_NAME
+readonly GUARDCTL="/usr/local/bin/guardctl"
+readonly TARGETS_DIR="/etc/guard-lib/targets"
+readonly STATE_DIR="/var/lib/guard-lib-migration"
+readonly HOOKS_DIR="/etc/pacman.d/hooks"
+# Plugins must live somewhere permanent: guardctl records an ABSOLUTE plugin
+# path in the instance conf, so pointing it at a repo checkout (or worse, a git
+# worktree) silently breaks enforcement the day that directory moves.
+readonly PLUGIN_INSTALL_DIR="/usr/local/share/guard-lib-plugins"
+
+readonly LEGACY_HOOKS=(
+	"10-unlock-etc-hosts.hook"
+	"90-relock-etc-hosts.hook"
+)
+readonly LEGACY_UNITS=(
+	"hosts-guard.path"
+	"hosts-guard.service"
+	"hosts-bind-mount.service"
+	"nsswitch-guard.path"
+	"nsswitch-guard.service"
+	"resolved-guard.path"
+	"resolved-guard.service"
+)
+
+DRY_RUN=0
+MODE="migrate"
+
+SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
+readonly SCRIPT_DIR
+PLUGIN_SRC_DIR="$(readlink -f "$SCRIPT_DIR/../../periodic_background/hosts/guard/plugins")"
+readonly PLUGIN_SRC_DIR
+
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[0;33m'
+BLUE='\033[0;34m'
+NC='\033[0m'
+
+msg() { printf "${GREEN}[✓]${NC} %s\n" "$*"; }
+note() { printf "${BLUE}[i]${NC} %s\n" "$*"; }
+warn() { printf "${YELLOW}[!]${NC} %s\n" "$*"; }
+err() { printf "${RED}[✗]${NC} %s\n" "$*" >&2; }
+
+usage() {
+	cat <<EOF
+Usage: sudo $SCRIPT_NAME [OPTIONS]
+
+Finishes the guard-lib migration for /etc/hosts, /etc/nsswitch.conf and
+/etc/systemd/resolved.conf, then retires the legacy hosts-guard layer.
+
+Options:
+      --rollback   Undo: uninstall the guard-lib instances, restore the legacy
+                   pacman hooks and re-enable the legacy units.
+      --status     Show what is currently registered; change nothing.
+      --dry-run    Print what would happen; change nothing.
+  -h, --help       Show this help
+EOF
+	exit 0
+}
+
+run() {
+	if ((DRY_RUN == 1)); then
+		printf "${YELLOW}DRY-RUN:${NC} %s\n" "$*"
+		return 0
+	fi
+	"$@"
+}
+
+require_root() {
+	if [[ $EUID -ne 0 ]]; then
+		exec sudo -E bash "$0" "$@"
+	fi
+}
+
+# ----------------------------------------------------------------------------
+# Instance table: name | target | bind_mount | plugin basename | also_watch
+# ----------------------------------------------------------------------------
+instance_spec() { # <name> -> echoes "target|bind|plugin|also_watch"
+	case "$1" in
+	hosts) echo "/etc/hosts|yes||" ;;
+	nsswitch) echo "/etc/nsswitch.conf|no|nsswitch-plugin.sh|" ;;
+	resolved) echo "/etc/systemd/resolved.conf|no|resolved-plugin.sh|/etc/systemd/resolved.conf.d" ;;
+	*) return 1 ;;
+	esac
+}
+
+readonly INSTANCES=(hosts nsswitch resolved)
+
+instance_registered() { [[ -f "$TARGETS_DIR/$1.conf" ]]; }
+
+# ----------------------------------------------------------------------------
+# Preconditions
+# ----------------------------------------------------------------------------
+validate_requirements() {
+	[[ -x $GUARDCTL ]] || {
+		err "guardctl not found at $GUARDCTL - run guard-lib's install.sh first"
+		exit 1
+	}
+
+	local unit
+	for unit in guard-file@.path guard-file@.service guard-bind-mount@.service; do
+		[[ -f "/etc/systemd/system/$unit" ]] || {
+			err "missing systemd template /etc/systemd/system/$unit - run guard-lib's install.sh first"
+			exit 1
+		}
+	done
+
+	[[ -d $PLUGIN_SRC_DIR ]] || {
+		err "plugin sources not found at $PLUGIN_SRC_DIR"
+		exit 1
+	}
+
+	# A live transaction would race every chattr and umount below.
+	if [[ -e /var/lib/pacman/db.lck ]]; then
+		err "/var/lib/pacman/db.lck exists - a pacman transaction is in flight"
+		exit 1
+	fi
+}
+
+# ----------------------------------------------------------------------------
+# Status
+# ----------------------------------------------------------------------------
+show_status() {
+	printf "\n%s\n" "=== guard-lib instances ==="
+	local name
+	for name in "${INSTANCES[@]}"; do
+		if instance_registered "$name"; then
+			msg "$name registered"
+			"$GUARDCTL" file-guard status "$name" 2>&1 | sed 's/^/      /'
+		else
+			warn "$name NOT registered"
+		fi
+	done
+
+	printf "\n%s\n" "=== legacy pacman hooks ==="
+	local hook
+	for hook in "${LEGACY_HOOKS[@]}"; do
+		if [[ -f "$HOOKS_DIR/$hook" ]]; then
+			warn "$hook still present"
+		else
+			msg "$hook retired"
+		fi
+	done
+
+	printf "\n%s\n" "=== legacy systemd units ==="
+	local unit state
+	for unit in "${LEGACY_UNITS[@]}"; do
+		# `is-enabled` PRINTS the state but EXITS 1 for anything not enabled, so
+		# `|| echo absent` would append a second line to a perfectly good answer.
+		state="$(systemctl is-enabled "$unit" 2>/dev/null)" || true
+		[[ -n $state ]] || state="absent"
+		if [[ $state == "enabled" ]]; then
+			warn "$unit is $state"
+		else
+			msg "$unit is $state"
+		fi
+	done
+}
+
+# ----------------------------------------------------------------------------
+# Rollback state
+# ----------------------------------------------------------------------------
+save_rollback_state() {
+	run mkdir -p "$STATE_DIR/hooks"
+
+	local hook
+	for hook in "${LEGACY_HOOKS[@]}"; do
+		if [[ -f "$HOOKS_DIR/$hook" && ! -f "$STATE_DIR/hooks/$hook" ]]; then
+			run cp -a "$HOOKS_DIR/$hook" "$STATE_DIR/hooks/$hook"
+			note "backed up $hook"
+		fi
+	done
+
+	# Record enablement so rollback restores the exact prior state rather than
+	# guessing (resolved-guard.* were already disabled here, for instance).
+	if [[ ! -f "$STATE_DIR/units.state" ]] && ((DRY_RUN == 0)); then
+		local unit ustate
+		for unit in "${LEGACY_UNITS[@]}"; do
+			ustate="$(systemctl is-enabled "$unit" 2>/dev/null)" || true
+			[[ -n $ustate ]] || ustate="absent"
+			printf '%s=%s\n' "$unit" "$ustate"
+		done >"$STATE_DIR/units.state"
+		note "recorded legacy unit states in $STATE_DIR/units.state"
+	fi
+}
+
+install_plugins() {
+	run mkdir -p "$PLUGIN_INSTALL_DIR"
+	local src
+	for src in "$PLUGIN_SRC_DIR"/*.sh; do
+		[[ -e $src ]] || continue
+		run install -m 755 "$src" "$PLUGIN_INSTALL_DIR/$(basename "$src")"
+	done
+	msg "plugins installed to $PLUGIN_INSTALL_DIR"
+}
+
+# ----------------------------------------------------------------------------
+# Migration
+# ----------------------------------------------------------------------------
+stop_legacy_units_for() { # <instance name>
+	local prefix="$1"
+	local unit
+	for unit in "${LEGACY_UNITS[@]}"; do
+		[[ $unit == "$prefix"-* ]] || continue
+		systemctl list-unit-files "$unit" &>/dev/null || continue
+		run systemctl disable --now "$unit" 2>/dev/null || true
+	done
+}
+
+# Unmount every stacked bind layer so chattr can reach the real inode. Mirrors
+# the legacy pacman-pre-unlock-hosts.sh, which had to do exactly this.
+# Every branch is a full `if` and the function ends in an explicit `return 0`.
+# The obvious `((i > 20)) && break` form leaves the loop body's exit status at 1
+# whenever the arithmetic is false, and under `set -e` that aborted the whole
+# rollback silently — but ONLY when the loop actually ran, so --dry-run and an
+# already-unmounted target both looked fine. It cost a live /etc/hosts left with
+# neither the immutable flag nor its read-only mount to notice.
+collapse_mounts() { # <path>
+	local target="$1" i=0
+	while mountpoint -q "$target"; do
+		if ! run umount -l "$target" 2>/dev/null; then
+			break
+		fi
+		i=$((i + 1))
+		if ((i > 20)); then
+			break
+		fi
+		# Nothing is really unmounted under --dry-run, so `mountpoint` stays
+		# true forever; report the intent once instead of 20 identical lines.
+		if ((DRY_RUN == 1)); then
+			break
+		fi
+	done
+	return 0
+}
+
+migrate_instance() { # <name>
+	local name="$1"
+	local spec target bind plugin also_watch
+	spec="$(instance_spec "$name")"
+	IFS='|' read -r target bind plugin also_watch <<<"$spec"
+
+	if instance_registered "$name"; then
+		msg "$name already registered - skipping"
+		return 0
+	fi
+
+	if [[ ! -e $target ]]; then
+		warn "$name: target $target does not exist - skipping"
+		return 0
+	fi
+
+	note "migrating $name ($target)"
+
+	stop_legacy_units_for "$name"
+	collapse_mounts "$target"
+	run chattr -i "$target" 2>/dev/null || true
+
+	local -a args=(file-guard install "$name" --target "$target")
+	[[ $bind == "yes" ]] && args+=(--bind-mount)
+	[[ -n $plugin ]] && args+=(--plugin "$PLUGIN_INSTALL_DIR/$plugin")
+	[[ -n $also_watch ]] && args+=(--also-watch "$also_watch")
+
+	run "$GUARDCTL" "${args[@]}"
+	msg "$name migrated"
+}
+
+retire_legacy_hooks() {
+	local hook removed=0
+	for hook in "${LEGACY_HOOKS[@]}"; do
+		if [[ -f "$HOOKS_DIR/$hook" ]]; then
+			run rm -f "$HOOKS_DIR/$hook"
+			msg "retired $hook"
+			removed=1
+		fi
+	done
+	((removed == 0)) && msg "legacy pacman hooks already retired"
+	return 0
+}
+
+do_migrate() {
+	validate_requirements
+	save_rollback_state
+	install_plugins
+
+	local name
+	for name in "${INSTANCES[@]}"; do
+		migrate_instance "$name"
+	done
+
+	# Only retire the legacy hooks once guard-lib actually owns something -
+	# otherwise a failed migration would leave the files with NO pacman
+	# unlock hook at all, and the next transaction would fight chattr +i.
+	local registered=0
+	for name in "${INSTANCES[@]}"; do
+		instance_registered "$name" && registered=$((registered + 1))
+	done
+
+	if ((registered == 0)) && ((DRY_RUN == 0)); then
+		err "no guard-lib instance registered - refusing to retire the legacy hooks"
+		exit 1
+	fi
+
+	retire_legacy_hooks
+
+	printf "\n"
+	msg "migration complete ($registered instance(s) registered)"
+	note "verify with: sudo $SCRIPT_NAME --status"
+	note "undo with:   sudo $SCRIPT_NAME --rollback"
+}
+
+# ----------------------------------------------------------------------------
+# Rollback
+# ----------------------------------------------------------------------------
+do_rollback() {
+	[[ -d $STATE_DIR ]] || {
+		err "no rollback state at $STATE_DIR - nothing to roll back to"
+		exit 1
+	}
+
+	local name
+	for name in "${INSTANCES[@]}"; do
+		if instance_registered "$name"; then
+			# Keep the canonical: it is the known-good copy of the file, and
+			# the legacy layer re-snapshots from the live file, which may have
+			# drifted. Losing it would turn a rollback into data loss.
+			run "$GUARDCTL" file-guard uninstall "$name" --keep-canonical || true
+			msg "uninstalled guard-lib instance $name"
+		fi
+	done
+
+	local hook
+	for hook in "${LEGACY_HOOKS[@]}"; do
+		if [[ -f "$STATE_DIR/hooks/$hook" ]]; then
+			run cp -a "$STATE_DIR/hooks/$hook" "$HOOKS_DIR/$hook"
+			msg "restored $hook"
+		fi
+	done
+
+	if [[ -f "$STATE_DIR/units.state" ]]; then
+		local unit state
+		while IFS='=' read -r unit state; do
+			[[ $state == "enabled" ]] || continue
+			run systemctl enable --now "$unit" 2>/dev/null || warn "could not re-enable $unit"
+			msg "re-enabled $unit"
+		done <"$STATE_DIR/units.state"
+	else
+		warn "no units.state recorded - legacy units left as-is"
+	fi
+
+	# Re-assert the legacy protections NOW rather than waiting for a watcher.
+	# `guardctl file-guard uninstall` chattr -i's the target on its way out, and
+	# the legacy *-guard.path units only fire on PathModified - so without this
+	# a rolled-back /etc/hosts sits with NO immutable flag until something
+	# happens to write to it. Caught by actually running --rollback and looking
+	# at lsattr, which is the only reason it is handled at all.
+	#
+	# Order matters for hosts specifically: it is bind-mounted read-only, and
+	# chattr cannot write through a ro mount. enforce-hosts.sh does a bare
+	# `chattr +i` with no collapse (it relies on running before the mount at
+	# boot), so it fails silently here unless the mount is dropped first. So:
+	# collapse, enforce, then let the bind-mount unit rebuild the ro layer.
+	collapse_mounts /etc/hosts
+
+	local svc
+	for svc in hosts-guard.service nsswitch-guard.service resolved-guard.service; do
+		systemctl cat "$svc" &>/dev/null || continue
+		run systemctl start "$svc" 2>/dev/null || warn "could not run $svc"
+	done
+
+	if systemctl cat hosts-bind-mount.service &>/dev/null; then
+		run systemctl restart hosts-bind-mount.service 2>/dev/null || warn "could not restart hosts-bind-mount.service"
+	fi
+	msg "re-ran legacy enforcement (restores chattr +i and the ro bind mount)"
+
+	printf "\n"
+	msg "rollback complete"
+	note "verify with: sudo $SCRIPT_NAME --status"
+}
+
+# Parse arguments
+require_root "$@"
+
+while [[ $# -gt 0 ]]; do
+	case "$1" in
+	--rollback)
+		MODE="rollback"
+		shift
+		;;
+	--status)
+		MODE="status"
+		shift
+		;;
+	--dry-run)
+		DRY_RUN=1
+		shift
+		;;
+	-h | --help)
+		usage
+		;;
+	*)
+		err "Unknown option: $1"
+		exit 1
+		;;
+	esac
+done
+
+case "$MODE" in
+migrate) do_migrate ;;
+rollback) do_rollback ;;
+status) show_status ;;
+esac
--- a/meta/scripts/run_with_heavy_lock.sh
+++ b/meta/scripts/run_with_heavy_lock.sh
@ -0,0 +1,31 @@
+#!/bin/bash
+# Run a command under this machine's heavy-job lock, if it has one.
+#
+# Front-end for .pre-commit-config.yaml so the config stays portable: the lock
+# lives at a machine-specific absolute path (installed by
+# install_pacman_wrapper.sh), and hardcoding that into a tracked config would
+# break the hook on any clone that has not installed it. Missing lock => run the
+# command unserialised, exactly as before.
+#
+# Usage: run_with_heavy_lock.sh <lock-name> <command> [args...]
+
+set -uo pipefail
+
+readonly HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh"
+
+if [[ $# -lt 2 ]]; then
+	echo "usage: $(basename "$0") <lock-name> <command> [args...]" >&2
+	exit 2
+fi
+
+lock_name="$1"
+shift
+
+if [[ -r $HEAVY_JOB_LOCK_LIB ]]; then
+	# shellcheck source=/dev/null
+	source "$HEAVY_JOB_LOCK_LIB"
+	with_heavy_lock "$lock_name" -- "$@"
+	exit $?
+fi
+
+exec "$@"