From 79829ec8a2a994dd6b86c1af328cd157c382bb84 Mon Sep 17 00:00:00 2001 From: Krzysztof Rudnicki Date: Mon, 3 Aug 2026 15:14:16 +0200 Subject: [PATCH] Diagnose the pacman first-hook stall; fix the drift that hid a stale wrapper pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on retry. This investigates it and repairs the real defects found on the way. The stall is NOT fixed, and deliberately so: no verified defect was found, so nothing pacman-side was changed. - Not guard-lib. /etc/guard-lib/targets was empty, so that hook script provably exits in milliseconds. The failures share POSITION (first hook of the transaction), not identity: one stall hit the hosts hook back when that one ran first. - Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB of memory pressure) produced zero stalls. Against the historical 4-in-56 rate that is ~0.9% likely, so the trigger is conditional on something a synthetic load does not recreate. Memory pressure is untested at the load that actually occurred - not refuted. So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and its --watch mode passively captures /proc//{stack,wchan,syscall}, the process tree, meminfo and PSI from the next REAL stall. No pacman-side fix should ship before that dump exists. Three real defects were found and fixed: 1. The deployed pacman wrapper was 19916 B against a 27510 B source, never sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used. It hid because check_pacman_wrapper() only tested that files EXIST. Both installers now write a drift manifest covering source AND installed hashes; the checker replays it with sha256sum -c and fails closed when it cannot verify. Manifest paths are absolute - relative ones resolve to nothing under systemd's cwd=/ and would have looped reinstalling forever. 2. The guard-lib migration (66c4698) was never finished here: guard-lib was installed with zero registered targets while the legacy hosts-guard layer did all the work, so every transaction ran two PreTransaction hooks and the first one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts, nsswitch and resolved, retires the legacy hooks and units, and has a --rollback that was actually exercised. Plugins install to a stable path because guardctl records absolute plugin paths. Side effect: one hook pair instead of two, transactions ~5-6s -> ~2-3s. 3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh makes the overlap impossible: one system-wide flock shared between root and the user (a per-user lock would give root's pacman and the user's builds separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only - read-only queries stay instant - plus makepkg_capped and the coverage hook. Always fails open. Two bugs surfaced only by running --rollback, both of which briefly left /etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while legacy watchers only fire on PathModified, and `((i > 20)) && break` left the loop body's status at 1, which under set -e aborted the rollback silently - but only when the loop actually ran, so --dry-run looked healthy. Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not exist here). Not wired: Flutter/Gradle builds, which have no single entry point. Co-Authored-By: Claude Opus 5 (1M context) --- .pre-commit-config.yaml | 5 +- .../pacman-hook-stall-2026-08-03.json | 20 + .../pacman-hook-stall-2026-08-03.json | 66 +++ .../check_and_enable_services.sh | 62 ++- .../pacman/install_makepkg_wrapper.sh | 22 + .../pacman/install_pacman_wrapper.sh | 238 +++++---- .../pacman/makepkg_capped.sh | 47 +- .../pacman/pacman_wrapper.sh | 20 +- .../utils/heavy_job_lock.sh | 232 +++++++++ .../fixes/diagnose_pacman_hook_stall.sh | 493 ++++++++++++++++++ .../fixes/migrate_hosts_guard_to_guard_lib.sh | 444 ++++++++++++++++ meta/scripts/run_with_heavy_lock.sh | 31 ++ 12 files changed, 1571 insertions(+), 109 deletions(-) create mode 100644 docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json create mode 100644 docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json create mode 100755 linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh create mode 100755 linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh create mode 100755 linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh create mode 100755 meta/scripts/run_with_heavy_lock.sh diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b2efa382..90586284 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -240,7 +240,10 @@ repos: hooks: - id: pytest-coverage name: pytest with coverage enforcement - entry: python meta/scripts/pytest_changed_packages.py + # Serialised against builds and pacman transactions on this 7.6 GB + # machine via the heavy-job lock. run_with_heavy_lock.sh is a plain + # passthrough on any clone that has not installed one. + entry: bash meta/scripts/run_with_heavy_lock.sh pytest python meta/scripts/pytest_changed_packages.py language: system types: [python] pass_filenames: true diff --git a/docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json b/docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json new file mode 100644 index 00000000..9690a3bb --- /dev/null +++ b/docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json @@ -0,0 +1,20 @@ +{ + "title": "Diagnose the pacman first-hook stall; fix wrapper drift, finish the guard-lib migration, serialise heavy jobs", + "objective": "pacman intermittently freezes at its first PreTransaction hook (5 abandoned transactions since 2026-07-29, stalls of 4 minutes to overnight). Establish what is actually failing rather than assuming, then repair the real deployment defects found along the way. Success means: the stall is either root-caused or instrumented so the next real occurrence is captured; the deployed pacman wrapper matches its source and drift can no longer hide; there is one guard system for /etc/hosts instead of two half-installed ones; and concurrent heavy jobs on this 7.6 GB machine are prevented by a mechanism rather than by a human remembering a rule.", + "acceptance_criteria": [ + "The guard-lib hook is proven not to be the cause (its target list is empty, so the script body cannot execute)", + "A repro harness drives real pacman transactions and captures /proc//stack, wchan, syscall, process tree, meminfo and PSI on a stall", + "A passive --watch mode captures the same evidence from a real (not synthetic) stall", + "Deployed pacman_wrapper matches the repo source, and check_and_enable_services.sh detects content drift in both directions (source moved on / deployed tampered) and fails closed when unverifiable", + "hosts, nsswitch and resolved are guardctl file-guard instances; the legacy pacman hooks and units are retired; /etc/hosts keeps chattr +i, its read-only bind mount, its exact contents and its blocking behaviour", + "The migration has a --rollback that restores the legacy layer INCLUDING the immutable flag, verified by running it", + "A heavy-job lock serialises pacman transactions, makepkg builds and the coverage hook, names the current holder when it makes something wait, and always fails open" + ], + "out_of_scope": [ + "Shipping a pacman-side fix for the stall itself: it could not be reproduced, so there is no verified defect to fix yet", + "The shutdown-schedule guard instance: /etc/shutdown-schedule.conf does not exist on this machine, so registering it would mean installing the midnight-shutdown feature rather than finishing this migration", + "Wiring the lock into Flutter/Gradle builds: the repo has no single build entry point to wrap; the lock is available as a CLI for those", + "Anything on the desktop machine, which is not reachable from this session" + ], + "verifier": "sudo diagnose_pacman_hook_stall.sh --runs N [--with-load]; sudo migrate_hosts_guard_to_guard_lib.sh --status/--rollback; sudo check_and_enable_services.sh --status with deliberately induced drift; timed heavy_job_lock.sh contention tests; pre-commit run --files " +} diff --git a/docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json b/docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json new file mode 100644 index 00000000..26d4c423 --- /dev/null +++ b/docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json @@ -0,0 +1,66 @@ +{ + "intent": "Investigate pacman freezing at its first PreTransaction hook, and repair the deployment defects the investigation exposed: a stale pacman wrapper that no check could see, a half-finished guard-lib migration, and unserialised heavy jobs on a 7.6 GB machine.", + "scope": [ + "linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh (new)", + "linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh (new)", + "linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh (new)", + "meta/scripts/run_with_heavy_lock.sh (new)", + "check_and_enable_services.sh, install_pacman_wrapper.sh, install_makepkg_wrapper.sh, pacman_wrapper.sh, makepkg_capped.sh, .pre-commit-config.yaml", + "NON-GOAL: a pacman-side fix for the stall - it could not be reproduced, so there is no verified defect to fix", + "NON-GOAL: the shutdown-schedule guard instance (/etc/shutdown-schedule.conf absent on this machine)" + ], + "changes": [ + "Established from /var/log/pacman.log that the stall hit 5 transactions since 2026-07-29, always at the FIRST PreTransaction hook (4/56 guard-lib runs, plus 1/142 hosts-hook runs from when THAT hook was first). What the failures share is position, not identity.", + "Proved the guard-lib hook is not the cause: /etc/guard-lib/targets is empty since install, so the loop body never executes and the script reaches exit 0 in milliseconds.", + "New diagnose_pacman_hook_stall.sh: drives real transactions (reinstalling the 0-file base-devel meta-package from cache), with a watchdog that dumps /proc//{stack,wchan,syscall,status}, the process tree, meminfo and PSI when a live pacman sits on a hook line; --with-load applies memory pressure; --watch passively captures a real stall.", + "Redeployed the pacman wrapper: 19916 B -> 27515 B. The old copy had its own inline stale-lock handler but never sourced the shared pacman_lock_lib.sh that makepkg_wrapper already used.", + "Both installers now write a drift manifest (absolute paths) covering source AND installed hashes; check_and_enable_services.sh replays it with sha256sum -c and fails closed when the manifest is missing.", + "New migrate_hosts_guard_to_guard_lib.sh: registers hosts/nsswitch/resolved as guardctl file-guard instances, installs plugins to a stable /usr/local/share path (guardctl records absolute plugin paths), retires the legacy hooks and units, and provides --rollback/--status/--dry-run.", + "New heavy_job_lock.sh: one system-wide flock shared between root and the user, serialising pacman transactions, makepkg builds and the coverage hook; names the current holder; always fails open." + ], + "verification": [ + { + "command": "sudo diagnose_pacman_hook_stall.sh --runs 40 (idle)", + "result": "pass", + "evidence": "40 transactions, 0 stalls, 3/5/8 s min/median/max" + }, + { + "command": "sudo diagnose_pacman_hook_stall.sh --runs 25 --with-load", + "result": "pass", + "evidence": "2578 MB hog, MemAvailable -> 1657 MB; 25 transactions, 0 stalls, 4/4/8 s. NOT a reproduction: 65 clean runs against a historical 7% rate is ~0.9% likely, so the trigger is conditional on something this synthetic load does not recreate. Memory pressure is neither confirmed nor refuted." + }, + { + "command": "sudo check_and_enable_services.sh --status, across five induced states", + "result": "pass", + "evidence": "clean -> no drift; repo source edited -> detected; deployed file tampered -> detected; restored -> clean; manifest removed -> 'deployed version unverifiable' (fails closed)" + }, + { + "command": "sudo migrate_hosts_guard_to_guard_lib.sh (then --rollback, then migrate again)", + "result": "pass", + "evidence": "3 instances registered; /etc/hosts byte-identical (171441 lines), chattr +i, ro bind mount intact, www.youtube.com -> 0.0.0.0, root write refused with EPERM; only ONE PreTransaction hook now and transactions dropped from ~5-6 s to ~3-4 s; rollback restores hooks, units, +i and the bind mount" + }, + { + "command": "heavy_job_lock.sh contention tests (timed)", + "result": "pass", + "evidence": "--fail-fast returns 75 naming the holder; waiting mode started job B 0.01 s after job A ended; unusable lock path runs the command anyway (fail open); with the wrapper deployed, `pacman -Q` took 0 s while a real -U transaction waited 18 s behind a held lock" + }, + { + "command": "pre-commit run --files <10 changed files>", + "result": "pass", + "evidence": "shellcheck, codespell, secrets, binary-block, polling-anti-pattern all Passed; only the evidence/contract hooks failed, which this artifact and its contract resolve" + } + ], + "risks": [ + "The stall is NOT fixed - it is only instrumented. --watch must capture a real occurrence before any pacman-side change is justified.", + "The drift manifest currently records paths inside the git worktree this work was done in. Both installers must be re-run from ~/testsAndMisc after merging, or the check will report permanent drift once the worktree is removed.", + "Retiring the legacy hosts-guard layer means /etc/hosts protection now depends entirely on guard-lib; --rollback is the escape hatch and has been exercised.", + "Two bugs were found ONLY by running --rollback: fg_uninstall clears chattr +i on its way out (legacy watchers only fire on PathModified), and a `((i > 20)) && break` left the loop body's status at 1, which under set -e silently aborted the rollback - but only when the loop actually ran, so --dry-run looked healthy. Both fixed; both had briefly left /etc/hosts unprotected.", + "A pacman transaction can now wait up to 30 minutes behind a build before failing open. Read-only queries are deliberately exempt." + ], + "rollback": [ + "guard-lib migration: sudo migrate_hosts_guard_to_guard_lib.sh --rollback, then confirm lsattr /etc/hosts shows 'i' and findmnt shows the ro bind mount", + "pacman wrapper: sudo ln -sfn /usr/bin/pacman.orig /usr/bin/pacman bypasses the wrapper entirely; pre-change copies are in the session scratchpad", + "heavy-job lock: rm /etc/tmpfiles.d/heavy-job-lock.conf /run/heavy-job.lock - every call site fails open when the lock is absent", + "After any rollback, validate: pacman -Q works, a real -U transaction completes, getent hosts www.youtube.com still returns 0.0.0.0, and /etc/hosts is still 171441 lines" + ] +} diff --git a/linux_configuration/scripts/periodic_background/check_and_enable_services.sh b/linux_configuration/scripts/periodic_background/check_and_enable_services.sh index cf15a271..a976c165 100755 --- a/linux_configuration/scripts/periodic_background/check_and_enable_services.sh +++ b/linux_configuration/scripts/periodic_background/check_and_enable_services.sh @@ -32,6 +32,9 @@ CONFIG_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")" # Script paths PACMAN_WRAPPER_INSTALL="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh" MAKEPKG_WRAPPER_INSTALL="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh" +# Drift manifests written by the two installers above (see deployment_drift). +PACMAN_WRAPPER_MANIFEST="/var/lib/pacman-wrapper/source.sha256" +MAKEPKG_WRAPPER_MANIFEST="/var/lib/pacman-wrapper/makepkg-source.sha256" MIDNIGHT_SHUTDOWN_SCRIPT="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/setup_midnight_shutdown.sh" STARTUP_MONITOR_SCRIPT="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/setup_pc_startup_monitor.sh" PERIODIC_SYSTEM_SCRIPT="$CONFIG_DIR/scripts/periodic_background/setup_periodic_system.sh" @@ -109,6 +112,28 @@ hosts_pacman_hooks_installed() { [[ -f /etc/pacman.d/hooks/90-relock-etc-hosts.hook ]]; } } +# Replay a drift manifest written at install time. The manifest holds plain +# sha256sum lines covering BOTH the repo sources an installer copied from and +# the installed copies it produced, so one `sha256sum -c` answers both "has the +# repo moved on?" and "did someone edit the deployed file?". +# +# This exists because an existence check is not a deployment check. On +# 2026-08-03 /usr/local/bin/pacman_wrapper was 19916 B against a 27510 B source +# — missing the policy integrity manifest, the pacman_lock_lib stale-lock fix +# and the guard-lib fallbacks — while every check here reported "ok" hourly, +# because the file was present and /usr/bin/pacman was a symlink. Both of those +# facts stayed true the entire time the wrapper was a week out of date. +# +# Return: 0 verified, 1 drift detected, 2 no manifest (install predates this). +# Callers must treat 2 as actionable rather than a pass: "cannot verify" is +# precisely the state that hid the stale wrapper. +deployment_drift() { # + local manifest="$1" + [[ -f $manifest ]] || return 2 + sha256sum -c --status "$manifest" 2>/dev/null || return 1 + return 0 +} + run() { if [[ $DRY_RUN -eq 1 ]]; then echo -e "${YELLOW}DRY-RUN:${NC} $*" @@ -280,6 +305,22 @@ check_pacman_wrapper() { fi done + # Content drift — the check everything above misses. A correct symlink and a + # present file say nothing about WHICH version is deployed. + local drift_rc=0 + deployment_drift "$PACMAN_WRAPPER_MANIFEST" || drift_rc=$? + case $drift_rc in + 0) msg "Wrapper matches the source it was installed from (no drift)" ;; + 1) + issues+=("Deployed pacman wrapper differs from its install manifest (stale or tampered)") + status="error" + ;; + 2) + issues+=("No drift manifest at $PACMAN_WRAPPER_MANIFEST — deployed version unverifiable") + status="error" + ;; + esac + # Report and fix if [[ $status == "error" ]]; then for issue in "${issues[@]}"; do @@ -292,8 +333,10 @@ check_pacman_wrapper() { if [[ -f $PACMAN_WRAPPER_INSTALL ]]; then run bash "$PACMAN_WRAPPER_INSTALL" ((FIXES_APPLIED++)) || true - # Re-verify after fix - if [[ $DRY_RUN -eq 0 ]] && [[ -L /usr/bin/pacman ]] && [[ -f /usr/bin/pacman.orig ]] && [[ -f /usr/local/bin/pacman_wrapper ]]; then + # Re-verify after fix. Content is part of "fixed": a reinstall + # that leaves the manifest failing has repaired nothing. + if [[ $DRY_RUN -eq 0 ]] && [[ -L /usr/bin/pacman ]] && [[ -f /usr/bin/pacman.orig ]] && + [[ -f /usr/local/bin/pacman_wrapper ]] && deployment_drift "$PACMAN_WRAPPER_MANIFEST"; then status="ok" fi else @@ -360,6 +403,21 @@ check_makepkg_wrapper() { status="error" fi + # Content drift (see check_pacman_wrapper for why existence is not enough) + local drift_rc=0 + deployment_drift "$MAKEPKG_WRAPPER_MANIFEST" || drift_rc=$? + case $drift_rc in + 0) msg "Makepkg wrapper matches the source it was installed from (no drift)" ;; + 1) + issues+=("Deployed makepkg wrapper differs from its install manifest (stale or tampered)") + status="error" + ;; + 2) + issues+=("No drift manifest at $MAKEPKG_WRAPPER_MANIFEST — deployed version unverifiable") + status="error" + ;; + esac + # Report and fix if [[ $status == "error" ]]; then for issue in "${issues[@]}"; do diff --git a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh index 1cab0821..d4a7c1da 100755 --- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh +++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh @@ -33,6 +33,7 @@ REWRAP_SOURCE="$SRC_DIR/rewrap_pkg_managers.sh" REWRAP_DEST="${INSTALL_DIR}/rewrap_pkg_managers.sh" HOOK_SOURCE="$SRC_DIR/96-restore-pkg-wrappers.hook" HOOK_DEST="/etc/pacman.d/hooks/96-restore-pkg-wrappers.hook" +SOURCE_MANIFEST="/var/lib/pacman-wrapper/makepkg-source.sha256" # Copy a file, unlocking an immutable destination first (the shared lib may be # chattr +i from install_pacman_wrapper.sh's integrity protection). @@ -83,5 +84,26 @@ deploy_file "$HOOK_SOURCE" "$HOOK_DEST" 644 echo -e "${BLUE}Creating symbolic link /usr/bin/makepkg -> ${WRAPPER_DEST}...${NC}" ln -sf "$WRAPPER_DEST" /usr/bin/makepkg +# Drift manifest — see deployment_drift() in check_and_enable_services.sh. Same +# rationale as the pacman wrapper's: "the file exists" is not "the right version +# is deployed", and only the second one is worth re-checking every hour. +mkdir -p "$(dirname "$SOURCE_MANIFEST")" +{ + sha256sum "$WRAPPER_SOURCE" "$LOCK_LIB_SOURCE" "$REWRAP_SOURCE" "$HOOK_SOURCE" || { + echo -e "${RED}Failed to checksum makepkg wrapper sources${NC}" >&2 + exit 1 + } + sha256sum "$WRAPPER_DEST" "$REWRAP_DEST" "$HOOK_DEST" || { + echo -e "${RED}Failed to checksum installed makepkg wrapper${NC}" >&2 + exit 1 + } +} >"$SOURCE_MANIFEST" + +if [[ ! -s $SOURCE_MANIFEST ]]; then + echo -e "${RED}Error: drift manifest was not created or is empty${NC}" >&2 + exit 1 +fi +chmod 644 "$SOURCE_MANIFEST" + echo -e "${GREEN}makepkg wrapper installed.${NC} Original preserved at ${CYAN}/usr/bin/makepkg.orig${NC}" echo -e "Upgrade-survival hook installed at ${CYAN}${HOOK_DEST}${NC}" diff --git a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh index 0a01b79b..170cb2f0 100755 --- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh +++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh @@ -5,9 +5,9 @@ set -euo pipefail # Auto-sudo functionality if [ "$EUID" -ne 0 ]; then - echo "Executing with sudo..." - sudo "$0" "$@" - exit $? + echo "Executing with sudo..." + sudo "$0" "$@" + exit $? fi # Colors @@ -27,6 +27,7 @@ WHITELIST_SOURCE="$(dirname "$0")/pacman_whitelist.txt" GREYLIST_SOURCE="$(dirname "$0")/pacman_greylist.txt" MAKEPKG_CAPPED_SOURCE="$(dirname "$0")/makepkg_capped.sh" MKPKG_SOURCE="$(dirname "$0")/mkpkg.sh" +HEAVY_LOCK_SOURCE="$(dirname "$0")/../../utils/heavy_job_lock.sh" INSTALL_DIR="/usr/local/bin" WRAPPER_DEST="${INSTALL_DIR}/pacman_wrapper" LOCK_LIB_DEST="${INSTALL_DIR}/pacman_lock_lib.sh" @@ -36,8 +37,10 @@ WHITELIST_DEST="${INSTALL_DIR}/pacman_whitelist.txt" GREYLIST_DEST="${INSTALL_DIR}/pacman_greylist.txt" MAKEPKG_CAPPED_DEST="${INSTALL_DIR}/makepkg_capped" MKPKG_DEST="${INSTALL_DIR}/mkpkg" +HEAVY_LOCK_DEST="${INSTALL_DIR}/heavy_job_lock.sh" INTEGRITY_DIR="/var/lib/pacman-wrapper" INTEGRITY_FILE="${INTEGRITY_DIR}/policy.sha256" +SOURCE_MANIFEST="${INTEGRITY_DIR}/source.sha256" LEECHBLOCK_INSTALLER_SOURCE="$(dirname "$0")/../install_leechblock.sh" LEECHBLOCK_DEFAULTS_SOURCE="$(dirname "$0")/../leechblock_defaults.json" LEECHBLOCK_SEEDER_SOURCE="$(dirname "$0")/../seed_leechblock_storage.js" @@ -53,63 +56,63 @@ VBOX_ENFORCE_DEST="${VBOX_INSTALL_DIR}/enforce_vbox_hosts.sh" declare -a RELock_FILES=() is_immutable_file() { - local file_path="$1" - [[ -e "$file_path" ]] || return 1 - [[ $(lsattr -d "$file_path" 2>/dev/null | awk '{print $1}') == *i* ]] + local file_path="$1" + [[ -e "$file_path" ]] || return 1 + [[ $(lsattr -d "$file_path" 2>/dev/null | awk '{print $1}') == *i* ]] } unlock_immutable_file_if_needed() { - local file_path="$1" - if ! command -v chattr >/dev/null 2>&1; then - return 0 - fi - if is_immutable_file "$file_path"; then - chattr -i "$file_path" - RELock_FILES+=("$file_path") - fi + local file_path="$1" + if ! command -v chattr >/dev/null 2>&1; then + return 0 + fi + if is_immutable_file "$file_path"; then + chattr -i "$file_path" + RELock_FILES+=("$file_path") + fi } relock_files_on_exit() { - if ! command -v chattr >/dev/null 2>&1; then - return - fi - for file_path in "${RELock_FILES[@]}"; do - [[ -e "$file_path" ]] || continue - chattr +i "$file_path" 2>/dev/null || true - done + if ! command -v chattr >/dev/null 2>&1; then + return + fi + for file_path in "${RELock_FILES[@]}"; do + [[ -e "$file_path" ]] || continue + chattr +i "$file_path" 2>/dev/null || true + done } copy_managed_file() { - local source_file="$1" - local dest_file="$2" - local required="$3" - local label="$4" + local source_file="$1" + local dest_file="$2" + local required="$3" + local label="$4" - if [[ ! -f "$source_file" ]]; then - if [[ "$required" == "required" ]]; then - echo -e "${RED}Error:${NC} Missing required ${label} at ${source_file}" >&2 - exit 1 - fi - echo -e "${YELLOW}Warning:${NC} Missing ${label} at ${source_file}" >&2 - return - fi + if [[ ! -f "$source_file" ]]; then + if [[ "$required" == "required" ]]; then + echo -e "${RED}Error:${NC} Missing required ${label} at ${source_file}" >&2 + exit 1 + fi + echo -e "${YELLOW}Warning:${NC} Missing ${label} at ${source_file}" >&2 + return + fi - unlock_immutable_file_if_needed "$dest_file" - cp "$source_file" "$dest_file" + unlock_immutable_file_if_needed "$dest_file" + cp "$source_file" "$dest_file" } trap relock_files_on_exit EXIT # Check if script is run as root if [ "$EUID" -ne 0 ]; then - echo -e "${RED}Please run as root${NC}" - exit 1 + echo -e "${RED}Please run as root${NC}" + exit 1 fi # Check if the wrapper script exists if [ ! -f "$WRAPPER_SOURCE" ]; then - echo -e "${RED}Error: Wrapper script not found at ${WRAPPER_SOURCE}${NC}" - exit 1 + echo -e "${RED}Error: Wrapper script not found at ${WRAPPER_SOURCE}${NC}" + exit 1 fi echo -e "${CYAN}Installing pacman wrapper...${NC}" @@ -128,7 +131,12 @@ copy_managed_file "$MAKEPKG_CAPPED_SOURCE" "$MAKEPKG_CAPPED_DEST" required "make chmod +x "$MAKEPKG_CAPPED_DEST" copy_managed_file "$MKPKG_SOURCE" "$MKPKG_DEST" required "mkpkg helper" chmod +x "$MKPKG_DEST" -chmod 644 "$WORDS_DEST" "$BLOCKED_DEST" "$WHITELIST_DEST" "$GREYLIST_DEST" 2> /dev/null || true +# The heavy-job lock is sourced by pacman_wrapper.sh and makepkg_capped.sh from +# a fixed absolute path, so it has to land next to them rather than be read out +# of a repo checkout that may move. +copy_managed_file "$HEAVY_LOCK_SOURCE" "$HEAVY_LOCK_DEST" required "heavy-job lock library" +chmod 755 "$HEAVY_LOCK_DEST" +chmod 644 "$WORDS_DEST" "$BLOCKED_DEST" "$WHITELIST_DEST" "$GREYLIST_DEST" 2>/dev/null || true # Automatically use symbolic link installation method echo -e "${YELLOW}Installing using symbolic link method...${NC}" @@ -138,8 +146,8 @@ echo -e "${YELLOW}Installing using symbolic link method...${NC}" # NEVER when it is already our symlink — copying the symlink's target would put # the wrapper into pacman.orig and cause an exec loop. if [ ! -L /usr/bin/pacman ]; then - echo -e "${BLUE}Backing up original pacman to /usr/bin/pacman.orig...${NC}" - cp -f /usr/bin/pacman /usr/bin/pacman.orig + echo -e "${BLUE}Backing up original pacman to /usr/bin/pacman.orig...${NC}" + cp -f /usr/bin/pacman /usr/bin/pacman.orig fi # Update the PACMAN_BIN variable in the wrapper to point to the original @@ -160,81 +168,135 @@ missing_files=() [[ ! -f "$LOCK_LIB_DEST" ]] && missing_files+=("$LOCK_LIB_DEST") if [[ ${#missing_files[@]} -gt 0 ]]; then - echo -e "${RED}Error: Critical policy files are missing:${NC}" - printf '%s\n' "${missing_files[@]}" >&2 - echo -e "${RED}Installation incomplete. Cannot create integrity file.${NC}" - exit 1 + echo -e "${RED}Error: Critical policy files are missing:${NC}" + printf '%s\n' "${missing_files[@]}" >&2 + echo -e "${RED}Installation incomplete. Cannot create integrity file.${NC}" + exit 1 fi { - sha256sum "$BLOCKED_DEST" || { echo -e "${RED}Failed to checksum blocked list${NC}" >&2; exit 1; } - sha256sum "$GREYLIST_DEST" || { echo -e "${RED}Failed to checksum greylist${NC}" >&2; exit 1; } - # The shared stale-lock library is executed (sourced) by the wrapper, so it is - # integrity-checked too: pacman_wrapper.sh sources it only AFTER - # verify_policy_integrity passes, so a tampered lib is rejected before it runs. - sha256sum "$LOCK_LIB_DEST" || { echo -e "${RED}Failed to checksum lock library${NC}" >&2; exit 1; } - # Whitelist is optional - if [[ -f "$WHITELIST_DEST" ]]; then - sha256sum "$WHITELIST_DEST" || { echo -e "${RED}Failed to checksum whitelist${NC}" >&2; exit 1; } - fi -} > "$INTEGRITY_FILE" + sha256sum "$BLOCKED_DEST" || { + echo -e "${RED}Failed to checksum blocked list${NC}" >&2 + exit 1 + } + sha256sum "$GREYLIST_DEST" || { + echo -e "${RED}Failed to checksum greylist${NC}" >&2 + exit 1 + } + # The shared stale-lock library is executed (sourced) by the wrapper, so it is + # integrity-checked too: pacman_wrapper.sh sources it only AFTER + # verify_policy_integrity passes, so a tampered lib is rejected before it runs. + sha256sum "$LOCK_LIB_DEST" || { + echo -e "${RED}Failed to checksum lock library${NC}" >&2 + exit 1 + } + # Whitelist is optional + if [[ -f "$WHITELIST_DEST" ]]; then + sha256sum "$WHITELIST_DEST" || { + echo -e "${RED}Failed to checksum whitelist${NC}" >&2 + exit 1 + } + fi +} >"$INTEGRITY_FILE" # Verify integrity file was created and has content if [[ ! -s "$INTEGRITY_FILE" ]]; then - echo -e "${RED}Error: Integrity file was not created or is empty${NC}" - exit 1 + echo -e "${RED}Error: Integrity file was not created or is empty${NC}" + exit 1 fi # Make integrity file immutable chmod 400 "$INTEGRITY_FILE" -if command -v chattr > /dev/null 2>&1; then - chattr +i "$INTEGRITY_FILE" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make integrity file immutable${NC}" +if command -v chattr >/dev/null 2>&1; then + chattr +i "$INTEGRITY_FILE" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make integrity file immutable${NC}" fi +# Record a drift manifest: the hash of every SOURCE file we installed from, and +# of the installed copies we produced. check_and_enable_services.sh replays it +# with `sha256sum -c`, which answers both questions in one command: +# - a source line fails => the repo moved on, this deployment is stale +# - a dest line fails => someone edited the installed copy directly +# Neither was detectable before: the checker only tested that files EXIST, which +# is why a wrapper 7.6 KB behind the repo (no integrity manifest, no +# pacman_lock_lib, no guard-lib fallbacks) ran unnoticed for a week while the +# hourly maintenance timer kept reporting "ok". +# Deliberately NOT chattr +i (unlike the policy integrity file above): this is a +# drift record, not a security boundary, and it must be rewritable each install. +echo -e "${BLUE}Recording deployment drift manifest...${NC}" +# ABSOLUTE paths only. The SOURCE_* vars above are built from `dirname "$0"`, so +# they are relative whenever the installer is invoked by a relative path — and +# check_and_enable_services.sh replays this manifest from systemd with cwd=/, +# where relative entries resolve to nothing. sha256sum -c would report those as +# failures, the checker would read that as drift, and it would reinstall on +# every single hourly run forever. +manifest_sources=() +for src in "$WRAPPER_SOURCE" "$LOCK_LIB_SOURCE" "$BLOCKED_SOURCE" "$GREYLIST_SOURCE" \ + "$MAKEPKG_CAPPED_SOURCE" "$MKPKG_SOURCE" "$WHITELIST_SOURCE"; do + [[ -f "$src" ]] || continue # whitelist is optional + manifest_sources+=("$(readlink -f "$src")") +done + +{ + sha256sum "${manifest_sources[@]}" || { + echo -e "${RED}Failed to checksum wrapper sources${NC}" >&2 + exit 1 + } + sha256sum "$LOCK_LIB_DEST" || { + echo -e "${RED}Failed to checksum installed lock lib${NC}" >&2 + exit 1 + } +} >"$SOURCE_MANIFEST" + +if [[ ! -s "$SOURCE_MANIFEST" ]]; then + echo -e "${RED}Error: drift manifest was not created or is empty${NC}" >&2 + exit 1 +fi +chmod 644 "$SOURCE_MANIFEST" + # Make policy files immutable to prevent easy tampering echo -e "${BLUE}Protecting policy files from modification...${NC}" -if command -v chattr > /dev/null 2>&1; then - chattr +i "$BLOCKED_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make blocked list immutable${NC}" - chattr +i "$GREYLIST_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make greylist immutable${NC}" - chattr +i "$LOCK_LIB_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make lock library immutable${NC}" - # Note: whitelist is intentionally left modifiable for user convenience +if command -v chattr >/dev/null 2>&1; then + chattr +i "$BLOCKED_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make blocked list immutable${NC}" + chattr +i "$GREYLIST_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make greylist immutable${NC}" + chattr +i "$LOCK_LIB_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make lock library immutable${NC}" + # Note: whitelist is intentionally left modifiable for user convenience else - echo -e "${YELLOW}Warning: chattr not available, policy files will not be immutable${NC}" + echo -e "${YELLOW}Warning: chattr not available, policy files will not be immutable${NC}" fi # Install LeechBlock installer and defaults if available mkdir -p "$LEECHBLOCK_INSTALL_DIR" if [ -f "$LEECHBLOCK_INSTALLER_SOURCE" ]; then - echo -e "${BLUE}Installing LeechBlock installer to ${LEECHBLOCK_INSTALLER_DEST}...${NC}" - cp "$LEECHBLOCK_INSTALLER_SOURCE" "$LEECHBLOCK_INSTALLER_DEST" - chmod +x "$LEECHBLOCK_INSTALLER_DEST" - echo -e "${GREEN}LeechBlock installer deployed to ${LEECHBLOCK_INSTALLER_DEST}${NC}" + echo -e "${BLUE}Installing LeechBlock installer to ${LEECHBLOCK_INSTALLER_DEST}...${NC}" + cp "$LEECHBLOCK_INSTALLER_SOURCE" "$LEECHBLOCK_INSTALLER_DEST" + chmod +x "$LEECHBLOCK_INSTALLER_DEST" + echo -e "${GREEN}LeechBlock installer deployed to ${LEECHBLOCK_INSTALLER_DEST}${NC}" else - echo -e "${YELLOW}LeechBlock installer not found at ${LEECHBLOCK_INSTALLER_SOURCE}, skipping...${NC}" + echo -e "${YELLOW}LeechBlock installer not found at ${LEECHBLOCK_INSTALLER_SOURCE}, skipping...${NC}" fi if [ -f "$LEECHBLOCK_DEFAULTS_SOURCE" ]; then - cp "$LEECHBLOCK_DEFAULTS_SOURCE" "$LEECHBLOCK_DEFAULTS_DEST" - echo -e "${GREEN}LeechBlock defaults deployed to ${LEECHBLOCK_DEFAULTS_DEST}${NC}" + cp "$LEECHBLOCK_DEFAULTS_SOURCE" "$LEECHBLOCK_DEFAULTS_DEST" + echo -e "${GREEN}LeechBlock defaults deployed to ${LEECHBLOCK_DEFAULTS_DEST}${NC}" fi if [ -f "$LEECHBLOCK_SEEDER_SOURCE" ]; then - cp "$LEECHBLOCK_SEEDER_SOURCE" "$LEECHBLOCK_SEEDER_DEST" - echo -e "${GREEN}LeechBlock seeder deployed to ${LEECHBLOCK_SEEDER_DEST}${NC}" + cp "$LEECHBLOCK_SEEDER_SOURCE" "$LEECHBLOCK_SEEDER_DEST" + echo -e "${GREEN}LeechBlock seeder deployed to ${LEECHBLOCK_SEEDER_DEST}${NC}" fi if [ -f "$LEECHBLOCK_PKG_SOURCE" ]; then - cp "$LEECHBLOCK_PKG_SOURCE" "${LEECHBLOCK_INSTALL_DIR}/package.json" - echo -e "${BLUE}Installing Node.js deps in ${LEECHBLOCK_INSTALL_DIR}...${NC}" - npm install --prefix "$LEECHBLOCK_INSTALL_DIR" 2>&1 | grep -v '^npm warn' || true + cp "$LEECHBLOCK_PKG_SOURCE" "${LEECHBLOCK_INSTALL_DIR}/package.json" + echo -e "${BLUE}Installing Node.js deps in ${LEECHBLOCK_INSTALL_DIR}...${NC}" + npm install --prefix "$LEECHBLOCK_INSTALL_DIR" 2>&1 | grep -v '^npm warn' || true fi # Install VirtualBox enforcement script if available if [ -f "$VBOX_ENFORCE_SOURCE" ]; then - echo -e "${BLUE}Installing VirtualBox hosts enforcement script...${NC}" - mkdir -p "$VBOX_INSTALL_DIR" - cp "$VBOX_ENFORCE_SOURCE" "$VBOX_ENFORCE_DEST" - chmod +x "$VBOX_ENFORCE_DEST" - echo -e "${GREEN}VirtualBox enforcement script installed to ${VBOX_ENFORCE_DEST}${NC}" + echo -e "${BLUE}Installing VirtualBox hosts enforcement script...${NC}" + mkdir -p "$VBOX_INSTALL_DIR" + cp "$VBOX_ENFORCE_SOURCE" "$VBOX_ENFORCE_DEST" + chmod +x "$VBOX_ENFORCE_DEST" + echo -e "${GREEN}VirtualBox enforcement script installed to ${VBOX_ENFORCE_DEST}${NC}" else - echo -e "${YELLOW}VirtualBox enforcement script not found, skipping...${NC}" + echo -e "${YELLOW}VirtualBox enforcement script not found, skipping...${NC}" fi # Create symbolic link @@ -243,12 +305,12 @@ ln -sf "$WRAPPER_DEST" /usr/bin/pacman echo -e "${GREEN}Installation complete!${NC}" echo -e "Pacman is now wrapped. The original pacman is available at ${CYAN}/usr/bin/pacman.orig${NC}" if [ -f "$MAKEPKG_CAPPED_DEST" ]; then - echo -e "Run constrained package builds with: ${CYAN}pacman --makepkg-capped ${NC}" + echo -e "Run constrained package builds with: ${CYAN}pacman --makepkg-capped ${NC}" fi if [ -f "$MKPKG_DEST" ]; then - echo -e "Shortcut available: ${CYAN}mkpkg ${NC}" + echo -e "Shortcut available: ${CYAN}mkpkg ${NC}" fi echo -e "${CYAN}Policy files are now protected with immutable attributes.${NC}" if [ -f "$VBOX_ENFORCE_DEST" ]; then - echo -e "${CYAN}VirtualBox VMs will automatically be configured to use host's /etc/hosts.${NC}" + echo -e "${CYAN}VirtualBox VMs will automatically be configured to use host's /etc/hosts.${NC}" fi diff --git a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/makepkg_capped.sh b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/makepkg_capped.sh index 86d440bc..33430462 100755 --- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/makepkg_capped.sh +++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/makepkg_capped.sh @@ -11,33 +11,48 @@ IONICE_CLASS="${MAKEPKG_IONICE_CLASS:-2}" IONICE_LEVEL="${MAKEPKG_IONICE_LEVEL:-7}" if ! command -v makepkg >/dev/null 2>&1; then - echo "makepkg_capped: makepkg not found in PATH" >&2 - exit 127 + echo "makepkg_capped: makepkg not found in PATH" >&2 + exit 127 +fi + +# Serialise against other heavy jobs (Gradle/Flutter builds, coverage runs, +# pacman transactions) on this 7.6 GB machine. Re-exec ourselves under the lock +# instead of wrapping each branch below, so all three fallback paths are covered +# by one guard. HEAVY_JOB_LOCK_HELD keeps the re-exec from recursing. +# Note the cgroup caps below are NOT a substitute: MemoryMax defaults to 12G on +# a machine with 7.6G, so they bound one build's shape, not the machine's total. +HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh" +if [[ -r $HEAVY_JOB_LOCK_LIB && -z ${HEAVY_JOB_LOCK_HELD:-} ]]; then + # shellcheck source=/dev/null + source "$HEAVY_JOB_LOCK_LIB" + export HEAVY_JOB_LOCK_HELD=1 + with_heavy_lock makepkg -- "$0" "$@" + exit $? fi run_makepkg_fallback() { - exec ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \ - nice -n "$NICE_LEVEL" \ - makepkg "$@" + exec ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \ + nice -n "$NICE_LEVEL" \ + makepkg "$@" } if ! command -v systemd-run >/dev/null 2>&1; then - echo "makepkg_capped: systemd-run unavailable, falling back to local limits" >&2 - run_makepkg_fallback "$@" + echo "makepkg_capped: systemd-run unavailable, falling back to local limits" >&2 + run_makepkg_fallback "$@" fi # Keep builds in the user manager scope. If this fails (no user manager), # gracefully fall back to ionice+nice only. if ! systemd-run --user --scope --quiet true 2>/dev/null; then - echo "makepkg_capped: user systemd scope unavailable, falling back" >&2 - run_makepkg_fallback "$@" + echo "makepkg_capped: user systemd scope unavailable, falling back" >&2 + run_makepkg_fallback "$@" fi exec systemd-run --user --scope --quiet --same-dir --collect \ - -p "CPUQuota=${CPU_QUOTA}" \ - -p "MemoryMax=${MEMORY_MAX}" \ - -p "MemorySwapMax=0" \ - -p "TasksMax=${TASKS_MAX}" \ - ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \ - nice -n "$NICE_LEVEL" \ - makepkg "$@" + -p "CPUQuota=${CPU_QUOTA}" \ + -p "MemoryMax=${MEMORY_MAX}" \ + -p "MemorySwapMax=0" \ + -p "TasksMax=${TASKS_MAX}" \ + ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \ + nice -n "$NICE_LEVEL" \ + makepkg "$@" diff --git a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/pacman_wrapper.sh b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/pacman_wrapper.sh index 75deeeae..46f6e0c2 100755 --- a/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/pacman_wrapper.sh +++ b/linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/pacman_wrapper.sh @@ -12,6 +12,10 @@ BOLD='\033[1m' NC='\033[0m' # No Color PACMAN_BIN="/usr/bin/pacman" +# Installed alongside the wrapper by install_pacman_wrapper.sh. Absent on a +# machine that has not run it yet, in which case transactions simply are not +# serialised (see the guarded source below). +HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh" MAKEPKG_CAPPED_BIN="/usr/local/bin/makepkg_capped" declare -a BLOCKED_KEYWORDS_LIST=() @@ -87,7 +91,7 @@ load_policy_lists() { continue fi target_array+=("${line,,}") - done < "$file_path" + done <"$file_path" } if [[ -f $blocked_file ]]; then @@ -741,7 +745,19 @@ if should_use_wrapper_guard_lib_fallback "$@"; then manual_guard_lib_fallback=1 fi -"$PACMAN_BIN" "$@" +# Serialise real TRANSACTIONS against builds and coverage runs on this 7.6 GB +# box (see utils/heavy_job_lock.sh). Deliberately scoped by needs_unlock, i.e. +# -S/-U/-R only: read-only queries like `pacman -Q` take no db lock and run +# constantly from background services here, so making those wait behind a +# 20-minute Gradle build would be worse than the contention it avoids. +# with_heavy_lock always fails open, so this can never brick package management. +if [[ -r $HEAVY_JOB_LOCK_LIB ]] && needs_unlock "$@"; then + # shellcheck source=/dev/null + source "$HEAVY_JOB_LOCK_LIB" + with_heavy_lock pacman -- "$PACMAN_BIN" "$@" +else + "$PACMAN_BIN" "$@" +fi exit_code=$? if [[ $manual_guard_lib_fallback -eq 1 ]]; then diff --git a/linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh b/linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh new file mode 100755 index 00000000..1c98fc2c --- /dev/null +++ b/linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh @@ -0,0 +1,232 @@ +#!/bin/bash + +# ============================================================================ +# heavy_job_lock.sh - one heavy job at a time on this machine +# +# This box has 7.6 GB. On 2026-08-03 a Gradle/Flutter APK build, a full +# `pytest --cov` run and Chromium overlapped; an earlier overlap the same week +# got a whole session OOM-killed (exit 137/143). The rule "one heavy job at a +# time" was written down in ~/.claude/memories/mistakes.md - but a rule that +# only works when a human remembers it is deployment hygiene, not a fix. This +# makes the overlap impossible instead. +# +# Usage as a CLI: +# heavy_job_lock.sh --name gradle -- flutter build apk --debug +# heavy_job_lock.sh --name pytest --fail-fast -- pytest --cov +# +# Usage as a library: +# source /path/to/heavy_job_lock.sh +# with_heavy_lock gradle -- ./gradlew assembleDebug +# +# Setup (once, as root - /run is tmpfs so the lock needs recreating on boot): +# sudo heavy_job_lock.sh --install +# +# Design notes: +# - ONE system-wide lock shared between root and the user. pacman runs as +# root while builds run as the user, so a per-user /run/user/$UID lock +# would give them separate locks and no mutual exclusion at all. +# - The lock file is mode 0666 (created by root via tmpfiles.d) precisely so +# an unprivileged build can take the same lock a root pacman takes. +# - FAIL OPEN, always. A lock this central must never be able to brick +# package management or builds: if the lock file cannot be created, or the +# wait times out, we warn and run the command anyway. Blocking work forever +# is a worse failure than the memory contention we are avoiding. +# ============================================================================ + +HEAVY_JOB_LOCK_FILE="${HEAVY_JOB_LOCK_FILE:-/run/heavy-job.lock}" +# Long enough for a real Gradle build to finish ahead of us, short enough that +# a wedged holder cannot stall a machine indefinitely. +HEAVY_JOB_LOCK_TIMEOUT="${HEAVY_JOB_LOCK_TIMEOUT:-1800}" +HEAVY_JOB_TMPFILES_CONF="/etc/tmpfiles.d/heavy-job-lock.conf" + +_hjl_warn() { printf '[heavy-job-lock] %s\n' "$*" >&2; } + +# Who holds it right now? Best-effort: the holder writes " " +# after acquiring, so a refusal can name the culprit instead of just failing. +_hjl_holder() { + local holder="" + [[ -r $HEAVY_JOB_LOCK_FILE ]] && read -r holder <"$HEAVY_JOB_LOCK_FILE" 2>/dev/null + printf '%s' "${holder:-unknown}" +} + +# Create the lock file world-writable. Only meaningful as root; unprivileged +# callers just discover they cannot and fail open. +_hjl_ensure_lock_file() { + [[ -e $HEAVY_JOB_LOCK_FILE ]] && return 0 + if [[ $EUID -eq 0 ]]; then + install -m 0666 /dev/null "$HEAVY_JOB_LOCK_FILE" 2>/dev/null && return 0 + else + # Braces around the redirect: a failing redirection is reported by the + # shell itself, before the command runs, so `cmd 2>/dev/null` does not + # suppress it. The group does. + { : >"$HEAVY_JOB_LOCK_FILE"; } 2>/dev/null && + chmod 0666 "$HEAVY_JOB_LOCK_FILE" 2>/dev/null && return 0 + fi + return 1 +} + +# with_heavy_lock [--fail-fast] -- +with_heavy_lock() { + local name="$1" + shift + local fail_fast=0 + + while [[ $# -gt 0 ]]; do + case "$1" in + --fail-fast) + fail_fast=1 + shift + ;; + --) + shift + break + ;; + *) break ;; + esac + done + + if [[ $# -eq 0 ]]; then + _hjl_warn "with_heavy_lock: no command given" + return 2 + fi + + if ! _hjl_ensure_lock_file; then + _hjl_warn "cannot create $HEAVY_JOB_LOCK_FILE - running '$name' UNSERIALISED" + _hjl_warn "run 'sudo $0 --install' once to enable serialisation" + "$@" + return $? + fi + + local lock_fd + # `<>` (read-write), NOT `>`. Redirecting with `>` truncates the file at + # open time, so a second caller wiped the holder's " " + # line before it could read it — every refusal said "held by unknown". + exec {lock_fd}<>"$HEAVY_JOB_LOCK_FILE" || { + _hjl_warn "cannot open lock - running '$name' unserialised" + "$@" + return $? + } + + local acquired=0 + if ((fail_fast == 1)); then + if flock --nonblock "$lock_fd"; then + acquired=1 + else + _hjl_warn "refusing to start '$name': held by $(_hjl_holder)" + exec {lock_fd}>&- + return 75 # EX_TEMPFAIL + fi + else + local holder + holder="$(_hjl_holder)" + if ! flock --nonblock "$lock_fd"; then + _hjl_warn "waiting for '$holder' to finish before starting '$name' (timeout ${HEAVY_JOB_LOCK_TIMEOUT}s)" + fi + if flock --timeout "$HEAVY_JOB_LOCK_TIMEOUT" "$lock_fd"; then + acquired=1 + else + # Fail open: see the header. A stuck holder must not permanently + # block builds or package management. + _hjl_warn "timed out after ${HEAVY_JOB_LOCK_TIMEOUT}s waiting for $(_hjl_holder)" + _hjl_warn "proceeding UNSERIALISED with '$name'" + fi + fi + + if ((acquired == 1)); then + printf '%s %s %s\n' "$name" "$$" "$(date -Is 2>/dev/null || echo unknown)" \ + >"$HEAVY_JOB_LOCK_FILE" 2>/dev/null || true + fi + + local rc=0 + "$@" || rc=$? + + # Closing the fd releases the lock; do it explicitly so the holder line is + # not left implying we still hold it. + exec {lock_fd}>&- + return "$rc" +} + +# ---------------------------------------------------------------------------- +# CLI (skipped when this file is sourced as a library) +# ---------------------------------------------------------------------------- +_hjl_install() { + if [[ $EUID -ne 0 ]]; then + _hjl_warn "--install must run as root" + return 1 + fi + # /run is tmpfs: without a tmpfiles.d entry the lock vanishes every boot and + # the first unprivileged caller silently falls back to unserialised. + printf 'f %s 0666 root root -\n' "$HEAVY_JOB_LOCK_FILE" >"$HEAVY_JOB_TMPFILES_CONF" + systemd-tmpfiles --create "$HEAVY_JOB_TMPFILES_CONF" 2>/dev/null || true + _hjl_ensure_lock_file || { + _hjl_warn "failed to create $HEAVY_JOB_LOCK_FILE" + return 1 + } + printf 'installed %s and %s\n' "$HEAVY_JOB_TMPFILES_CONF" "$HEAVY_JOB_LOCK_FILE" +} + +_hjl_usage() { + cat < [--fail-fast] -- + $(basename "$0") --install # once, as root + $(basename "$0") --status + +Serialises heavy jobs (builds, coverage runs, pacman transactions) so only one +runs at a time on this memory-constrained machine. Always fails open. + +Options: + --name