mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 19:27:03 +01:00
Diagnose the pacman first-hook stall; fix the drift that hid a stale wrapper
pacman intermittently freezes at its FIRST PreTransaction hook - 5 abandoned
transactions since 2026-07-29, stalls of 4 minutes to overnight, always fine on
retry. This investigates it and repairs the real defects found on the way.
The stall is NOT fixed, and deliberately so: no verified defect was found, so
nothing pacman-side was changed.
- Not guard-lib. /etc/guard-lib/targets was empty, so that hook script
provably exits in milliseconds. The failures share POSITION (first hook of
the transaction), not identity: one stall hit the hosts hook back when that
one ran first.
- Not reproducible on demand. 65 real transactions (40 idle, 25 under 2.6 GB
of memory pressure) produced zero stalls. Against the historical 4-in-56
rate that is ~0.9% likely, so the trigger is conditional on something a
synthetic load does not recreate. Memory pressure is untested at the load
that actually occurred - not refuted.
So diagnose_pacman_hook_stall.sh drives real transactions with a watchdog, and
its --watch mode passively captures /proc/<pid>/{stack,wchan,syscall}, the
process tree, meminfo and PSI from the next REAL stall. No pacman-side fix
should ship before that dump exists.
Three real defects were found and fixed:
1. The deployed pacman wrapper was 19916 B against a 27510 B source, never
sourcing the shared pacman_lock_lib.sh that makepkg_wrapper already used.
It hid because check_pacman_wrapper() only tested that files EXIST. Both
installers now write a drift manifest covering source AND installed hashes;
the checker replays it with sha256sum -c and fails closed when it cannot
verify. Manifest paths are absolute - relative ones resolve to nothing under
systemd's cwd=/ and would have looped reinstalling forever.
2. The guard-lib migration (66c4698) was never finished here: guard-lib was
installed with zero registered targets while the legacy hosts-guard layer did
all the work, so every transaction ran two PreTransaction hooks and the first
one did nothing. migrate_hosts_guard_to_guard_lib.sh registers hosts,
nsswitch and resolved, retires the legacy hooks and units, and has a
--rollback that was actually exercised. Plugins install to a stable path
because guardctl records absolute plugin paths. Side effect: one hook pair
instead of two, transactions ~5-6s -> ~2-3s.
3. "One heavy job at a time" was a rule a human had to remember. heavy_job_lock.sh
makes the overlap impossible: one system-wide flock shared between root and
the user (a per-user lock would give root's pacman and the user's builds
separate locks and no exclusion at all). Wired into pacman TRANSACTIONS only
- read-only queries stay instant - plus makepkg_capped and the coverage hook.
Always fails open.
Two bugs surfaced only by running --rollback, both of which briefly left
/etc/hosts with no immutable flag: fg_uninstall clears +i on its way out while
legacy watchers only fire on PathModified, and `((i > 20)) && break` left the
loop body's status at 1, which under set -e aborted the rollback silently - but
only when the loop actually ran, so --dry-run looked healthy.
Not migrated: the shutdown-schedule guard (/etc/shutdown-schedule.conf does not
exist here). Not wired: Flutter/Gradle builds, which have no single entry point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
4d9f486176
commit
79829ec8a2
@ -240,7 +240,10 @@ repos:
|
||||
hooks:
|
||||
- id: pytest-coverage
|
||||
name: pytest with coverage enforcement
|
||||
entry: python meta/scripts/pytest_changed_packages.py
|
||||
# Serialised against builds and pacman transactions on this 7.6 GB
|
||||
# machine via the heavy-job lock. run_with_heavy_lock.sh is a plain
|
||||
# passthrough on any clone that has not installed one.
|
||||
entry: bash meta/scripts/run_with_heavy_lock.sh pytest python meta/scripts/pytest_changed_packages.py
|
||||
language: system
|
||||
types: [python]
|
||||
pass_filenames: true
|
||||
|
||||
20
docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json
Normal file
20
docs/superpowers/contracts/pacman-hook-stall-2026-08-03.json
Normal file
@ -0,0 +1,20 @@
|
||||
{
|
||||
"title": "Diagnose the pacman first-hook stall; fix wrapper drift, finish the guard-lib migration, serialise heavy jobs",
|
||||
"objective": "pacman intermittently freezes at its first PreTransaction hook (5 abandoned transactions since 2026-07-29, stalls of 4 minutes to overnight). Establish what is actually failing rather than assuming, then repair the real deployment defects found along the way. Success means: the stall is either root-caused or instrumented so the next real occurrence is captured; the deployed pacman wrapper matches its source and drift can no longer hide; there is one guard system for /etc/hosts instead of two half-installed ones; and concurrent heavy jobs on this 7.6 GB machine are prevented by a mechanism rather than by a human remembering a rule.",
|
||||
"acceptance_criteria": [
|
||||
"The guard-lib hook is proven not to be the cause (its target list is empty, so the script body cannot execute)",
|
||||
"A repro harness drives real pacman transactions and captures /proc/<pid>/stack, wchan, syscall, process tree, meminfo and PSI on a stall",
|
||||
"A passive --watch mode captures the same evidence from a real (not synthetic) stall",
|
||||
"Deployed pacman_wrapper matches the repo source, and check_and_enable_services.sh detects content drift in both directions (source moved on / deployed tampered) and fails closed when unverifiable",
|
||||
"hosts, nsswitch and resolved are guardctl file-guard instances; the legacy pacman hooks and units are retired; /etc/hosts keeps chattr +i, its read-only bind mount, its exact contents and its blocking behaviour",
|
||||
"The migration has a --rollback that restores the legacy layer INCLUDING the immutable flag, verified by running it",
|
||||
"A heavy-job lock serialises pacman transactions, makepkg builds and the coverage hook, names the current holder when it makes something wait, and always fails open"
|
||||
],
|
||||
"out_of_scope": [
|
||||
"Shipping a pacman-side fix for the stall itself: it could not be reproduced, so there is no verified defect to fix yet",
|
||||
"The shutdown-schedule guard instance: /etc/shutdown-schedule.conf does not exist on this machine, so registering it would mean installing the midnight-shutdown feature rather than finishing this migration",
|
||||
"Wiring the lock into Flutter/Gradle builds: the repo has no single build entry point to wrap; the lock is available as a CLI for those",
|
||||
"Anything on the desktop machine, which is not reachable from this session"
|
||||
],
|
||||
"verifier": "sudo diagnose_pacman_hook_stall.sh --runs N [--with-load]; sudo migrate_hosts_guard_to_guard_lib.sh --status/--rollback; sudo check_and_enable_services.sh --status with deliberately induced drift; timed heavy_job_lock.sh contention tests; pre-commit run --files <changed>"
|
||||
}
|
||||
66
docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json
Normal file
66
docs/superpowers/evidence/pacman-hook-stall-2026-08-03.json
Normal file
@ -0,0 +1,66 @@
|
||||
{
|
||||
"intent": "Investigate pacman freezing at its first PreTransaction hook, and repair the deployment defects the investigation exposed: a stale pacman wrapper that no check could see, a half-finished guard-lib migration, and unserialised heavy jobs on a 7.6 GB machine.",
|
||||
"scope": [
|
||||
"linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh (new)",
|
||||
"linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh (new)",
|
||||
"linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh (new)",
|
||||
"meta/scripts/run_with_heavy_lock.sh (new)",
|
||||
"check_and_enable_services.sh, install_pacman_wrapper.sh, install_makepkg_wrapper.sh, pacman_wrapper.sh, makepkg_capped.sh, .pre-commit-config.yaml",
|
||||
"NON-GOAL: a pacman-side fix for the stall - it could not be reproduced, so there is no verified defect to fix",
|
||||
"NON-GOAL: the shutdown-schedule guard instance (/etc/shutdown-schedule.conf absent on this machine)"
|
||||
],
|
||||
"changes": [
|
||||
"Established from /var/log/pacman.log that the stall hit 5 transactions since 2026-07-29, always at the FIRST PreTransaction hook (4/56 guard-lib runs, plus 1/142 hosts-hook runs from when THAT hook was first). What the failures share is position, not identity.",
|
||||
"Proved the guard-lib hook is not the cause: /etc/guard-lib/targets is empty since install, so the loop body never executes and the script reaches exit 0 in milliseconds.",
|
||||
"New diagnose_pacman_hook_stall.sh: drives real transactions (reinstalling the 0-file base-devel meta-package from cache), with a watchdog that dumps /proc/<pid>/{stack,wchan,syscall,status}, the process tree, meminfo and PSI when a live pacman sits on a hook line; --with-load applies memory pressure; --watch passively captures a real stall.",
|
||||
"Redeployed the pacman wrapper: 19916 B -> 27515 B. The old copy had its own inline stale-lock handler but never sourced the shared pacman_lock_lib.sh that makepkg_wrapper already used.",
|
||||
"Both installers now write a drift manifest (absolute paths) covering source AND installed hashes; check_and_enable_services.sh replays it with sha256sum -c and fails closed when the manifest is missing.",
|
||||
"New migrate_hosts_guard_to_guard_lib.sh: registers hosts/nsswitch/resolved as guardctl file-guard instances, installs plugins to a stable /usr/local/share path (guardctl records absolute plugin paths), retires the legacy hooks and units, and provides --rollback/--status/--dry-run.",
|
||||
"New heavy_job_lock.sh: one system-wide flock shared between root and the user, serialising pacman transactions, makepkg builds and the coverage hook; names the current holder; always fails open."
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"command": "sudo diagnose_pacman_hook_stall.sh --runs 40 (idle)",
|
||||
"result": "pass",
|
||||
"evidence": "40 transactions, 0 stalls, 3/5/8 s min/median/max"
|
||||
},
|
||||
{
|
||||
"command": "sudo diagnose_pacman_hook_stall.sh --runs 25 --with-load",
|
||||
"result": "pass",
|
||||
"evidence": "2578 MB hog, MemAvailable -> 1657 MB; 25 transactions, 0 stalls, 4/4/8 s. NOT a reproduction: 65 clean runs against a historical 7% rate is ~0.9% likely, so the trigger is conditional on something this synthetic load does not recreate. Memory pressure is neither confirmed nor refuted."
|
||||
},
|
||||
{
|
||||
"command": "sudo check_and_enable_services.sh --status, across five induced states",
|
||||
"result": "pass",
|
||||
"evidence": "clean -> no drift; repo source edited -> detected; deployed file tampered -> detected; restored -> clean; manifest removed -> 'deployed version unverifiable' (fails closed)"
|
||||
},
|
||||
{
|
||||
"command": "sudo migrate_hosts_guard_to_guard_lib.sh (then --rollback, then migrate again)",
|
||||
"result": "pass",
|
||||
"evidence": "3 instances registered; /etc/hosts byte-identical (171441 lines), chattr +i, ro bind mount intact, www.youtube.com -> 0.0.0.0, root write refused with EPERM; only ONE PreTransaction hook now and transactions dropped from ~5-6 s to ~3-4 s; rollback restores hooks, units, +i and the bind mount"
|
||||
},
|
||||
{
|
||||
"command": "heavy_job_lock.sh contention tests (timed)",
|
||||
"result": "pass",
|
||||
"evidence": "--fail-fast returns 75 naming the holder; waiting mode started job B 0.01 s after job A ended; unusable lock path runs the command anyway (fail open); with the wrapper deployed, `pacman -Q` took 0 s while a real -U transaction waited 18 s behind a held lock"
|
||||
},
|
||||
{
|
||||
"command": "pre-commit run --files <10 changed files>",
|
||||
"result": "pass",
|
||||
"evidence": "shellcheck, codespell, secrets, binary-block, polling-anti-pattern all Passed; only the evidence/contract hooks failed, which this artifact and its contract resolve"
|
||||
}
|
||||
],
|
||||
"risks": [
|
||||
"The stall is NOT fixed - it is only instrumented. --watch must capture a real occurrence before any pacman-side change is justified.",
|
||||
"The drift manifest currently records paths inside the git worktree this work was done in. Both installers must be re-run from ~/testsAndMisc after merging, or the check will report permanent drift once the worktree is removed.",
|
||||
"Retiring the legacy hosts-guard layer means /etc/hosts protection now depends entirely on guard-lib; --rollback is the escape hatch and has been exercised.",
|
||||
"Two bugs were found ONLY by running --rollback: fg_uninstall clears chattr +i on its way out (legacy watchers only fire on PathModified), and a `((i > 20)) && break` left the loop body's status at 1, which under set -e silently aborted the rollback - but only when the loop actually ran, so --dry-run looked healthy. Both fixed; both had briefly left /etc/hosts unprotected.",
|
||||
"A pacman transaction can now wait up to 30 minutes behind a build before failing open. Read-only queries are deliberately exempt."
|
||||
],
|
||||
"rollback": [
|
||||
"guard-lib migration: sudo migrate_hosts_guard_to_guard_lib.sh --rollback, then confirm lsattr /etc/hosts shows 'i' and findmnt shows the ro bind mount",
|
||||
"pacman wrapper: sudo ln -sfn /usr/bin/pacman.orig /usr/bin/pacman bypasses the wrapper entirely; pre-change copies are in the session scratchpad",
|
||||
"heavy-job lock: rm /etc/tmpfiles.d/heavy-job-lock.conf /run/heavy-job.lock - every call site fails open when the lock is absent",
|
||||
"After any rollback, validate: pacman -Q works, a real -U transaction completes, getent hosts www.youtube.com still returns 0.0.0.0, and /etc/hosts is still 171441 lines"
|
||||
]
|
||||
}
|
||||
@ -32,6 +32,9 @@ CONFIG_DIR="$(dirname "$(dirname "$SCRIPT_DIR")")"
|
||||
# Script paths
|
||||
PACMAN_WRAPPER_INSTALL="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/pacman/install_pacman_wrapper.sh"
|
||||
MAKEPKG_WRAPPER_INSTALL="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/pacman/install_makepkg_wrapper.sh"
|
||||
# Drift manifests written by the two installers above (see deployment_drift).
|
||||
PACMAN_WRAPPER_MANIFEST="/var/lib/pacman-wrapper/source.sha256"
|
||||
MAKEPKG_WRAPPER_MANIFEST="/var/lib/pacman-wrapper/makepkg-source.sha256"
|
||||
MIDNIGHT_SHUTDOWN_SCRIPT="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/setup_midnight_shutdown.sh"
|
||||
STARTUP_MONITOR_SCRIPT="$CONFIG_DIR/scripts/periodic_background/digital_wellbeing/setup_pc_startup_monitor.sh"
|
||||
PERIODIC_SYSTEM_SCRIPT="$CONFIG_DIR/scripts/periodic_background/setup_periodic_system.sh"
|
||||
@ -109,6 +112,28 @@ hosts_pacman_hooks_installed() {
|
||||
[[ -f /etc/pacman.d/hooks/90-relock-etc-hosts.hook ]]; }
|
||||
}
|
||||
|
||||
# Replay a drift manifest written at install time. The manifest holds plain
|
||||
# sha256sum lines covering BOTH the repo sources an installer copied from and
|
||||
# the installed copies it produced, so one `sha256sum -c` answers both "has the
|
||||
# repo moved on?" and "did someone edit the deployed file?".
|
||||
#
|
||||
# This exists because an existence check is not a deployment check. On
|
||||
# 2026-08-03 /usr/local/bin/pacman_wrapper was 19916 B against a 27510 B source
|
||||
# — missing the policy integrity manifest, the pacman_lock_lib stale-lock fix
|
||||
# and the guard-lib fallbacks — while every check here reported "ok" hourly,
|
||||
# because the file was present and /usr/bin/pacman was a symlink. Both of those
|
||||
# facts stayed true the entire time the wrapper was a week out of date.
|
||||
#
|
||||
# Return: 0 verified, 1 drift detected, 2 no manifest (install predates this).
|
||||
# Callers must treat 2 as actionable rather than a pass: "cannot verify" is
|
||||
# precisely the state that hid the stale wrapper.
|
||||
deployment_drift() { # <manifest-path>
|
||||
local manifest="$1"
|
||||
[[ -f $manifest ]] || return 2
|
||||
sha256sum -c --status "$manifest" 2>/dev/null || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
run() {
|
||||
if [[ $DRY_RUN -eq 1 ]]; then
|
||||
echo -e "${YELLOW}DRY-RUN:${NC} $*"
|
||||
@ -280,6 +305,22 @@ check_pacman_wrapper() {
|
||||
fi
|
||||
done
|
||||
|
||||
# Content drift — the check everything above misses. A correct symlink and a
|
||||
# present file say nothing about WHICH version is deployed.
|
||||
local drift_rc=0
|
||||
deployment_drift "$PACMAN_WRAPPER_MANIFEST" || drift_rc=$?
|
||||
case $drift_rc in
|
||||
0) msg "Wrapper matches the source it was installed from (no drift)" ;;
|
||||
1)
|
||||
issues+=("Deployed pacman wrapper differs from its install manifest (stale or tampered)")
|
||||
status="error"
|
||||
;;
|
||||
2)
|
||||
issues+=("No drift manifest at $PACMAN_WRAPPER_MANIFEST — deployed version unverifiable")
|
||||
status="error"
|
||||
;;
|
||||
esac
|
||||
|
||||
# Report and fix
|
||||
if [[ $status == "error" ]]; then
|
||||
for issue in "${issues[@]}"; do
|
||||
@ -292,8 +333,10 @@ check_pacman_wrapper() {
|
||||
if [[ -f $PACMAN_WRAPPER_INSTALL ]]; then
|
||||
run bash "$PACMAN_WRAPPER_INSTALL"
|
||||
((FIXES_APPLIED++)) || true
|
||||
# Re-verify after fix
|
||||
if [[ $DRY_RUN -eq 0 ]] && [[ -L /usr/bin/pacman ]] && [[ -f /usr/bin/pacman.orig ]] && [[ -f /usr/local/bin/pacman_wrapper ]]; then
|
||||
# Re-verify after fix. Content is part of "fixed": a reinstall
|
||||
# that leaves the manifest failing has repaired nothing.
|
||||
if [[ $DRY_RUN -eq 0 ]] && [[ -L /usr/bin/pacman ]] && [[ -f /usr/bin/pacman.orig ]] &&
|
||||
[[ -f /usr/local/bin/pacman_wrapper ]] && deployment_drift "$PACMAN_WRAPPER_MANIFEST"; then
|
||||
status="ok"
|
||||
fi
|
||||
else
|
||||
@ -360,6 +403,21 @@ check_makepkg_wrapper() {
|
||||
status="error"
|
||||
fi
|
||||
|
||||
# Content drift (see check_pacman_wrapper for why existence is not enough)
|
||||
local drift_rc=0
|
||||
deployment_drift "$MAKEPKG_WRAPPER_MANIFEST" || drift_rc=$?
|
||||
case $drift_rc in
|
||||
0) msg "Makepkg wrapper matches the source it was installed from (no drift)" ;;
|
||||
1)
|
||||
issues+=("Deployed makepkg wrapper differs from its install manifest (stale or tampered)")
|
||||
status="error"
|
||||
;;
|
||||
2)
|
||||
issues+=("No drift manifest at $MAKEPKG_WRAPPER_MANIFEST — deployed version unverifiable")
|
||||
status="error"
|
||||
;;
|
||||
esac
|
||||
|
||||
# Report and fix
|
||||
if [[ $status == "error" ]]; then
|
||||
for issue in "${issues[@]}"; do
|
||||
|
||||
@ -33,6 +33,7 @@ REWRAP_SOURCE="$SRC_DIR/rewrap_pkg_managers.sh"
|
||||
REWRAP_DEST="${INSTALL_DIR}/rewrap_pkg_managers.sh"
|
||||
HOOK_SOURCE="$SRC_DIR/96-restore-pkg-wrappers.hook"
|
||||
HOOK_DEST="/etc/pacman.d/hooks/96-restore-pkg-wrappers.hook"
|
||||
SOURCE_MANIFEST="/var/lib/pacman-wrapper/makepkg-source.sha256"
|
||||
|
||||
# Copy a file, unlocking an immutable destination first (the shared lib may be
|
||||
# chattr +i from install_pacman_wrapper.sh's integrity protection).
|
||||
@ -83,5 +84,26 @@ deploy_file "$HOOK_SOURCE" "$HOOK_DEST" 644
|
||||
echo -e "${BLUE}Creating symbolic link /usr/bin/makepkg -> ${WRAPPER_DEST}...${NC}"
|
||||
ln -sf "$WRAPPER_DEST" /usr/bin/makepkg
|
||||
|
||||
# Drift manifest — see deployment_drift() in check_and_enable_services.sh. Same
|
||||
# rationale as the pacman wrapper's: "the file exists" is not "the right version
|
||||
# is deployed", and only the second one is worth re-checking every hour.
|
||||
mkdir -p "$(dirname "$SOURCE_MANIFEST")"
|
||||
{
|
||||
sha256sum "$WRAPPER_SOURCE" "$LOCK_LIB_SOURCE" "$REWRAP_SOURCE" "$HOOK_SOURCE" || {
|
||||
echo -e "${RED}Failed to checksum makepkg wrapper sources${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
sha256sum "$WRAPPER_DEST" "$REWRAP_DEST" "$HOOK_DEST" || {
|
||||
echo -e "${RED}Failed to checksum installed makepkg wrapper${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
} >"$SOURCE_MANIFEST"
|
||||
|
||||
if [[ ! -s $SOURCE_MANIFEST ]]; then
|
||||
echo -e "${RED}Error: drift manifest was not created or is empty${NC}" >&2
|
||||
exit 1
|
||||
fi
|
||||
chmod 644 "$SOURCE_MANIFEST"
|
||||
|
||||
echo -e "${GREEN}makepkg wrapper installed.${NC} Original preserved at ${CYAN}/usr/bin/makepkg.orig${NC}"
|
||||
echo -e "Upgrade-survival hook installed at ${CYAN}${HOOK_DEST}${NC}"
|
||||
|
||||
@ -27,6 +27,7 @@ WHITELIST_SOURCE="$(dirname "$0")/pacman_whitelist.txt"
|
||||
GREYLIST_SOURCE="$(dirname "$0")/pacman_greylist.txt"
|
||||
MAKEPKG_CAPPED_SOURCE="$(dirname "$0")/makepkg_capped.sh"
|
||||
MKPKG_SOURCE="$(dirname "$0")/mkpkg.sh"
|
||||
HEAVY_LOCK_SOURCE="$(dirname "$0")/../../utils/heavy_job_lock.sh"
|
||||
INSTALL_DIR="/usr/local/bin"
|
||||
WRAPPER_DEST="${INSTALL_DIR}/pacman_wrapper"
|
||||
LOCK_LIB_DEST="${INSTALL_DIR}/pacman_lock_lib.sh"
|
||||
@ -36,8 +37,10 @@ WHITELIST_DEST="${INSTALL_DIR}/pacman_whitelist.txt"
|
||||
GREYLIST_DEST="${INSTALL_DIR}/pacman_greylist.txt"
|
||||
MAKEPKG_CAPPED_DEST="${INSTALL_DIR}/makepkg_capped"
|
||||
MKPKG_DEST="${INSTALL_DIR}/mkpkg"
|
||||
HEAVY_LOCK_DEST="${INSTALL_DIR}/heavy_job_lock.sh"
|
||||
INTEGRITY_DIR="/var/lib/pacman-wrapper"
|
||||
INTEGRITY_FILE="${INTEGRITY_DIR}/policy.sha256"
|
||||
SOURCE_MANIFEST="${INTEGRITY_DIR}/source.sha256"
|
||||
LEECHBLOCK_INSTALLER_SOURCE="$(dirname "$0")/../install_leechblock.sh"
|
||||
LEECHBLOCK_DEFAULTS_SOURCE="$(dirname "$0")/../leechblock_defaults.json"
|
||||
LEECHBLOCK_SEEDER_SOURCE="$(dirname "$0")/../seed_leechblock_storage.js"
|
||||
@ -128,7 +131,12 @@ copy_managed_file "$MAKEPKG_CAPPED_SOURCE" "$MAKEPKG_CAPPED_DEST" required "make
|
||||
chmod +x "$MAKEPKG_CAPPED_DEST"
|
||||
copy_managed_file "$MKPKG_SOURCE" "$MKPKG_DEST" required "mkpkg helper"
|
||||
chmod +x "$MKPKG_DEST"
|
||||
chmod 644 "$WORDS_DEST" "$BLOCKED_DEST" "$WHITELIST_DEST" "$GREYLIST_DEST" 2> /dev/null || true
|
||||
# The heavy-job lock is sourced by pacman_wrapper.sh and makepkg_capped.sh from
|
||||
# a fixed absolute path, so it has to land next to them rather than be read out
|
||||
# of a repo checkout that may move.
|
||||
copy_managed_file "$HEAVY_LOCK_SOURCE" "$HEAVY_LOCK_DEST" required "heavy-job lock library"
|
||||
chmod 755 "$HEAVY_LOCK_DEST"
|
||||
chmod 644 "$WORDS_DEST" "$BLOCKED_DEST" "$WHITELIST_DEST" "$GREYLIST_DEST" 2>/dev/null || true
|
||||
|
||||
# Automatically use symbolic link installation method
|
||||
echo -e "${YELLOW}Installing using symbolic link method...${NC}"
|
||||
@ -167,17 +175,29 @@ if [[ ${#missing_files[@]} -gt 0 ]]; then
|
||||
fi
|
||||
|
||||
{
|
||||
sha256sum "$BLOCKED_DEST" || { echo -e "${RED}Failed to checksum blocked list${NC}" >&2; exit 1; }
|
||||
sha256sum "$GREYLIST_DEST" || { echo -e "${RED}Failed to checksum greylist${NC}" >&2; exit 1; }
|
||||
sha256sum "$BLOCKED_DEST" || {
|
||||
echo -e "${RED}Failed to checksum blocked list${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
sha256sum "$GREYLIST_DEST" || {
|
||||
echo -e "${RED}Failed to checksum greylist${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
# The shared stale-lock library is executed (sourced) by the wrapper, so it is
|
||||
# integrity-checked too: pacman_wrapper.sh sources it only AFTER
|
||||
# verify_policy_integrity passes, so a tampered lib is rejected before it runs.
|
||||
sha256sum "$LOCK_LIB_DEST" || { echo -e "${RED}Failed to checksum lock library${NC}" >&2; exit 1; }
|
||||
sha256sum "$LOCK_LIB_DEST" || {
|
||||
echo -e "${RED}Failed to checksum lock library${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
# Whitelist is optional
|
||||
if [[ -f "$WHITELIST_DEST" ]]; then
|
||||
sha256sum "$WHITELIST_DEST" || { echo -e "${RED}Failed to checksum whitelist${NC}" >&2; exit 1; }
|
||||
sha256sum "$WHITELIST_DEST" || {
|
||||
echo -e "${RED}Failed to checksum whitelist${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
fi
|
||||
} > "$INTEGRITY_FILE"
|
||||
} >"$INTEGRITY_FILE"
|
||||
|
||||
# Verify integrity file was created and has content
|
||||
if [[ ! -s "$INTEGRITY_FILE" ]]; then
|
||||
@ -187,13 +207,55 @@ fi
|
||||
|
||||
# Make integrity file immutable
|
||||
chmod 400 "$INTEGRITY_FILE"
|
||||
if command -v chattr > /dev/null 2>&1; then
|
||||
if command -v chattr >/dev/null 2>&1; then
|
||||
chattr +i "$INTEGRITY_FILE" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make integrity file immutable${NC}"
|
||||
fi
|
||||
|
||||
# Record a drift manifest: the hash of every SOURCE file we installed from, and
|
||||
# of the installed copies we produced. check_and_enable_services.sh replays it
|
||||
# with `sha256sum -c`, which answers both questions in one command:
|
||||
# - a source line fails => the repo moved on, this deployment is stale
|
||||
# - a dest line fails => someone edited the installed copy directly
|
||||
# Neither was detectable before: the checker only tested that files EXIST, which
|
||||
# is why a wrapper 7.6 KB behind the repo (no integrity manifest, no
|
||||
# pacman_lock_lib, no guard-lib fallbacks) ran unnoticed for a week while the
|
||||
# hourly maintenance timer kept reporting "ok".
|
||||
# Deliberately NOT chattr +i (unlike the policy integrity file above): this is a
|
||||
# drift record, not a security boundary, and it must be rewritable each install.
|
||||
echo -e "${BLUE}Recording deployment drift manifest...${NC}"
|
||||
# ABSOLUTE paths only. The SOURCE_* vars above are built from `dirname "$0"`, so
|
||||
# they are relative whenever the installer is invoked by a relative path — and
|
||||
# check_and_enable_services.sh replays this manifest from systemd with cwd=/,
|
||||
# where relative entries resolve to nothing. sha256sum -c would report those as
|
||||
# failures, the checker would read that as drift, and it would reinstall on
|
||||
# every single hourly run forever.
|
||||
manifest_sources=()
|
||||
for src in "$WRAPPER_SOURCE" "$LOCK_LIB_SOURCE" "$BLOCKED_SOURCE" "$GREYLIST_SOURCE" \
|
||||
"$MAKEPKG_CAPPED_SOURCE" "$MKPKG_SOURCE" "$WHITELIST_SOURCE"; do
|
||||
[[ -f "$src" ]] || continue # whitelist is optional
|
||||
manifest_sources+=("$(readlink -f "$src")")
|
||||
done
|
||||
|
||||
{
|
||||
sha256sum "${manifest_sources[@]}" || {
|
||||
echo -e "${RED}Failed to checksum wrapper sources${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
sha256sum "$LOCK_LIB_DEST" || {
|
||||
echo -e "${RED}Failed to checksum installed lock lib${NC}" >&2
|
||||
exit 1
|
||||
}
|
||||
} >"$SOURCE_MANIFEST"
|
||||
|
||||
if [[ ! -s "$SOURCE_MANIFEST" ]]; then
|
||||
echo -e "${RED}Error: drift manifest was not created or is empty${NC}" >&2
|
||||
exit 1
|
||||
fi
|
||||
chmod 644 "$SOURCE_MANIFEST"
|
||||
|
||||
# Make policy files immutable to prevent easy tampering
|
||||
echo -e "${BLUE}Protecting policy files from modification...${NC}"
|
||||
if command -v chattr > /dev/null 2>&1; then
|
||||
if command -v chattr >/dev/null 2>&1; then
|
||||
chattr +i "$BLOCKED_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make blocked list immutable${NC}"
|
||||
chattr +i "$GREYLIST_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make greylist immutable${NC}"
|
||||
chattr +i "$LOCK_LIB_DEST" 2>/dev/null || echo -e "${YELLOW}Warning: Could not make lock library immutable${NC}"
|
||||
|
||||
@ -15,6 +15,21 @@ if ! command -v makepkg >/dev/null 2>&1; then
|
||||
exit 127
|
||||
fi
|
||||
|
||||
# Serialise against other heavy jobs (Gradle/Flutter builds, coverage runs,
|
||||
# pacman transactions) on this 7.6 GB machine. Re-exec ourselves under the lock
|
||||
# instead of wrapping each branch below, so all three fallback paths are covered
|
||||
# by one guard. HEAVY_JOB_LOCK_HELD keeps the re-exec from recursing.
|
||||
# Note the cgroup caps below are NOT a substitute: MemoryMax defaults to 12G on
|
||||
# a machine with 7.6G, so they bound one build's shape, not the machine's total.
|
||||
HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh"
|
||||
if [[ -r $HEAVY_JOB_LOCK_LIB && -z ${HEAVY_JOB_LOCK_HELD:-} ]]; then
|
||||
# shellcheck source=/dev/null
|
||||
source "$HEAVY_JOB_LOCK_LIB"
|
||||
export HEAVY_JOB_LOCK_HELD=1
|
||||
with_heavy_lock makepkg -- "$0" "$@"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
run_makepkg_fallback() {
|
||||
exec ionice -c "$IONICE_CLASS" -n "$IONICE_LEVEL" \
|
||||
nice -n "$NICE_LEVEL" \
|
||||
|
||||
@ -12,6 +12,10 @@ BOLD='\033[1m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
PACMAN_BIN="/usr/bin/pacman"
|
||||
# Installed alongside the wrapper by install_pacman_wrapper.sh. Absent on a
|
||||
# machine that has not run it yet, in which case transactions simply are not
|
||||
# serialised (see the guarded source below).
|
||||
HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh"
|
||||
MAKEPKG_CAPPED_BIN="/usr/local/bin/makepkg_capped"
|
||||
|
||||
declare -a BLOCKED_KEYWORDS_LIST=()
|
||||
@ -87,7 +91,7 @@ load_policy_lists() {
|
||||
continue
|
||||
fi
|
||||
target_array+=("${line,,}")
|
||||
done < "$file_path"
|
||||
done <"$file_path"
|
||||
}
|
||||
|
||||
if [[ -f $blocked_file ]]; then
|
||||
@ -741,7 +745,19 @@ if should_use_wrapper_guard_lib_fallback "$@"; then
|
||||
manual_guard_lib_fallback=1
|
||||
fi
|
||||
|
||||
"$PACMAN_BIN" "$@"
|
||||
# Serialise real TRANSACTIONS against builds and coverage runs on this 7.6 GB
|
||||
# box (see utils/heavy_job_lock.sh). Deliberately scoped by needs_unlock, i.e.
|
||||
# -S/-U/-R only: read-only queries like `pacman -Q` take no db lock and run
|
||||
# constantly from background services here, so making those wait behind a
|
||||
# 20-minute Gradle build would be worse than the contention it avoids.
|
||||
# with_heavy_lock always fails open, so this can never brick package management.
|
||||
if [[ -r $HEAVY_JOB_LOCK_LIB ]] && needs_unlock "$@"; then
|
||||
# shellcheck source=/dev/null
|
||||
source "$HEAVY_JOB_LOCK_LIB"
|
||||
with_heavy_lock pacman -- "$PACMAN_BIN" "$@"
|
||||
else
|
||||
"$PACMAN_BIN" "$@"
|
||||
fi
|
||||
exit_code=$?
|
||||
|
||||
if [[ $manual_guard_lib_fallback -eq 1 ]]; then
|
||||
|
||||
232
linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh
Executable file
232
linux_configuration/scripts/periodic_background/utils/heavy_job_lock.sh
Executable file
@ -0,0 +1,232 @@
|
||||
#!/bin/bash
|
||||
|
||||
# ============================================================================
|
||||
# heavy_job_lock.sh - one heavy job at a time on this machine
|
||||
#
|
||||
# This box has 7.6 GB. On 2026-08-03 a Gradle/Flutter APK build, a full
|
||||
# `pytest --cov` run and Chromium overlapped; an earlier overlap the same week
|
||||
# got a whole session OOM-killed (exit 137/143). The rule "one heavy job at a
|
||||
# time" was written down in ~/.claude/memories/mistakes.md - but a rule that
|
||||
# only works when a human remembers it is deployment hygiene, not a fix. This
|
||||
# makes the overlap impossible instead.
|
||||
#
|
||||
# Usage as a CLI:
|
||||
# heavy_job_lock.sh --name gradle -- flutter build apk --debug
|
||||
# heavy_job_lock.sh --name pytest --fail-fast -- pytest --cov
|
||||
#
|
||||
# Usage as a library:
|
||||
# source /path/to/heavy_job_lock.sh
|
||||
# with_heavy_lock gradle -- ./gradlew assembleDebug
|
||||
#
|
||||
# Setup (once, as root - /run is tmpfs so the lock needs recreating on boot):
|
||||
# sudo heavy_job_lock.sh --install
|
||||
#
|
||||
# Design notes:
|
||||
# - ONE system-wide lock shared between root and the user. pacman runs as
|
||||
# root while builds run as the user, so a per-user /run/user/$UID lock
|
||||
# would give them separate locks and no mutual exclusion at all.
|
||||
# - The lock file is mode 0666 (created by root via tmpfiles.d) precisely so
|
||||
# an unprivileged build can take the same lock a root pacman takes.
|
||||
# - FAIL OPEN, always. A lock this central must never be able to brick
|
||||
# package management or builds: if the lock file cannot be created, or the
|
||||
# wait times out, we warn and run the command anyway. Blocking work forever
|
||||
# is a worse failure than the memory contention we are avoiding.
|
||||
# ============================================================================
|
||||
|
||||
HEAVY_JOB_LOCK_FILE="${HEAVY_JOB_LOCK_FILE:-/run/heavy-job.lock}"
|
||||
# Long enough for a real Gradle build to finish ahead of us, short enough that
|
||||
# a wedged holder cannot stall a machine indefinitely.
|
||||
HEAVY_JOB_LOCK_TIMEOUT="${HEAVY_JOB_LOCK_TIMEOUT:-1800}"
|
||||
HEAVY_JOB_TMPFILES_CONF="/etc/tmpfiles.d/heavy-job-lock.conf"
|
||||
|
||||
_hjl_warn() { printf '[heavy-job-lock] %s\n' "$*" >&2; }
|
||||
|
||||
# Who holds it right now? Best-effort: the holder writes "<name> <pid> <date>"
|
||||
# after acquiring, so a refusal can name the culprit instead of just failing.
|
||||
_hjl_holder() {
|
||||
local holder=""
|
||||
[[ -r $HEAVY_JOB_LOCK_FILE ]] && read -r holder <"$HEAVY_JOB_LOCK_FILE" 2>/dev/null
|
||||
printf '%s' "${holder:-unknown}"
|
||||
}
|
||||
|
||||
# Create the lock file world-writable. Only meaningful as root; unprivileged
|
||||
# callers just discover they cannot and fail open.
|
||||
_hjl_ensure_lock_file() {
|
||||
[[ -e $HEAVY_JOB_LOCK_FILE ]] && return 0
|
||||
if [[ $EUID -eq 0 ]]; then
|
||||
install -m 0666 /dev/null "$HEAVY_JOB_LOCK_FILE" 2>/dev/null && return 0
|
||||
else
|
||||
# Braces around the redirect: a failing redirection is reported by the
|
||||
# shell itself, before the command runs, so `cmd 2>/dev/null` does not
|
||||
# suppress it. The group does.
|
||||
{ : >"$HEAVY_JOB_LOCK_FILE"; } 2>/dev/null &&
|
||||
chmod 0666 "$HEAVY_JOB_LOCK_FILE" 2>/dev/null && return 0
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# with_heavy_lock <name> [--fail-fast] -- <command...>
|
||||
with_heavy_lock() {
|
||||
local name="$1"
|
||||
shift
|
||||
local fail_fast=0
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--fail-fast)
|
||||
fail_fast=1
|
||||
shift
|
||||
;;
|
||||
--)
|
||||
shift
|
||||
break
|
||||
;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
_hjl_warn "with_heavy_lock: no command given"
|
||||
return 2
|
||||
fi
|
||||
|
||||
if ! _hjl_ensure_lock_file; then
|
||||
_hjl_warn "cannot create $HEAVY_JOB_LOCK_FILE - running '$name' UNSERIALISED"
|
||||
_hjl_warn "run 'sudo $0 --install' once to enable serialisation"
|
||||
"$@"
|
||||
return $?
|
||||
fi
|
||||
|
||||
local lock_fd
|
||||
# `<>` (read-write), NOT `>`. Redirecting with `>` truncates the file at
|
||||
# open time, so a second caller wiped the holder's "<name> <pid> <date>"
|
||||
# line before it could read it — every refusal said "held by unknown".
|
||||
exec {lock_fd}<>"$HEAVY_JOB_LOCK_FILE" || {
|
||||
_hjl_warn "cannot open lock - running '$name' unserialised"
|
||||
"$@"
|
||||
return $?
|
||||
}
|
||||
|
||||
local acquired=0
|
||||
if ((fail_fast == 1)); then
|
||||
if flock --nonblock "$lock_fd"; then
|
||||
acquired=1
|
||||
else
|
||||
_hjl_warn "refusing to start '$name': held by $(_hjl_holder)"
|
||||
exec {lock_fd}>&-
|
||||
return 75 # EX_TEMPFAIL
|
||||
fi
|
||||
else
|
||||
local holder
|
||||
holder="$(_hjl_holder)"
|
||||
if ! flock --nonblock "$lock_fd"; then
|
||||
_hjl_warn "waiting for '$holder' to finish before starting '$name' (timeout ${HEAVY_JOB_LOCK_TIMEOUT}s)"
|
||||
fi
|
||||
if flock --timeout "$HEAVY_JOB_LOCK_TIMEOUT" "$lock_fd"; then
|
||||
acquired=1
|
||||
else
|
||||
# Fail open: see the header. A stuck holder must not permanently
|
||||
# block builds or package management.
|
||||
_hjl_warn "timed out after ${HEAVY_JOB_LOCK_TIMEOUT}s waiting for $(_hjl_holder)"
|
||||
_hjl_warn "proceeding UNSERIALISED with '$name'"
|
||||
fi
|
||||
fi
|
||||
|
||||
if ((acquired == 1)); then
|
||||
printf '%s %s %s\n' "$name" "$$" "$(date -Is 2>/dev/null || echo unknown)" \
|
||||
>"$HEAVY_JOB_LOCK_FILE" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
local rc=0
|
||||
"$@" || rc=$?
|
||||
|
||||
# Closing the fd releases the lock; do it explicitly so the holder line is
|
||||
# not left implying we still hold it.
|
||||
exec {lock_fd}>&-
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# CLI (skipped when this file is sourced as a library)
|
||||
# ----------------------------------------------------------------------------
|
||||
_hjl_install() {
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
_hjl_warn "--install must run as root"
|
||||
return 1
|
||||
fi
|
||||
# /run is tmpfs: without a tmpfiles.d entry the lock vanishes every boot and
|
||||
# the first unprivileged caller silently falls back to unserialised.
|
||||
printf 'f %s 0666 root root -\n' "$HEAVY_JOB_LOCK_FILE" >"$HEAVY_JOB_TMPFILES_CONF"
|
||||
systemd-tmpfiles --create "$HEAVY_JOB_TMPFILES_CONF" 2>/dev/null || true
|
||||
_hjl_ensure_lock_file || {
|
||||
_hjl_warn "failed to create $HEAVY_JOB_LOCK_FILE"
|
||||
return 1
|
||||
}
|
||||
printf 'installed %s and %s\n' "$HEAVY_JOB_TMPFILES_CONF" "$HEAVY_JOB_LOCK_FILE"
|
||||
}
|
||||
|
||||
_hjl_usage() {
|
||||
cat <<EOF
|
||||
Usage: $(basename "$0") --name <label> [--fail-fast] -- <command...>
|
||||
$(basename "$0") --install # once, as root
|
||||
$(basename "$0") --status
|
||||
|
||||
Serialises heavy jobs (builds, coverage runs, pacman transactions) so only one
|
||||
runs at a time on this memory-constrained machine. Always fails open.
|
||||
|
||||
Options:
|
||||
--name <label> Name recorded as the lock holder (shown to whoever waits)
|
||||
--fail-fast Refuse immediately if held, instead of waiting
|
||||
--install Create the lock file + tmpfiles.d entry (root)
|
||||
--status Show the current holder
|
||||
EOF
|
||||
}
|
||||
|
||||
# ${BASH_SOURCE[0]} != $0 means we were sourced - define functions, run nothing.
|
||||
if [[ ${BASH_SOURCE[0]} == "${0}" ]]; then
|
||||
set -uo pipefail
|
||||
|
||||
_name="heavy-job"
|
||||
_fail_fast=()
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--name)
|
||||
_name="$2"
|
||||
shift 2
|
||||
;;
|
||||
--fail-fast)
|
||||
_fail_fast=(--fail-fast)
|
||||
shift
|
||||
;;
|
||||
--install)
|
||||
_hjl_install
|
||||
exit $?
|
||||
;;
|
||||
--status)
|
||||
printf 'lock file: %s\n' "$HEAVY_JOB_LOCK_FILE"
|
||||
printf 'holder: %s\n' "$(_hjl_holder)"
|
||||
exit 0
|
||||
;;
|
||||
-h | --help)
|
||||
_hjl_usage
|
||||
exit 0
|
||||
;;
|
||||
--)
|
||||
shift
|
||||
break
|
||||
;;
|
||||
*)
|
||||
_hjl_warn "unknown option: $1"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
_hjl_usage
|
||||
exit 2
|
||||
fi
|
||||
|
||||
with_heavy_lock "$_name" "${_fail_fast[@]}" -- "$@"
|
||||
exit $?
|
||||
fi
|
||||
493
linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh
Executable file
493
linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh
Executable file
@ -0,0 +1,493 @@
|
||||
#!/bin/bash
|
||||
|
||||
# ============================================================================
|
||||
# diagnose_pacman_hook_stall.sh
|
||||
#
|
||||
# Reproduces and instruments the intermittent pacman freeze that happens at the
|
||||
# FIRST PreTransaction hook of a transaction:
|
||||
#
|
||||
# :: Running pre-transaction hooks...
|
||||
# (1/2) guard-lib: unlocking protected files before pacman transaction
|
||||
# <hangs for minutes>
|
||||
#
|
||||
# /var/log/pacman.log shows the hook's "running '...hook'..." line and then
|
||||
# nothing at all - the next entry is the user's retry. Five such transactions
|
||||
# were abandoned between 2026-07-29 and 2026-08-03. The hook script itself is
|
||||
# provably a no-op on this machine (/etc/guard-lib/targets is empty, so its
|
||||
# loop body never executes), which means the stall is pacman forking/exec'ing
|
||||
# the hook, not the hook doing work.
|
||||
#
|
||||
# This script drives real transactions in a loop and, when one stalls, captures
|
||||
# the kernel-side evidence needed to tell "blocked in a syscall" from "the box
|
||||
# could not spawn a process": /proc/<pid>/stack, wchan, syscall, the process
|
||||
# tree, and memory/PSI counters.
|
||||
#
|
||||
# The repro target is a 0-file meta-package (base-devel), reinstalled from the
|
||||
# local cache: a real transaction that fires every hook but writes no files and
|
||||
# needs no network.
|
||||
# ============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_NAME="$(basename "$0")"
|
||||
readonly SCRIPT_NAME
|
||||
readonly PACMAN_BIN="/usr/bin/pacman.orig"
|
||||
readonly PACMAN_LOG="/var/log/pacman.log"
|
||||
readonly CACHE_DIR="/var/cache/pacman/pkg"
|
||||
|
||||
# Defaults
|
||||
RUNS=40
|
||||
PACKAGE="base-devel"
|
||||
STALL_TIMEOUT=20 # seconds with no new pacman.log line => declare a stall
|
||||
HARD_TIMEOUT=120 # seconds before we give up and kill the stalled transaction
|
||||
OUT_DIR="/var/log/pacman-hook-stall"
|
||||
WITH_LOAD=0
|
||||
WATCH_MODE=0
|
||||
LOAD_FLOOR_MB=800 # never push MemAvailable below this
|
||||
LOAD_MIN_FREE_MB=1500 # refuse to start --with-load below this much available
|
||||
|
||||
HOG_FILE=""
|
||||
LOAD_PID=""
|
||||
PACMAN_PID=""
|
||||
STALLS=0
|
||||
RUN_INDEX=0
|
||||
LAST_ELAPSED=0
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $SCRIPT_NAME [OPTIONS]
|
||||
|
||||
Drives repeated pacman transactions and captures diagnostics when one stalls
|
||||
at its first PreTransaction hook.
|
||||
|
||||
Options:
|
||||
-n, --runs N Number of transactions to drive (default: $RUNS)
|
||||
-p, --package NAME Package to reinstall from cache (default: $PACKAGE)
|
||||
-t, --timeout S Seconds of pacman.log silence => stall (default: $STALL_TIMEOUT)
|
||||
--hard-timeout S Seconds before killing a stalled run (default: $HARD_TIMEOUT)
|
||||
--with-load Also apply memory pressure (the hypothesis under test).
|
||||
Run this with nothing else going - it deliberately
|
||||
breaks the one-heavy-job-at-a-time rule.
|
||||
--watch Passive mode: drive nothing, just watch pacman.log and
|
||||
dump diagnostics when someone else's transaction stalls
|
||||
on a hook. Intended to run as a systemd service.
|
||||
-o, --out DIR Where to write stall dumps (default: $OUT_DIR)
|
||||
-h, --help Show this help
|
||||
|
||||
Exit status: 0 if the loop completed (with or without stalls), non-zero on a
|
||||
setup failure. The stall count is reported in the summary.
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
stop_load
|
||||
if [[ -n "$PACMAN_PID" ]] && kill -0 "$PACMAN_PID" 2>/dev/null; then
|
||||
echo "Cleaning up: killing in-flight pacman ($PACMAN_PID)" >&2
|
||||
kill_tree "$PACMAN_PID"
|
||||
fi
|
||||
exit "$rc"
|
||||
}
|
||||
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Preconditions
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
require_root() {
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
exec sudo -E bash "$0" "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
validate_requirements() {
|
||||
local tool
|
||||
for tool in "$PACMAN_BIN" ps awk sed journalctl; do
|
||||
command -v "$tool" >/dev/null 2>&1 || [[ -x "$tool" ]] || {
|
||||
echo "Error: required tool '$tool' not found" >&2
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
[[ -r "$PACMAN_LOG" ]] || {
|
||||
echo "Error: cannot read $PACMAN_LOG" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Refuse to run alongside another *transaction*: a collision would look
|
||||
# exactly like the stall we are hunting. db.lck is the authoritative signal
|
||||
# - pacman holds it for the whole transaction. Deliberately NOT a pgrep for
|
||||
# "pacman": read-only queries (pacman -Qi) take no lock, are harmless, and
|
||||
# run constantly here from background services.
|
||||
if [[ -e /var/lib/pacman/db.lck ]]; then
|
||||
echo "Error: /var/lib/pacman/db.lck exists - another transaction is in" >&2
|
||||
echo "flight (or a stale lock remains). Resolve it before running." >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Resolve the cached package file for the installed version of $PACKAGE.
|
||||
resolve_package_file() {
|
||||
local version
|
||||
version="$("$PACMAN_BIN" -Q "$PACKAGE" 2>/dev/null | awk '{print $2}')"
|
||||
[[ -n "$version" ]] || {
|
||||
echo "Error: package '$PACKAGE' is not installed" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
local candidate
|
||||
for candidate in "$CACHE_DIR/$PACKAGE-$version"-*.pkg.tar.zst; do
|
||||
[[ -e "$candidate" ]] || continue
|
||||
printf '%s\n' "$candidate"
|
||||
return 0
|
||||
done
|
||||
|
||||
echo "Error: no cached package for $PACKAGE-$version in $CACHE_DIR" >&2
|
||||
echo "Hint: run '$PACMAN_BIN -Sw $PACKAGE' first" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Load generator (tmpfs allocation = real anonymous-ish memory, no extra deps)
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
mem_available_mb() {
|
||||
local kb=0 key value rest
|
||||
while read -r key value rest; do
|
||||
if [[ $key == "MemAvailable:" ]]; then
|
||||
kb="$value"
|
||||
break
|
||||
fi
|
||||
done </proc/meminfo
|
||||
printf '%d\n' $((kb / 1024))
|
||||
}
|
||||
|
||||
start_load() {
|
||||
((WITH_LOAD == 1)) || return 0
|
||||
|
||||
local avail
|
||||
avail="$(mem_available_mb)"
|
||||
if ((avail < LOAD_MIN_FREE_MB)); then
|
||||
echo "Error: only ${avail} MB available; --with-load needs >= ${LOAD_MIN_FREE_MB} MB" >&2
|
||||
echo "Close Chromium / builds first." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
local target=$((avail - LOAD_FLOOR_MB))
|
||||
HOG_FILE="$(mktemp /dev/shm/hook-stall-hog.XXXXXX)"
|
||||
echo "Applying memory pressure: allocating ${target} MB (floor ${LOAD_FLOOR_MB} MB available)"
|
||||
dd if=/dev/zero of="$HOG_FILE" bs=1M count="$target" status=none 2>/dev/null || true
|
||||
echo "MemAvailable now: $(mem_available_mb) MB"
|
||||
}
|
||||
|
||||
stop_load() {
|
||||
if [[ -n "$LOAD_PID" ]] && kill -0 "$LOAD_PID" 2>/dev/null; then
|
||||
kill "$LOAD_PID" 2>/dev/null || true
|
||||
fi
|
||||
if [[ -n "$HOG_FILE" && -e "$HOG_FILE" ]]; then
|
||||
rm -f "$HOG_FILE"
|
||||
HOG_FILE=""
|
||||
fi
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Diagnostics capture
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
# Print every descendant PID of $1, including $1 itself.
|
||||
descendant_pids() {
|
||||
local root="$1"
|
||||
local pids=("$root")
|
||||
local i=0
|
||||
while ((i < ${#pids[@]})); do
|
||||
local parent="${pids[i]}"
|
||||
local child
|
||||
while read -r child; do
|
||||
[[ -n "$child" ]] && pids+=("$child")
|
||||
done < <(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ')
|
||||
i=$((i + 1))
|
||||
done
|
||||
printf '%s\n' "${pids[@]}"
|
||||
}
|
||||
|
||||
kill_tree() {
|
||||
local root="$1"
|
||||
local pid
|
||||
while read -r pid; do
|
||||
kill -9 "$pid" 2>/dev/null || true
|
||||
done < <(descendant_pids "$root" | tac)
|
||||
}
|
||||
|
||||
capture_stall() {
|
||||
local root_pid="$1"
|
||||
local run="$2"
|
||||
local stamp
|
||||
printf -v stamp '%(%Y%m%d-%H%M%S)T' -1
|
||||
|
||||
# Classify before capturing. The stall we are hunting leaves pacman.log
|
||||
# ending on a hook's "running '...hook'..." line; anything else stalled
|
||||
# somewhere unrelated and must not be counted as a reproduction.
|
||||
local kind="other"
|
||||
if tail -1 "$PACMAN_LOG" 2>/dev/null | grep -q "running '.*\.hook'"; then
|
||||
kind="hook"
|
||||
fi
|
||||
|
||||
local dir="$OUT_DIR/${stamp}-run${run}-${kind}"
|
||||
mkdir -p "$dir"
|
||||
|
||||
echo " !! STALL ($kind) - capturing diagnostics to $dir"
|
||||
tail -1 "$PACMAN_LOG" >"$dir/last-log-line.txt" 2>&1 || true
|
||||
|
||||
ps -eo pid,ppid,stat,wchan:32,etime,args --forest >"$dir/ps-forest.txt" 2>&1 || true
|
||||
|
||||
local pid
|
||||
while read -r pid; do
|
||||
[[ -d "/proc/$pid" ]] || continue
|
||||
{
|
||||
echo "=== pid $pid ==="
|
||||
echo "--- cmdline ---"
|
||||
tr '\0' ' ' <"/proc/$pid/cmdline" 2>/dev/null || true
|
||||
echo
|
||||
echo "--- wchan ---"
|
||||
cat "/proc/$pid/wchan" 2>/dev/null || true
|
||||
echo
|
||||
echo "--- syscall ---"
|
||||
cat "/proc/$pid/syscall" 2>/dev/null || true
|
||||
echo "--- stack ---"
|
||||
cat "/proc/$pid/stack" 2>/dev/null || echo "(unavailable - needs CONFIG_STACKTRACE)"
|
||||
echo "--- status ---"
|
||||
grep -E "^(Name|State|Threads|VmRSS|SigBlk|SigIgn|SigCgt)" "/proc/$pid/status" 2>/dev/null || true
|
||||
echo "--- smaps_rollup (Pss) ---"
|
||||
grep -E "^Pss:" "/proc/$pid/smaps_rollup" 2>/dev/null || true
|
||||
echo
|
||||
} >>"$dir/procs.txt" 2>&1
|
||||
done < <(descendant_pids "$root_pid")
|
||||
|
||||
cp /proc/meminfo "$dir/meminfo.txt" 2>/dev/null || true
|
||||
{
|
||||
echo "--- pressure/memory ---"
|
||||
cat /proc/pressure/memory 2>/dev/null || true
|
||||
echo "--- pressure/io ---"
|
||||
cat /proc/pressure/io 2>/dev/null || true
|
||||
echo "--- pressure/cpu ---"
|
||||
cat /proc/pressure/cpu 2>/dev/null || true
|
||||
} >"$dir/pressure.txt" 2>&1
|
||||
|
||||
journalctl -k -n 50 --no-pager >"$dir/dmesg-tail.txt" 2>&1 || true
|
||||
tail -30 "$PACMAN_LOG" >"$dir/pacman-log-tail.txt" 2>&1 || true
|
||||
|
||||
STALLS=$((STALLS + 1))
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# The repro loop
|
||||
# ----------------------------------------------------------------------------
|
||||
|
||||
log_size() { stat -c %s "$PACMAN_LOG" 2>/dev/null || echo 0; }
|
||||
|
||||
# Run one transaction, watching pacman.log for silence. Sets LAST_ELAPSED and,
|
||||
# when it hangs, bumps STALLS via capture_stall. Deliberately NOT called in a
|
||||
# command substitution: both counters and PACMAN_PID (which the EXIT trap needs)
|
||||
# must live in this shell, not a subshell.
|
||||
run_one() {
|
||||
local pkg_file="$1"
|
||||
local run="$2"
|
||||
|
||||
local started
|
||||
printf -v started '%(%s)T' -1
|
||||
|
||||
"$PACMAN_BIN" -U --noconfirm "$pkg_file" >/dev/null 2>&1 &
|
||||
PACMAN_PID=$!
|
||||
|
||||
local last_size last_change now captured=0
|
||||
last_size="$(log_size)"
|
||||
printf -v last_change '%(%s)T' -1
|
||||
|
||||
while kill -0 "$PACMAN_PID" 2>/dev/null; do
|
||||
sleep 1
|
||||
local size
|
||||
size="$(log_size)"
|
||||
printf -v now '%(%s)T' -1
|
||||
|
||||
if [[ $size != "$last_size" ]]; then
|
||||
last_size="$size"
|
||||
last_change="$now"
|
||||
elif ((captured == 0 && now - last_change >= STALL_TIMEOUT)); then
|
||||
capture_stall "$PACMAN_PID" "$run"
|
||||
captured=1
|
||||
fi
|
||||
|
||||
if ((now - started >= HARD_TIMEOUT)); then
|
||||
echo " !! hard timeout (${HARD_TIMEOUT}s) - killing run $run"
|
||||
((captured == 0)) && capture_stall "$PACMAN_PID" "$run"
|
||||
kill_tree "$PACMAN_PID"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
wait "$PACMAN_PID" 2>/dev/null || true
|
||||
PACMAN_PID=""
|
||||
|
||||
local ended
|
||||
printf -v ended '%(%s)T' -1
|
||||
LAST_ELAPSED=$((ended - started))
|
||||
}
|
||||
|
||||
# Passive mode: watch OTHER people's transactions instead of driving our own.
|
||||
#
|
||||
# This exists because the active repro loop could not reproduce the stall: 65
|
||||
# transactions (40 idle + 25 under 2.6 GB of memory pressure) ran clean, against
|
||||
# a historical rate of 4 in 56. At a uniform 7% the odds of 65 clean runs are
|
||||
# under 1%, so the trigger is conditional on something a synthetic loop does not
|
||||
# recreate - which means the evidence has to be captured from a real stall, when
|
||||
# it next happens, rather than manufactured.
|
||||
#
|
||||
# Runs as a systemd service; costs one stat(2) per second and nothing else.
|
||||
watch_forever() {
|
||||
echo "Watching $PACMAN_LOG for hook stalls (>= ${STALL_TIMEOUT}s silent)"
|
||||
echo "Dumps: $OUT_DIR"
|
||||
|
||||
local last_size last_change now armed=0 seq=0
|
||||
last_size="$(log_size)"
|
||||
printf -v last_change '%(%s)T' -1
|
||||
|
||||
while true; do
|
||||
sleep 1
|
||||
local size
|
||||
size="$(log_size)"
|
||||
printf -v now '%(%s)T' -1
|
||||
|
||||
if [[ $size != "$last_size" ]]; then
|
||||
last_size="$size"
|
||||
last_change="$now"
|
||||
armed=0
|
||||
continue
|
||||
fi
|
||||
|
||||
# Silence alone is normal - no transaction is running most of the time.
|
||||
# Only a LIVE pacman sitting on a hook line is the signature we want.
|
||||
((armed == 1)) && continue
|
||||
((now - last_change >= STALL_TIMEOUT)) || continue
|
||||
|
||||
local pacman_pid
|
||||
pacman_pid="$(pgrep -x pacman.orig | head -1 || true)"
|
||||
[[ -n $pacman_pid ]] || continue
|
||||
|
||||
seq=$((seq + 1))
|
||||
capture_stall "$pacman_pid" "watch${seq}"
|
||||
logger -t pacman-hook-stall "captured a stalled transaction (pid $pacman_pid)"
|
||||
armed=1
|
||||
done
|
||||
}
|
||||
|
||||
main() {
|
||||
validate_requirements
|
||||
|
||||
if ((WATCH_MODE == 1)); then
|
||||
mkdir -p "$OUT_DIR"
|
||||
watch_forever
|
||||
return
|
||||
fi
|
||||
|
||||
local pkg_file
|
||||
pkg_file="$(resolve_package_file)"
|
||||
|
||||
mkdir -p "$OUT_DIR"
|
||||
|
||||
echo "============================================================================"
|
||||
echo "pacman hook-stall diagnosis"
|
||||
echo "============================================================================"
|
||||
echo " package : $PACKAGE ($pkg_file)"
|
||||
echo " runs : $RUNS"
|
||||
echo " stall after : ${STALL_TIMEOUT}s of pacman.log silence"
|
||||
echo " hard timeout: ${HARD_TIMEOUT}s"
|
||||
local load_desc="idle"
|
||||
((WITH_LOAD == 1)) && load_desc="memory pressure ON"
|
||||
echo " load : $load_desc"
|
||||
echo " dumps : $OUT_DIR"
|
||||
echo
|
||||
|
||||
start_load
|
||||
|
||||
local durations=()
|
||||
while ((RUN_INDEX < RUNS)); do
|
||||
RUN_INDEX=$((RUN_INDEX + 1))
|
||||
run_one "$pkg_file" "$RUN_INDEX"
|
||||
durations+=("$LAST_ELAPSED")
|
||||
local marker=""
|
||||
((LAST_ELAPSED >= STALL_TIMEOUT)) && marker=" <-- slow"
|
||||
printf ' run %2d/%d: %3ds%s\n' "$RUN_INDEX" "$RUNS" "$LAST_ELAPSED" "$marker"
|
||||
done
|
||||
|
||||
stop_load
|
||||
|
||||
echo
|
||||
echo "============================================================================"
|
||||
echo "Summary"
|
||||
echo "============================================================================"
|
||||
printf ' transactions : %d\n' "$RUNS"
|
||||
printf ' stalls : %d\n' "$STALLS"
|
||||
printf ' durations : %s\n' "${durations[*]}"
|
||||
printf '%s\n' "${durations[@]}" | sort -n | awk '
|
||||
{ d[NR] = $1 }
|
||||
END {
|
||||
if (NR == 0) exit
|
||||
printf " min/median/max: %ds / %ds / %ds\n", d[1], d[int((NR+1)/2)], d[NR]
|
||||
}'
|
||||
if ((STALLS > 0)); then
|
||||
echo
|
||||
echo " Captured dumps:"
|
||||
find "$OUT_DIR" -mindepth 1 -maxdepth 1 -type d -printf ' %p\n' \
|
||||
2>/dev/null | sort | tail -"$STALLS"
|
||||
fi
|
||||
echo "============================================================================"
|
||||
}
|
||||
|
||||
# Escalate BEFORE parsing: after the parse loop "$@" has been shifted away, so
|
||||
# re-exec'ing under sudo there would silently drop every flag.
|
||||
require_root "$@"
|
||||
|
||||
# Parse arguments
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
-n | --runs)
|
||||
RUNS="$2"
|
||||
shift 2
|
||||
;;
|
||||
-p | --package)
|
||||
PACKAGE="$2"
|
||||
shift 2
|
||||
;;
|
||||
-t | --timeout)
|
||||
STALL_TIMEOUT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--hard-timeout)
|
||||
HARD_TIMEOUT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--with-load)
|
||||
WITH_LOAD=1
|
||||
shift
|
||||
;;
|
||||
--watch)
|
||||
WATCH_MODE=1
|
||||
shift
|
||||
;;
|
||||
-o | --out)
|
||||
OUT_DIR="$2"
|
||||
shift 2
|
||||
;;
|
||||
-h | --help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
echo "Unknown option: $1" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
main
|
||||
444
linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh
Executable file
444
linux_configuration/scripts/single_use/fixes/migrate_hosts_guard_to_guard_lib.sh
Executable file
@ -0,0 +1,444 @@
|
||||
#!/bin/bash
|
||||
|
||||
# ============================================================================
|
||||
# migrate_hosts_guard_to_guard_lib.sh
|
||||
#
|
||||
# Completes the guard-lib migration that commit 66c4698 started but never
|
||||
# finished on this machine.
|
||||
#
|
||||
# State this repairs: guard-lib is installed (guardctl, lib/, systemd templates,
|
||||
# and BOTH generic pacman hooks) with /etc/guard-lib/targets completely empty,
|
||||
# while the pre-guard-lib hosts-guard implementation still does all the real
|
||||
# work through its own per-file pacman hooks and systemd units. So every pacman
|
||||
# transaction runs two PreTransaction hooks where one would do, and the
|
||||
# guard-lib pair - the one that runs FIRST, and the one present for all four
|
||||
# recorded hook stalls - does literally nothing.
|
||||
#
|
||||
# After this runs there is one guard system: guardctl file-guard instances for
|
||||
# hosts, nsswitch and resolved, driven by guard-lib's generic unlock-all /
|
||||
# relock-all hooks. The legacy hooks and units are retired.
|
||||
#
|
||||
# NOT migrated here: the "shutdown-schedule" instance that
|
||||
# setup_midnight_shutdown.sh registers. Its target /etc/shutdown-schedule.conf
|
||||
# does not exist on this machine, so registering it would mean installing a
|
||||
# different feature, not finishing this migration. Run that script if you want
|
||||
# the midnight-shutdown guard.
|
||||
#
|
||||
# Every step is idempotent, and --rollback restores the legacy layer.
|
||||
# ============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_NAME="$(basename "$0")"
|
||||
readonly SCRIPT_NAME
|
||||
readonly GUARDCTL="/usr/local/bin/guardctl"
|
||||
readonly TARGETS_DIR="/etc/guard-lib/targets"
|
||||
readonly STATE_DIR="/var/lib/guard-lib-migration"
|
||||
readonly HOOKS_DIR="/etc/pacman.d/hooks"
|
||||
# Plugins must live somewhere permanent: guardctl records an ABSOLUTE plugin
|
||||
# path in the instance conf, so pointing it at a repo checkout (or worse, a git
|
||||
# worktree) silently breaks enforcement the day that directory moves.
|
||||
readonly PLUGIN_INSTALL_DIR="/usr/local/share/guard-lib-plugins"
|
||||
|
||||
readonly LEGACY_HOOKS=(
|
||||
"10-unlock-etc-hosts.hook"
|
||||
"90-relock-etc-hosts.hook"
|
||||
)
|
||||
readonly LEGACY_UNITS=(
|
||||
"hosts-guard.path"
|
||||
"hosts-guard.service"
|
||||
"hosts-bind-mount.service"
|
||||
"nsswitch-guard.path"
|
||||
"nsswitch-guard.service"
|
||||
"resolved-guard.path"
|
||||
"resolved-guard.service"
|
||||
)
|
||||
|
||||
DRY_RUN=0
|
||||
MODE="migrate"
|
||||
|
||||
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
|
||||
readonly SCRIPT_DIR
|
||||
PLUGIN_SRC_DIR="$(readlink -f "$SCRIPT_DIR/../../periodic_background/hosts/guard/plugins")"
|
||||
readonly PLUGIN_SRC_DIR
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[0;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m'
|
||||
|
||||
msg() { printf "${GREEN}[✓]${NC} %s\n" "$*"; }
|
||||
note() { printf "${BLUE}[i]${NC} %s\n" "$*"; }
|
||||
warn() { printf "${YELLOW}[!]${NC} %s\n" "$*"; }
|
||||
err() { printf "${RED}[✗]${NC} %s\n" "$*" >&2; }
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: sudo $SCRIPT_NAME [OPTIONS]
|
||||
|
||||
Finishes the guard-lib migration for /etc/hosts, /etc/nsswitch.conf and
|
||||
/etc/systemd/resolved.conf, then retires the legacy hosts-guard layer.
|
||||
|
||||
Options:
|
||||
--rollback Undo: uninstall the guard-lib instances, restore the legacy
|
||||
pacman hooks and re-enable the legacy units.
|
||||
--status Show what is currently registered; change nothing.
|
||||
--dry-run Print what would happen; change nothing.
|
||||
-h, --help Show this help
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
run() {
|
||||
if ((DRY_RUN == 1)); then
|
||||
printf "${YELLOW}DRY-RUN:${NC} %s\n" "$*"
|
||||
return 0
|
||||
fi
|
||||
"$@"
|
||||
}
|
||||
|
||||
require_root() {
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
exec sudo -E bash "$0" "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Instance table: name | target | bind_mount | plugin basename | also_watch
|
||||
# ----------------------------------------------------------------------------
|
||||
instance_spec() { # <name> -> echoes "target|bind|plugin|also_watch"
|
||||
case "$1" in
|
||||
hosts) echo "/etc/hosts|yes||" ;;
|
||||
nsswitch) echo "/etc/nsswitch.conf|no|nsswitch-plugin.sh|" ;;
|
||||
resolved) echo "/etc/systemd/resolved.conf|no|resolved-plugin.sh|/etc/systemd/resolved.conf.d" ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
readonly INSTANCES=(hosts nsswitch resolved)
|
||||
|
||||
instance_registered() { [[ -f "$TARGETS_DIR/$1.conf" ]]; }
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Preconditions
|
||||
# ----------------------------------------------------------------------------
|
||||
validate_requirements() {
|
||||
[[ -x $GUARDCTL ]] || {
|
||||
err "guardctl not found at $GUARDCTL - run guard-lib's install.sh first"
|
||||
exit 1
|
||||
}
|
||||
|
||||
local unit
|
||||
for unit in guard-file@.path guard-file@.service guard-bind-mount@.service; do
|
||||
[[ -f "/etc/systemd/system/$unit" ]] || {
|
||||
err "missing systemd template /etc/systemd/system/$unit - run guard-lib's install.sh first"
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
[[ -d $PLUGIN_SRC_DIR ]] || {
|
||||
err "plugin sources not found at $PLUGIN_SRC_DIR"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# A live transaction would race every chattr and umount below.
|
||||
if [[ -e /var/lib/pacman/db.lck ]]; then
|
||||
err "/var/lib/pacman/db.lck exists - a pacman transaction is in flight"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Status
|
||||
# ----------------------------------------------------------------------------
|
||||
show_status() {
|
||||
printf "\n%s\n" "=== guard-lib instances ==="
|
||||
local name
|
||||
for name in "${INSTANCES[@]}"; do
|
||||
if instance_registered "$name"; then
|
||||
msg "$name registered"
|
||||
"$GUARDCTL" file-guard status "$name" 2>&1 | sed 's/^/ /'
|
||||
else
|
||||
warn "$name NOT registered"
|
||||
fi
|
||||
done
|
||||
|
||||
printf "\n%s\n" "=== legacy pacman hooks ==="
|
||||
local hook
|
||||
for hook in "${LEGACY_HOOKS[@]}"; do
|
||||
if [[ -f "$HOOKS_DIR/$hook" ]]; then
|
||||
warn "$hook still present"
|
||||
else
|
||||
msg "$hook retired"
|
||||
fi
|
||||
done
|
||||
|
||||
printf "\n%s\n" "=== legacy systemd units ==="
|
||||
local unit state
|
||||
for unit in "${LEGACY_UNITS[@]}"; do
|
||||
# `is-enabled` PRINTS the state but EXITS 1 for anything not enabled, so
|
||||
# `|| echo absent` would append a second line to a perfectly good answer.
|
||||
state="$(systemctl is-enabled "$unit" 2>/dev/null)" || true
|
||||
[[ -n $state ]] || state="absent"
|
||||
if [[ $state == "enabled" ]]; then
|
||||
warn "$unit is $state"
|
||||
else
|
||||
msg "$unit is $state"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Rollback state
|
||||
# ----------------------------------------------------------------------------
|
||||
save_rollback_state() {
|
||||
run mkdir -p "$STATE_DIR/hooks"
|
||||
|
||||
local hook
|
||||
for hook in "${LEGACY_HOOKS[@]}"; do
|
||||
if [[ -f "$HOOKS_DIR/$hook" && ! -f "$STATE_DIR/hooks/$hook" ]]; then
|
||||
run cp -a "$HOOKS_DIR/$hook" "$STATE_DIR/hooks/$hook"
|
||||
note "backed up $hook"
|
||||
fi
|
||||
done
|
||||
|
||||
# Record enablement so rollback restores the exact prior state rather than
|
||||
# guessing (resolved-guard.* were already disabled here, for instance).
|
||||
if [[ ! -f "$STATE_DIR/units.state" ]] && ((DRY_RUN == 0)); then
|
||||
local unit ustate
|
||||
for unit in "${LEGACY_UNITS[@]}"; do
|
||||
ustate="$(systemctl is-enabled "$unit" 2>/dev/null)" || true
|
||||
[[ -n $ustate ]] || ustate="absent"
|
||||
printf '%s=%s\n' "$unit" "$ustate"
|
||||
done >"$STATE_DIR/units.state"
|
||||
note "recorded legacy unit states in $STATE_DIR/units.state"
|
||||
fi
|
||||
}
|
||||
|
||||
install_plugins() {
|
||||
run mkdir -p "$PLUGIN_INSTALL_DIR"
|
||||
local src
|
||||
for src in "$PLUGIN_SRC_DIR"/*.sh; do
|
||||
[[ -e $src ]] || continue
|
||||
run install -m 755 "$src" "$PLUGIN_INSTALL_DIR/$(basename "$src")"
|
||||
done
|
||||
msg "plugins installed to $PLUGIN_INSTALL_DIR"
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Migration
|
||||
# ----------------------------------------------------------------------------
|
||||
stop_legacy_units_for() { # <instance name>
|
||||
local prefix="$1"
|
||||
local unit
|
||||
for unit in "${LEGACY_UNITS[@]}"; do
|
||||
[[ $unit == "$prefix"-* ]] || continue
|
||||
systemctl list-unit-files "$unit" &>/dev/null || continue
|
||||
run systemctl disable --now "$unit" 2>/dev/null || true
|
||||
done
|
||||
}
|
||||
|
||||
# Unmount every stacked bind layer so chattr can reach the real inode. Mirrors
|
||||
# the legacy pacman-pre-unlock-hosts.sh, which had to do exactly this.
|
||||
# Every branch is a full `if` and the function ends in an explicit `return 0`.
|
||||
# The obvious `((i > 20)) && break` form leaves the loop body's exit status at 1
|
||||
# whenever the arithmetic is false, and under `set -e` that aborted the whole
|
||||
# rollback silently — but ONLY when the loop actually ran, so --dry-run and an
|
||||
# already-unmounted target both looked fine. It cost a live /etc/hosts left with
|
||||
# neither the immutable flag nor its read-only mount to notice.
|
||||
collapse_mounts() { # <path>
|
||||
local target="$1" i=0
|
||||
while mountpoint -q "$target"; do
|
||||
if ! run umount -l "$target" 2>/dev/null; then
|
||||
break
|
||||
fi
|
||||
i=$((i + 1))
|
||||
if ((i > 20)); then
|
||||
break
|
||||
fi
|
||||
# Nothing is really unmounted under --dry-run, so `mountpoint` stays
|
||||
# true forever; report the intent once instead of 20 identical lines.
|
||||
if ((DRY_RUN == 1)); then
|
||||
break
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
migrate_instance() { # <name>
|
||||
local name="$1"
|
||||
local spec target bind plugin also_watch
|
||||
spec="$(instance_spec "$name")"
|
||||
IFS='|' read -r target bind plugin also_watch <<<"$spec"
|
||||
|
||||
if instance_registered "$name"; then
|
||||
msg "$name already registered - skipping"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ ! -e $target ]]; then
|
||||
warn "$name: target $target does not exist - skipping"
|
||||
return 0
|
||||
fi
|
||||
|
||||
note "migrating $name ($target)"
|
||||
|
||||
stop_legacy_units_for "$name"
|
||||
collapse_mounts "$target"
|
||||
run chattr -i "$target" 2>/dev/null || true
|
||||
|
||||
local -a args=(file-guard install "$name" --target "$target")
|
||||
[[ $bind == "yes" ]] && args+=(--bind-mount)
|
||||
[[ -n $plugin ]] && args+=(--plugin "$PLUGIN_INSTALL_DIR/$plugin")
|
||||
[[ -n $also_watch ]] && args+=(--also-watch "$also_watch")
|
||||
|
||||
run "$GUARDCTL" "${args[@]}"
|
||||
msg "$name migrated"
|
||||
}
|
||||
|
||||
retire_legacy_hooks() {
|
||||
local hook removed=0
|
||||
for hook in "${LEGACY_HOOKS[@]}"; do
|
||||
if [[ -f "$HOOKS_DIR/$hook" ]]; then
|
||||
run rm -f "$HOOKS_DIR/$hook"
|
||||
msg "retired $hook"
|
||||
removed=1
|
||||
fi
|
||||
done
|
||||
((removed == 0)) && msg "legacy pacman hooks already retired"
|
||||
return 0
|
||||
}
|
||||
|
||||
do_migrate() {
|
||||
validate_requirements
|
||||
save_rollback_state
|
||||
install_plugins
|
||||
|
||||
local name
|
||||
for name in "${INSTANCES[@]}"; do
|
||||
migrate_instance "$name"
|
||||
done
|
||||
|
||||
# Only retire the legacy hooks once guard-lib actually owns something -
|
||||
# otherwise a failed migration would leave the files with NO pacman
|
||||
# unlock hook at all, and the next transaction would fight chattr +i.
|
||||
local registered=0
|
||||
for name in "${INSTANCES[@]}"; do
|
||||
instance_registered "$name" && registered=$((registered + 1))
|
||||
done
|
||||
|
||||
if ((registered == 0)) && ((DRY_RUN == 0)); then
|
||||
err "no guard-lib instance registered - refusing to retire the legacy hooks"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
retire_legacy_hooks
|
||||
|
||||
printf "\n"
|
||||
msg "migration complete ($registered instance(s) registered)"
|
||||
note "verify with: sudo $SCRIPT_NAME --status"
|
||||
note "undo with: sudo $SCRIPT_NAME --rollback"
|
||||
}
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# Rollback
|
||||
# ----------------------------------------------------------------------------
|
||||
do_rollback() {
|
||||
[[ -d $STATE_DIR ]] || {
|
||||
err "no rollback state at $STATE_DIR - nothing to roll back to"
|
||||
exit 1
|
||||
}
|
||||
|
||||
local name
|
||||
for name in "${INSTANCES[@]}"; do
|
||||
if instance_registered "$name"; then
|
||||
# Keep the canonical: it is the known-good copy of the file, and
|
||||
# the legacy layer re-snapshots from the live file, which may have
|
||||
# drifted. Losing it would turn a rollback into data loss.
|
||||
run "$GUARDCTL" file-guard uninstall "$name" --keep-canonical || true
|
||||
msg "uninstalled guard-lib instance $name"
|
||||
fi
|
||||
done
|
||||
|
||||
local hook
|
||||
for hook in "${LEGACY_HOOKS[@]}"; do
|
||||
if [[ -f "$STATE_DIR/hooks/$hook" ]]; then
|
||||
run cp -a "$STATE_DIR/hooks/$hook" "$HOOKS_DIR/$hook"
|
||||
msg "restored $hook"
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -f "$STATE_DIR/units.state" ]]; then
|
||||
local unit state
|
||||
while IFS='=' read -r unit state; do
|
||||
[[ $state == "enabled" ]] || continue
|
||||
run systemctl enable --now "$unit" 2>/dev/null || warn "could not re-enable $unit"
|
||||
msg "re-enabled $unit"
|
||||
done <"$STATE_DIR/units.state"
|
||||
else
|
||||
warn "no units.state recorded - legacy units left as-is"
|
||||
fi
|
||||
|
||||
# Re-assert the legacy protections NOW rather than waiting for a watcher.
|
||||
# `guardctl file-guard uninstall` chattr -i's the target on its way out, and
|
||||
# the legacy *-guard.path units only fire on PathModified - so without this
|
||||
# a rolled-back /etc/hosts sits with NO immutable flag until something
|
||||
# happens to write to it. Caught by actually running --rollback and looking
|
||||
# at lsattr, which is the only reason it is handled at all.
|
||||
#
|
||||
# Order matters for hosts specifically: it is bind-mounted read-only, and
|
||||
# chattr cannot write through a ro mount. enforce-hosts.sh does a bare
|
||||
# `chattr +i` with no collapse (it relies on running before the mount at
|
||||
# boot), so it fails silently here unless the mount is dropped first. So:
|
||||
# collapse, enforce, then let the bind-mount unit rebuild the ro layer.
|
||||
collapse_mounts /etc/hosts
|
||||
|
||||
local svc
|
||||
for svc in hosts-guard.service nsswitch-guard.service resolved-guard.service; do
|
||||
systemctl cat "$svc" &>/dev/null || continue
|
||||
run systemctl start "$svc" 2>/dev/null || warn "could not run $svc"
|
||||
done
|
||||
|
||||
if systemctl cat hosts-bind-mount.service &>/dev/null; then
|
||||
run systemctl restart hosts-bind-mount.service 2>/dev/null || warn "could not restart hosts-bind-mount.service"
|
||||
fi
|
||||
msg "re-ran legacy enforcement (restores chattr +i and the ro bind mount)"
|
||||
|
||||
printf "\n"
|
||||
msg "rollback complete"
|
||||
note "verify with: sudo $SCRIPT_NAME --status"
|
||||
}
|
||||
|
||||
# Parse arguments
|
||||
require_root "$@"
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--rollback)
|
||||
MODE="rollback"
|
||||
shift
|
||||
;;
|
||||
--status)
|
||||
MODE="status"
|
||||
shift
|
||||
;;
|
||||
--dry-run)
|
||||
DRY_RUN=1
|
||||
shift
|
||||
;;
|
||||
-h | --help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
err "Unknown option: $1"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
case "$MODE" in
|
||||
migrate) do_migrate ;;
|
||||
rollback) do_rollback ;;
|
||||
status) show_status ;;
|
||||
esac
|
||||
31
meta/scripts/run_with_heavy_lock.sh
Executable file
31
meta/scripts/run_with_heavy_lock.sh
Executable file
@ -0,0 +1,31 @@
|
||||
#!/bin/bash
|
||||
# Run a command under this machine's heavy-job lock, if it has one.
|
||||
#
|
||||
# Front-end for .pre-commit-config.yaml so the config stays portable: the lock
|
||||
# lives at a machine-specific absolute path (installed by
|
||||
# install_pacman_wrapper.sh), and hardcoding that into a tracked config would
|
||||
# break the hook on any clone that has not installed it. Missing lock => run the
|
||||
# command unserialised, exactly as before.
|
||||
#
|
||||
# Usage: run_with_heavy_lock.sh <lock-name> <command> [args...]
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
readonly HEAVY_JOB_LOCK_LIB="/usr/local/bin/heavy_job_lock.sh"
|
||||
|
||||
if [[ $# -lt 2 ]]; then
|
||||
echo "usage: $(basename "$0") <lock-name> <command> [args...]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
lock_name="$1"
|
||||
shift
|
||||
|
||||
if [[ -r $HEAVY_JOB_LOCK_LIB ]]; then
|
||||
# shellcheck source=/dev/null
|
||||
source "$HEAVY_JOB_LOCK_LIB"
|
||||
with_heavy_lock "$lock_name" -- "$@"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
exec "$@"
|
||||
Loading…
Reference in New Issue
Block a user