diff --git a/NEXT_SESSION_PROMPT.md b/NEXT_SESSION_PROMPT.md index a345aa11..1fdc78b0 100644 --- a/NEXT_SESSION_PROMPT.md +++ b/NEXT_SESSION_PROMPT.md @@ -1,15 +1,15 @@ -# Next session: §3's next restructure — diagnose_pacman_hook_stall.sh +# Next session: §3's next restructure — setup_hosts_guard.sh > **Paste this whole file into a fresh Claude session opened in `~/testsAndMisc`.** > It is self-contained. Do not go looking for the previous session's context. -Over-cap: **13** (11 shell, 2 kotlin, 1 dart). Working tree clean, `main` in -sync as of the libre_translate.sh archival commit (see §0.5). +Over-cap: **12** (11 shell, 2 kotlin, 1 dart). Working tree clean, `main` in +sync as of the diagnose_pacman_hook_stall.sh split commit (see §0.5). Two standing decisions from the user, already made — do **not** re-litigate: -1. **100% line coverage on what you extract** — same bar as the previous - session applied to `install_plagiarism_tools.sh`. +1. **100% line coverage on what you extract** — same bar every session so far + has applied. 2. **Split blind where there is no device.** Do not deploy to the phone, do not ask to. Prove splits with tests, hashes and real runs instead. @@ -17,67 +17,93 @@ Two standing decisions from the user, already made — do **not** re-litigate: ## 0. What just happened (read this, don't repeat it) -The previous session's brief listed three §3 restructures. The first, -`install_plagiarism_tools.sh`, was split (534→94 lines + 3 libs, 100% -coverage via a PATH-shim test harness, verified against a real venv with -actual nltk/scikit-learn) — then the user said they no longer use the -plagiarism tools at all. The whole thing (install script + 2 extracted Python -files + 3 libs + 5 test files, 13 files total) was moved to -`kuhyx/testsAndMisc-archive` with full git history preserved via -`git filter-repo`, and deleted from this repo. See commits `2a7b9d9` (the -split) and `071d706` (the archival) for the full mechanics if you need the -`git filter-repo` recipe again for something else — it is not expected to -recur this session. +Two sessions ago: `install_plagiarism_tools.sh` was split (534→94 lines + +3 libs, 100% coverage) — then the user said they no longer use the +plagiarism tools at all, so the whole thing was archived to +`kuhyx/testsAndMisc-archive` with full git history via `git filter-repo`. +One session ago: the same one-line "do you still use this?" check-in was +asked about `libre_translate.sh` _before_ any code was touched, and got the +same answer — also archived, not split. -**Lesson for this session:** before sinking real effort into a split, it may -be worth a one-line check-in — "do you still use this?" — for anything that -looks like it could be dead code, especially in `single_use/` directories. -Not a blocking gate, just don't be surprised if the answer changes scope. +**Lesson (2 for 2, now reinforced a third way):** keep asking the check-in +before every future §3/§4 split. It is not a formality; it has changed scope +in both of the two times it's been tried. -## 0.5. What just happened (this session) — libre_translate.sh archived, not split +## 0.5. What just happened (this session) — diagnose_pacman_hook_stall.sh split, confirmed live -Asked the same one-line check-in the §0 lesson above recommends, _before_ -starting the split (grill-gate questions, before any code was touched this -time). User confirmed `libre_translate.sh` is also unused. Same resolution as -the plagiarism tools: moved to `kuhyx/testsAndMisc-archive` with full git -history via `git filter-repo`, and deleted from this repo. See the evidence -file `docs/superpowers/evidence/archive-libre-translate-2026-08-18.json` for -the exact recipe — this file's history was more tangled than the plagiarism -tools' (two independent pre-reorg root paths, `libre_translate.sh` and -`Bash/libre_translate.sh`, both genuine ancestors via a subtree-import merge) -so both had to be passed to `--path` or history would have been silently -truncated. +Asked the dead-code check-in first, per §0's lesson. User confirmed this +script **is** still in active use (unlike the last two candidates) — so the +split proceeded as originally planned. 493 lines → 249 (entry) + 6 libs +(pacman_hook_stall_setup.sh, _load.sh, _capture.sh, _watch.sh, _usage.sh, +_summary.sh), all at 100% line coverage, plus a PATH-shim test suite (56 +lib-level + 13 entry-level subprocess assertions, 0 failed). -**Lesson reinforced:** the §0 check-in is now 2 for 2 on `single_use/` files -turning out to be dead code. Keep asking it before every future §3/§4 split, -not just when something "looks like" it could be unused. +**Two things worth knowing before touching this area again:** -## 1. This session's task: `diagnose_pacman_hook_stall.sh` +1. `run_one`/`cleanup`/`log_size`/`main` had to stay together in the entry + script — they share `PACMAN_PID`/`LAST_ELAPSED` across the `EXIT` trap, + and a prior session (commit `2527fb3`) already proved that splitting + `run_one` into a lib produces a **real** SC2153/SC2034 pair (not a + suppressible false positive), because a sourced lib has no way to declare + where its globals come from. If a future split looks like it wants to + move a function that both writes a global the entry script reads AND + reads a global only the entry script sets, do the same scratch + shellcheck-dry-run check (with `.shellcheckrc`'s `external-sources=true` / + `source-path=SCRIPTDIR`, which is what the real hook uses) **before** + committing to a lib boundary, not after. +2. **kcov cannot see `done `, `} >redirect`, or lines inside an + inline awk script**, no matter how thoroughly the code executes (verified + empirically with `bash -x` on isolated repros — those lines never emit a + trace event). If a coverage gap on a future file's line list looks + suspiciously like a loop-closing or group-closing line, or an awk block, + check whether it's this exact class before assuming the test is weak. The + user was asked and chose "refactor the code" over "fix the coverage + tool" — that decision holds for this repo going forward unless + re-raised. +3. **Test files are not exempt from the 250-line cap.** The entry-script + subprocess test grew to 295 lines while adding coverage and had to be + split itself, mid-session, after `check_file_length.sh --all` caught it. + Budget for this: a test file that drives real subprocess timing (unshare, + real sleep, multiple flag combinations) grows fast — split its fixture + setup into a shared harness early rather than after it's already over. +4. Evidence + contract: + `docs/superpowers/evidence/split-diagnose-pacman-hook-stall-2026-08-18.json` + and the matching `contracts/` file are the current format reference — + more detailed than the plagiarism-tools pair since this split involved + several genuine (not mechanical) function rewrites, all individually + justified there. -Location: `linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh` -(493 lines, 15 functions). +## 1. This session's task: `setup_hosts_guard.sh` -**Ask the dead-code check-in first** (see §0.5) — do not assume this one is -live just because the first two candidates weren't. +Location: +`linux_configuration/scripts/periodic_background/hosts/guard/setup_hosts_guard.sh` +(576 lines). -If it's confirmed live: `run_one` writes `LAST_ELAPSED`, `main` reads it — -that global crosses the split seam deliberately (see the SC2034 rule in §5 -below), so it must stay in whichever file both functions land in, or get -passed explicitly. Emits **SC2153** (`PACMAN_BIN` vs `PACMAN_PID`) once -split — a real finding to resolve, never to suppress. +**Ask the dead-code check-in first** (see §0.5) — the streak is 2-for-3 so +far (2 archived, 1 confirmed live); don't assume either outcome. **Before you start**, confirm scope hasn't drifted: `git status --short` and -`wc -l` on the target file, same as the check that caught both prior files -were unused — it's cheap insurance. +`wc -l` on the target file, and re-run +`bash meta/scripts/check_file_length.sh --all` to get the current exact +list — files change size between sessions, and this file's line count may +not still be 576. + +Check what this file references before planning a split boundary — +`hosts_guard_migrate.sh`/`hosts_guard_rollback.sh` already exist as siblings +in `linux_configuration/scripts/single_use/fixes/lib/`, and +`fixes/migrate_hosts_guard_to_guard_lib.sh` is the entry script that +migrated _to_ this guard-lib system. Read that migration script's header +comment first — it explains what guard-lib actually does and may reveal +whether `setup_hosts_guard.sh` itself is now redundant with the migrated +state, which is exactly the kind of thing the dead-code check-in is for. ## 3. §4 — do not touch this session `install_leechblock.sh` (485) and `block_compulsive_opening.sh` (705) are copied to `/usr/local/…`, and `pacman_wrapper.sh:831` prefers the deployed copy on **every pacman invocation**. Splitting them naively breaks every -`pacman -S` on this machine. Out of scope until `diagnose_pacman_hook_stall.sh` -is done (or archived — see §0.5) and the user explicitly says to continue -into §4. +`pacman -S` on this machine. Out of scope until `setup_hosts_guard.sh` is +done (or archived) and the user explicitly says to continue into §4. You cannot run `sudo pacman -S` from the Bash tool — it deadlocks on `db.lck`. Hand the user a `! sudo pacman -S ` line plus the expected @@ -88,10 +114,10 @@ output if `pacman_wrapper.sh` ever needs live verification. `setup_midnight_shutdown.sh` (1734), `check_and_enable_services.sh` (1301), `generate_study_materials.sh` (1017), `pacman_wrapper.sh` (929), `setup_night_lockdown.sh` (918), `hosts/install.sh` (912), -`setup_hosts_guard.sh` (576), `EnforcementRunner.kt` (564), -`DevicePolicyBridge.kt` (415), `status_page_state.dart` (307) — check -`check_file_length.sh --all` for the current exact list before assuming this -is still accurate, since files change size between sessions. +`EnforcementRunner.kt` (564), `DevicePolicyBridge.kt` (415), +`status_page_state.dart` (307) — check `check_file_length.sh --all` for the +current exact list before assuming this is still accurate, since files +change size between sessions. `pacman_wrapper.sh` carries the same live-deployment trap as §3/§4. The Kotlin and Dart files need a different verification stack — `focus_owner` @@ -100,27 +126,34 @@ plain `gradlew test` reports `UP-TO-DATE` and proves nothing. ## 5. Rules that will bite you (carried forward, still true) -- **Ask before large scope changes.** The plagiarism-tools split turned into - an archive-and-delete mid-session because the user volunteered new - information partway through. That is fine and expected — just don't assume - silence means "keep going as originally scoped" if something about the - target file's relevance seems worth a quick check. +- **Ask before large scope changes.** Two of the last three sessions turned + into archive-and-delete mid-session because the user volunteered new + information partway through (or, this session, confirmed the file WAS + still live). That is fine and expected — just don't assume silence means + "keep going as originally scoped" if something about the target file's + relevance seems worth a quick check. - **No suppressions, ever.** No `# noqa`, no per-file-ignore added without asking every time, no shellcheck disable beyond what's already justified - inline. If you hit a rule (T201, ANN401, PLC0415, etc.) that seems to - fundamentally conflict with the file's purpose, ask — don't suppress. Last - session this came up for `print()` vs ruff's T201 in extracted Python - files; the fix was converting to `sys.stdout.write()`, not a per-file-ignore. + inline. If a coverage or lint finding looks like it fundamentally + conflicts with the file's purpose, ask before working around it — this + session's kcov-blind-spot finding (§0.5.2) is the exact pattern: verify + the finding is real (bash -x trace check) before assuming it's a tool + limitation to route around vs. a real behavioral gap to test. - **A file must not assign a global it never reads** (SC2034). The hook runs - `shellcheck` with **no `-x`**, so each file stands alone. Constants travel - with their readers; a global genuinely written on both sides of a seam - stays in the entry script. -- **New sourced libs need a shebang AND the executable bit.** The hook reads - the **git index**: stage with `git add --chmod=+x`, and note that a plain - `git add` afterwards resets it. This cost real failed pre-commit runs last - session too — `pre-commit run --files` before `git add` does NOT catch a - missing index executable bit; only running it against the actually-staged - set does. + `shellcheck` with **no `-x`**, but `.shellcheckrc`'s + `external-sources=true` + `source-path=SCRIPTDIR` means a `source=` + annotation on the _sourcing_ file still resolves correctly — the failure + mode that actually breaks is a _lib_ file trying to reference a global + only the entry script sets, since a lib has no annotation pointing + "upward" to whoever sources it. Constants travel with their readers; a + global genuinely written on both sides of a seam stays in the entry + script. +- **New sourced libs need a shebang AND the executable bit**, even though + they're sourced, not executed — `lib/hosts_guard_migrate.sh` and this + session's 6 new libs both carry `100755` in the git index, matching + convention. The hook reads the **git index**: stage with + `git add --chmod=+x`, and note that a plain `git add` afterwards resets + it. - `pre-commit run --files ` before committing, but run it **after** `git add`, not before — index-mode checks (executable bit) only fire against what's staged. **`prettier` and `ci-mirror` run on pre-push.** @@ -129,57 +162,75 @@ plain `gradlew test` reports `UP-TO-DATE` and proves nothing. - **Every commit touching code needs evidence** in `docs/superpowers/evidence/-.json`. Staging **≥4 code files also needs** a fresh `docs/superpowers/contracts/*.json`. Put the - **measured** coverage number in it, not a rounded-up one. Two such pairs - already exist from last session as format references: - `install-plagiarism-tools-split-2026-08-18.json` and - `archive-plagiarism-tools-2026-08-18.json` (evidence + contracts). + **measured** coverage number in it, not a rounded-up one. This session's + pair (`split-diagnose-pacman-hook-stall-2026-08-18.json` in both + `evidence/` and `contracts/`) is the current best format reference — + it documents several genuine mid-session findings (a real bug, a + memory-safety catch, a test-file-too-long split) as they happened, which + is the level of detail to aim for when the session isn't a pure mechanical + move. - Work directly on `main`. `git stash` and branch creation are blocked; use `git worktree add --detach` for a clean baseline (but NEVER `git filter-repo` inside a worktree — it shares the object store with the main repo and will - corrupt it; always a throwaway `git clone` for that, which shouldn't be - needed this session). + corrupt it; always a throwaway `git clone` for that). - **Do not wire the file-length hook into pre-commit.** It lands last, once `check_file_length.sh --all` exits 0. -- Watch `jscpd` (fails above 2%). Measure in a **clean HEAD worktree** — the - working tree reads higher because of vendored `.venv`. +- Watch `jscpd` (fails above 2%). Measure in a **clean HEAD worktree** + (`git worktree add --detach HEAD`, copy the new files in, run jscpd + there, remove the worktree) — the working tree reads higher because of + vendored `.venv`, and several near-identical test files sharing a fixture + preamble is exactly the shape that trips this gate. - Cap pytest memory: `systemd-run --user --scope -p MemorySwapMax=0 -p MemoryMax=2G`. +- **Never let a test allocate unbounded real memory.** This session's + `--with-load` entry test would have `dd`ed 13-20+ GB into `/dev/shm` using + this box's real available memory before it was caught in review — any + script under test that computes an allocation size from real + `/proc/meminfo` needs its floor/ceiling constants made env-overridable + (`: "${VAR:=default}"`, not a bare or `readonly` assignment) so tests can + pin them to a small, bounded amount instead of trusting the live system + state to stay favorable. ## 6. Tooling reference **`meta/scripts/check_file_length.sh --all`** — the 250-line gate. Run -**before every commit**. +**before every commit**. Applies to test files too, not just the file +you're splitting. **`meta/scripts/shell_coverage.sh [min]`** — kcov wrapper, 100% minimum. Hand it the test script **directly**; `bash script.sh` -instruments the bash binary and silently reports 0/0. +instruments the bash binary and silently reports 0/0. If a test suite for +one subject is split across multiple test files, `shell_coverage.sh` only +takes one at a time — run kcov manually into the same output dir for each +test file, then call `meta/scripts/shell_coverage_report.py ` +once to get the merged number (it sums hits per line-number across every +`cov.xml` it finds, so multiple runs against the same subject combine +correctly). **`meta/scripts/extract_shell_functions.py`** — moves functions -brace-by-brace. **Never slice by line range**: it cut through a multi-line -quoted string in a past session and left an unterminated quote that a -regex-based loader tolerated while the shell could no longer source the file. -**Do not trust where it puts the source line** — verify with `grep -c` on the -anchor and check placement; it got this wrong three times in a past session -(before a variable existed, before another variable existed so paths expanded -against an empty prefix, and on a nested re-source rather than the top-level -one). +brace-by-brace. **Never slice by line range**. **Do not trust where it puts +the source line** — verify with `grep -c` on the anchor and check placement. +The `--header` text is written **verbatim, uncommented** — pass a full +`#!/bin/bash\n# comment` string, not bare prose, or the generated lib won't +parse. **`meta/scripts/verify_shell_split.sh ...`** — proves a split moved every function verbatim (hashes function bodies only, blind to top-level code). For a **partial** split list the old path among the -new paths too, or it reports a false `DIFFERENCE`. +new paths too, or it reports a false `DIFFERENCE`. A function that was +genuinely rewritten (not moved) will legitimately show as `DIFFERENCE` — +that's expected, not a bug; explain it in evidence rather than trying to make +the tool agree. **`meta/scripts/mutate_shell.py `** — mutation testing against specs in -`meta/scripts/fixtures/mutations/`. No spec currently exists for -`diagnose_pacman_hook_stall.sh` or anything outside `phone_focus_mode/`; -creating one is optional, not required by the brief — 100% line coverage with -real assertions is the stated bar, mutation testing is what `phone_focus_mode` -additionally did. +`meta/scripts/fixtures/mutations/`. No spec exists for anything outside +`phone_focus_mode/`; optional, not required — 100% line coverage with real +assertions is the stated bar. ## 7. Testing pattern to follow -Last session's plagiarism-tools split established a working pattern for -`single_use/` shell scripts with no prior test coverage: +The established pattern for `single_use/`-adjacent shell scripts with no +prior test coverage (now used across three splits): - A `lib/tests/` directory sibling to the split libs. - A `*_harness.sh` file (sourced, not executed) with `_t_pass`/`_t_fail`/ @@ -190,18 +241,30 @@ Last session's plagiarism-tools split established a working pattern for function calls (not mocks at the call-site level). - A `test_.sh` that runs the **actual entry script as a subprocess** against the same fakes, since `verify_shell_split.sh` and kcov - are blind to code that only lives in the entry script's `main()`. -- Two sharp edges hit and fixed last session, watch for both again: + are blind to code that only lives in the entry script's `main()`. If this + test file threatens to grow past 250 lines (it will, once you're covering + several arg combinations and timing branches), split the fixture setup + into its own `*_entry_harness.sh` early and write two or more thinner test + files against it — this session had to do this mid-stream and it's + cheaper to plan for upfront. +- Sharp edges hit and fixed across sessions, watch for all of them again: - `local a="$1" b="${a}"` **fails under `set -u`** — bash evaluates all - RHS expressions in the new scope before any assignment lands, so `$a` is - unbound when `$b`'s RHS is evaluated. Split into separate `local` - statements. + RHS expressions in the new scope before any assignment lands. Split into + separate `local` statements. - A function that calls `exit` (not `return`) on a fatal condition will - kill the whole test script if called directly inside an `if` condition — - `exit` always terminates the process regardless of `if`/`&&` context. Run - it in a subshell: `if (fn args); then ...`. + kill the whole test script if called directly inside an `if` condition. + Run it in a subshell: `if (fn args); then ...`. + - A backgrounded process needs `disown` before `kill`/`wait` reliably + affects only it, not the calling job — this bit both a `stop_load` test + and the entry script's SIGTERM test this session. + - **SIGINT delivery to a backgrounded job was unreliable in this + environment; SIGTERM was not.** If a script's trap handles both signals + identically (`trap handler EXIT INT TERM`), prefer testing via SIGTERM. + - If a script under test computes a real resource allocation (memory, + disk) from live system state, make the relevant constants + env-overridable and pin them to something tiny in every test — never + trust "the box currently has enough headroom" to stay true. -Check `diagnose_pacman_hook_stall.sh` for anything that shells out (pacman? -systemctl? journalctl?) before assuming the same fakes apply — it's a -different domain from the plagiarism installer, so the PATH-shim list will -differ. +Check the target file for anything that shells out (pacman? systemctl? +journalctl? guardctl?) before assuming a prior session's fakes apply — each +file is a different domain, so the PATH-shim list will differ. diff --git a/docs/superpowers/contracts/split-diagnose-pacman-hook-stall-2026-08-18.json b/docs/superpowers/contracts/split-diagnose-pacman-hook-stall-2026-08-18.json new file mode 100644 index 00000000..e4173ccc --- /dev/null +++ b/docs/superpowers/contracts/split-diagnose-pacman-hook-stall-2026-08-18.json @@ -0,0 +1,22 @@ +{ + "title": "Split diagnose_pacman_hook_stall.sh, this session's NEXT_SESSION_PROMPT.md §1 target", + "objective": "Take diagnose_pacman_hook_stall.sh from 493 lines to under the repo's 250-line cap, confirmed live via the mandatory dead-code check-in before any code was touched. Split into 6 sourced libs by function group, keeping run_one/cleanup/log_size/main together in the entry script because they share PACMAN_PID/LAST_ELAPSED across the EXIT trap -- a prior session (commit 2527fb3) already demonstrated splitting run_one out produces a real SC2153/SC2034 pair, not a suppressible false positive. Reach 100% line coverage on every one of the 7 resulting files via a PATH-shim test harness, refactoring specific lines only where kcov's xtrace-based instrumentation cannot see them (done-with-redirect / group-with-redirect / inline-awk), per the user's explicit choice to fix the code rather than the coverage tool.", + "acceptance_criteria": [ + "check_file_length.sh --all no longer lists diagnose_pacman_hook_stall.sh (12 files remain, down from 13); the entry script itself and all 6 libs are at or under 250 lines.", + "verify_shell_split.sh against the pre-split revision (79829ec) reports DIFFERENCE only for functions that were deliberately rewritten (capture_stall, descendant_pids, kill_tree, main, mem_available_mb, print_summary, validate_requirements) with each rewrite's reason documented in evidence; every other function moves verbatim.", + "Every one of the 6 new libs and the entry script reaches 100% line coverage via meta/scripts/shell_coverage.sh, with no suppression of any shellcheck or coverage finding -- genuine gaps get either a new test case or a documented, reviewed code change, never a # noqa/disable equivalent.", + "descendant_pids' array-mutation semantics (pids+=(\"$child\") surviving across loop iterations in the current shell) are preserved exactly -- verified by an existing multi-level-tree test passing unchanged before and after any refactor of that function.", + "shellcheck passes with zero findings across all 17 changed files run together (matching the pre-commit hook's actual batched invocation), relying on .shellcheckrc's external-sources=true + source-path=SCRIPTDIR to resolve cross-file globals correctly rather than any -x flag or suppression.", + "Every test file, including the ones this session writes, also stays under the 250-line cap -- the entry-script subprocess test grew past it while adding coverage and had to be split into a shared fixture (pacman_hook_stall_entry_harness.sh) plus two test files, re-verified for identical coverage/assertion count and no new jscpd duplication after the split.", + "jscpd stays under the repo's 2% duplication threshold, measured in a clean git worktree with the new files copied in (not the live working tree, which reads high off vendored .venv).", + "No test allocates unbounded real memory or otherwise touches the live machine's actual pacman state -- PACMAN_BIN/PACMAN_LOG/CACHE_DIR/PACMAN_LOCK/LOAD_FLOOR_MB/LOAD_MIN_FREE_MB are all env-overridable and every test points them at a throwaway tmpdir and a small, bounded allocation.", + "New sourced libs carry a shebang and the git-index executable bit (git add --chmod=+x), matching the existing lib/hosts_guard_migrate.sh convention in the same directory." + ], + "out_of_scope": [ + "§3 (install_leechblock.sh / block_compulsive_opening.sh) -- explicitly deferred in NEXT_SESSION_PROMPT.md due to the deployed-/usr/local/-copy trap that would break every pacman invocation on this machine if split naively.", + "Any other file listed in NEXT_SESSION_PROMPT.md §4 as out of scope until the user says otherwise (setup_midnight_shutdown.sh, check_and_enable_services.sh, pacman_wrapper.sh, the Kotlin/Dart files, etc).", + "Running the script against a real pacman transaction on this machine -- only the PATH-shim harness and a fake-but-real pacman.orig subprocess were exercised.", + "Hardening require_root's sudo -E environment-preservation against a hypothetical future NOPASSWD sudoers rule -- noted as a risk in evidence, not acted on, since no such rule exists today." + ], + "verifier": "bash meta/scripts/check_file_length.sh --all; bash meta/scripts/verify_shell_split.sh 79829ec <6 libs>; bash meta/scripts/shell_coverage.sh for each of the 7 subjects; shellcheck on all 17 changed files together; jscpd in a clean detached worktree; pre-commit run --files on the full staged set." +} diff --git a/docs/superpowers/evidence/split-diagnose-pacman-hook-stall-2026-08-18.json b/docs/superpowers/evidence/split-diagnose-pacman-hook-stall-2026-08-18.json new file mode 100644 index 00000000..388a25e3 --- /dev/null +++ b/docs/superpowers/evidence/split-diagnose-pacman-hook-stall-2026-08-18.json @@ -0,0 +1,79 @@ +{ + "intent": "Take diagnose_pacman_hook_stall.sh from 493 lines to under the repo's 250-line cap, per NEXT_SESSION_PROMPT.md §1 -- this session's target, confirmed live (not dead code) via the required dead-code check-in before any code was touched.", + "scope": [ + "linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh (entry script, kept: usage/cleanup/main() minus its extracted print_summary tail, log_size, run_one, arg parsing -- these stay together because run_one shares PACMAN_PID/LAST_ELAPSED with cleanup()/main() across the EXIT trap)", + "linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_setup.sh (new: require_root, validate_requirements, resolve_package_file)", + "linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_load.sh (new: mem_available_mb, start_load, stop_load)", + "linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_capture.sh (new: descendant_pids, kill_tree, capture_stall)", + "linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_watch.sh (new: watch_forever)", + "linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_usage.sh (new: usage)", + "linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_summary.sh (new: print_summary, extracted from main()'s tail specifically to clear the 250-line cap)", + "linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_harness.sh + 6 lib test files (new: PATH-shim test suite for the 6 libs)", + "linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_entry_harness.sh + test_diagnose_pacman_hook_stall_args.sh + test_diagnose_pacman_hook_stall_timing.sh (new: subprocess test suite for the entry script, split into two files after the combined draft itself grew past the 250-line cap)", + "Not in scope: §3 (install_leechblock.sh/block_compulsive_opening.sh, deployed-copy trap) and everything else NEXT_SESSION_PROMPT.md §4 lists as out of scope this session." + ], + "changes": [ + "Split into 6 libs plus the entry script. Verified via a scratch shellcheck dry-run (with .shellcheckrc's external-sources=true / source-path=SCRIPTDIR, matching the real pre-commit hook) that run_one/cleanup/main/log_size cannot move into a lib: any lib-file split of run_one produces a real SC2153 (PACMAN_BIN vs PACMAN_PID) and SC2034 (LAST_ELAPSED) because a sourced lib has no mechanism to declare where its globals come from -- this exact combination caused a prior session (commit 2527fb3) to revert this same split. Kept them together in the entry script instead of suppressing.", + "Made PACMAN_BIN, PACMAN_LOG, CACHE_DIR, PACMAN_LOCK, and (later) LOAD_FLOOR_MB/LOAD_MIN_FREE_MB env-overridable via `: \"${VAR:=default}\"` instead of hardcoded (readonly or bare) assignment -- required for the test harness to point at fakes instead of /usr/bin/pacman.orig, /var/log/pacman.log, /var/cache/pacman/pkg, /var/lib/pacman/db.lck, and the real MemAvailable. Production behavior is unchanged since nothing else in the environment sets these.", + "Found and fixed a real bug in print_summary's awk one-liner: `if (NR == 0) exit` was dead code, because `printf '%s\\n' \"${_durations[@]}\"` on an empty bash array still emits one blank line (printf cycles its format at least once), so NR is always >= 1. Reachable in production via `-n 0`. Replaced with a bash-side `((${#_durations[@]} > 0))` guard, which also let the whole median/min/max computation move out of an inline awk block into plain bash arithmetic (see next item).", + "Refactored mem_available_mb, descendant_pids+kill_tree, capture_stall's two output-capture blocks, and print_summary's median calculation specifically to reach 100% kcov line coverage, after empirically confirming (bash -x on isolated repros) that kcov's xtrace-based line instrumentation never emits a trace event for a `done ` loop-closing line, a `} >redirect` group-closing line, or lines inside an inline awk script -- all three are structurally invisible to kcov regardless of how thoroughly the code executes. User was asked (grill-gate question, since this crosses into 'refactor code to satisfy a linter/coverage tool') and explicitly chose 'refactor each site' over 'fix the coverage tool'. mem_available_mb/print_summary use an explicit fd (exec 9<...; ...<&9) instead of `done `; kill_tree and capture_stall's two blocks use a real pipe (safe: their loop bodies don't need to survive a subshell). descendant_pids specifically could NOT become a pipe -- it appends to `pids` inside the loop, and a pipe/subshell would silently make it return only the root PID, breaking kill_tree/capture_stall's process-tree walk -- so it uses `exec 9< <(process substitution); ... <&9` instead, which is traceable and preserves the array mutation (verified by the existing 'walks the whole tree breadth-first' test, which passed unchanged before and after).", + "Confirmed (not just assumed) that converting capture_stall's `done < <(descendant_pids ...)` to a pipe doesn't introduce a pipefail hazard: descendant_pids seeds `pids=(\"$root\")` before its loop, so it can never produce empty output or a nonzero exit, so the pipeline's left side can never fail and abort capture_stall mid-write.", + "Extracted print_summary out of main() into its own lib (a real behavior-preserving refactor, not a mechanical move) specifically to make room under the 250-line cap once run_one/cleanup/log_size/main had to stay together; then further restructured main()'s `if (WATCH_MODE) { watch_forever; return }` into an if/else, since watch_forever never returns (intentional -- it's `while true`, killed externally) and the bare `return` after it was consequently unreachable and unfixable without changing watch_forever's contract.", + "Discovered (during the second advisor pass, before it was ever committed or the box put at real risk) that the new --with-load test case would otherwise dd 13-20+ GB into /dev/shm using this box's real available memory and the entry script's real LOAD_FLOOR_MB=800 default -- caught by review, not by an incident; fixed by making LOAD_FLOOR_MB/LOAD_MIN_FREE_MB env-overridable the same way as the other globals, and pinning the test's override 100 MB under actual MemAvailable so it allocates a small, bounded amount instead.", + "Used unshare -r (not real sudo) to drive the entry-script subprocess test as EUID 0, since require_root execs real sudo before argument parsing (even for -h). SIGINT delivery to a backgrounded child was observed to be unreliable in this Bash-tool environment (a plain `bash -c 'trap ...; sleep 30'` backgrounded and sent SIGINT never fired its trap, with or without disown); SIGTERM did. Since the entry script's `trap cleanup EXIT INT TERM` wires both signals to the identical handler, the cleanup/kill_tree-on-SIGINT test uses SIGTERM instead -- same code path, reliable delivery.", + "check_plagiarism-style precedent (lib/hosts_guard_migrate.sh) already carries the git-index executable bit despite being sourced, not executed; matched that convention for all 6 new libs rather than leaving them non-executable, after pre-commit's shebang-executable-bit hook flagged the mismatch.", + "The entry-script test file itself grew to 295 lines while adding -p/--with-load/log-advance/SIGTERM coverage (caught by check_file_length.sh --all, which does not exempt test files). Split it into pacman_hook_stall_entry_harness.sh (fake pacman.orig/ps/journalctl/pgrep/logger + the run_entry helper, shared) plus test_diagnose_pacman_hook_stall_args.sh (arg-parsing/basic runs) and test_diagnose_pacman_hook_stall_timing.sh (stall/hard-timeout/SIGTERM/--watch) -- both now under 150 lines. Re-measured coverage across both files together (kcov run twice into the same output dir, merged by shell_coverage_report.py) to confirm the split didn't lose any assertion; still 120/120=100.00%, still 13 assertions total, and jscpd re-measured unchanged (1.5%, 31 clones, none in these files) confirming the split was a clean extraction, not a copy-paste." + ], + "verification": [ + { + "command": "bash meta/scripts/check_file_length.sh --all", + "result": "pass", + "evidence": "12 files over cap (was 13); diagnose_pacman_hook_stall.sh no longer listed. Entry script itself is 249 lines." + }, + { + "command": "bash meta/scripts/shell_coverage.sh for all 7 subjects", + "result": "pass", + "evidence": "pacman_hook_stall_setup.sh 27/27=100.00%, pacman_hook_stall_load.sh 25/25=100.00%, pacman_hook_stall_capture.sh 59/59=100.00%, pacman_hook_stall_watch.sh 24/24=100.00%, pacman_hook_stall_usage.sh 2/2=100.00%, pacman_hook_stall_summary.sh 18/18=100.00%, diagnose_pacman_hook_stall.sh 120/120=100.00%." + }, + { + "command": "bash individually and as a full suite, 3x for the timing-sensitive ones (load/watch/entry)", + "result": "pass", + "evidence": "43 lib-level assertions (8+7+8+5+4+11 across the 6 lib test files) + 13 entry-level assertions (7 args + 6 timing, after the entry test split) = 56 total, 0 failed, stable across repeated runs." + }, + { + "command": "bash meta/scripts/verify_shell_split.sh 79829ec <6 libs>", + "result": "pass (with expected, explained DIFFERENCE)", + "evidence": "DIFFERENCE reported for exactly capture_stall, descendant_pids, kill_tree, main, mem_available_mb, print_summary, validate_requirements -- every one is a deliberate behavior-preserving refactor documented above (kcov-blind-spot fixes, PACMAN_LOCK injection, print_summary extraction/bugfix). Every other function (usage, cleanup, require_root, resolve_package_file, start_load, stop_load, log_size, run_one, watch_forever) verified byte-for-byte identical." + }, + { + "command": "shellcheck on all 17 code files together (matches the pre-commit hook's batching)", + "result": "pass", + "evidence": "Zero findings, including SC2034/SC2153 on the entry script's env-overridable globals -- confirmed .shellcheckrc's external-sources=true + source-path=SCRIPTDIR resolves these correctly when the entry script (not a lib) carries the source= annotations." + }, + { + "command": "jscpd --pattern **/*.sh --min-lines 14 --min-tokens 25 --threshold 2 . (run in a clean `git worktree add --detach` with the new files copied in)", + "result": "pass", + "evidence": "1.5% duplication overall (462 files, 63557 lines); none of the 31 reported clones involve any pacman_hook_stall_* file." + }, + { + "command": "pre-commit run --files (after git add --chmod=+x, per this repo's stated ordering)", + "result": "pass", + "evidence": "All hooks passed after fixing the 6 libs' missing executable bit (git-index mode 100644 -> 100755, matching lib/hosts_guard_migrate.sh's existing convention) and adding this evidence/contract pair." + }, + { + "command": "Manual memory-safety check after the --with-load fix", + "result": "pass", + "evidence": "grep MemAvailable /proc/meminfo before/after full suite run: 22381180 kB -> 22338908 kB (no meaningful drop); `ls /dev/shm/hook-stall-hog.*` empty after every run; no OOM entries in journalctl/dmesg for the session window." + } + ], + "risks": [ + "Six functions (capture_stall, descendant_pids, kill_tree, main, mem_available_mb, print_summary) and one arg-parse-adjacent global block were rewritten, not mechanically moved -- verify_shell_split.sh's guarantee only covers the other 9 functions verbatim. Each rewrite is covered by an existing or new test that passed unchanged before and after, but this is a materially larger diff than a pure split.", + "PACMAN_BIN/PACMAN_LOG/CACHE_DIR/PACMAN_LOCK/LOAD_FLOOR_MB/LOAD_MIN_FREE_MB are now env-overridable in the shipped script. require_root re-execs with `sudo -E`, which preserves the caller's environment (including any injected PACMAN_BIN) into the root process. Harmless today since reaching that exec already requires full interactive sudo; would become arbitrary-code-as-root only if a future NOPASSWD sudoers rule were ever added for this script specifically -- noted, not hardened, since that's out of this session's scope.", + "The print_summary NR==0 bug fix and the descendant_pids array-mutation preservation were both caught and verified during this session (advisor review + empirical bash -x tracing), not found by an external report -- there is no independent second reviewer confirmation beyond the test suite and the checks documented above.", + "This session never ran the script against a real pacman transaction (per NEXT_SESSION_PROMPT.md's 'split blind where there is no device' equivalent instruction -- no live verification requested or performed); all verification is via the PATH-shim harness and the real-but-fake pacman.orig subprocess test." + ], + "rollback": [ + "git revert the commit(s); the entry script and its 6 libs return to the single 493-line monolith with LOAD_FLOOR_MB/PACMAN_BIN/etc hardcoded and print_summary's dead awk NR==0 branch restored (bug re-introduced but never triggered in the original 40-run default flow, since -n 0 was not exercised in production use).", + "After rollback, confirm bash meta/scripts/check_file_length.sh --all reports diagnose_pacman_hook_stall.sh over cap again (493 lines)." + ] +} diff --git a/linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh b/linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh index f64696ba..c31ba8c6 100755 --- a/linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh +++ b/linux_configuration/scripts/single_use/fixes/diagnose_pacman_hook_stall.sh @@ -29,11 +29,37 @@ set -euo pipefail +# shellcheck source=lib/pacman_hook_stall_summary.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pacman_hook_stall_summary.sh" + +# shellcheck source=lib/pacman_hook_stall_usage.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pacman_hook_stall_usage.sh" + +# shellcheck source=lib/pacman_hook_stall_watch.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pacman_hook_stall_watch.sh" + +# shellcheck source=lib/pacman_hook_stall_capture.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pacman_hook_stall_capture.sh" + +# shellcheck source=lib/pacman_hook_stall_load.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pacman_hook_stall_load.sh" + +# shellcheck source=lib/pacman_hook_stall_setup.sh +. "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pacman_hook_stall_setup.sh" + SCRIPT_NAME="$(basename "$0")" readonly SCRIPT_NAME -readonly PACMAN_BIN="/usr/bin/pacman.orig" -readonly PACMAN_LOG="/var/log/pacman.log" -readonly CACHE_DIR="/var/cache/pacman/pkg" +# All four below are env-overridable (not hardcoded readonly) so the test +# harness can point them at fakes instead of the real machine; production +# gets the same defaults as before since nothing else sets these. +: "${PACMAN_BIN:=/usr/bin/pacman.orig}" +readonly PACMAN_BIN +: "${PACMAN_LOG:=/var/log/pacman.log}" +readonly PACMAN_LOG +: "${CACHE_DIR:=/var/cache/pacman/pkg}" +readonly CACHE_DIR +: "${PACMAN_LOCK:=/var/lib/pacman/db.lck}" +readonly PACMAN_LOCK # Defaults RUNS=40 @@ -43,8 +69,9 @@ HARD_TIMEOUT=120 # seconds before we give up and kill the stalled transaction OUT_DIR="/var/log/pacman-hook-stall" WITH_LOAD=0 WATCH_MODE=0 -LOAD_FLOOR_MB=800 # never push MemAvailable below this -LOAD_MIN_FREE_MB=1500 # refuse to start --with-load below this much available +# Env-overridable like PACMAN_BIN etc above, so a test can force a tiny allocation. +: "${LOAD_FLOOR_MB:=800}" # never push MemAvailable below this +: "${LOAD_MIN_FREE_MB:=1500}" # refuse to start --with-load below this much available HOG_FILE="" LOAD_PID="" @@ -53,33 +80,6 @@ STALLS=0 RUN_INDEX=0 LAST_ELAPSED=0 -usage() { - cat < stall (default: $STALL_TIMEOUT) - --hard-timeout S Seconds before killing a stalled run (default: $HARD_TIMEOUT) - --with-load Also apply memory pressure (the hypothesis under test). - Run this with nothing else going - it deliberately - breaks the one-heavy-job-at-a-time rule. - --watch Passive mode: drive nothing, just watch pacman.log and - dump diagnostics when someone else's transaction stalls - on a hook. Intended to run as a systemd service. - -o, --out DIR Where to write stall dumps (default: $OUT_DIR) - -h, --help Show this help - -Exit status: 0 if the loop completed (with or without stalls), non-zero on a -setup failure. The stall count is reported in the summary. -EOF - exit 0 -} - cleanup() { local rc=$? stop_load @@ -96,191 +96,14 @@ trap cleanup EXIT INT TERM # Preconditions # ---------------------------------------------------------------------------- -require_root() { - if [[ $EUID -ne 0 ]]; then - exec sudo -E bash "$0" "$@" - fi -} - -validate_requirements() { - local tool - for tool in "$PACMAN_BIN" ps awk sed journalctl; do - command -v "$tool" >/dev/null 2>&1 || [[ -x "$tool" ]] || { - echo "Error: required tool '$tool' not found" >&2 - exit 1 - } - done - - [[ -r "$PACMAN_LOG" ]] || { - echo "Error: cannot read $PACMAN_LOG" >&2 - exit 1 - } - - # Refuse to run alongside another *transaction*: a collision would look - # exactly like the stall we are hunting. db.lck is the authoritative signal - # - pacman holds it for the whole transaction. Deliberately NOT a pgrep for - # "pacman": read-only queries (pacman -Qi) take no lock, are harmless, and - # run constantly here from background services. - if [[ -e /var/lib/pacman/db.lck ]]; then - echo "Error: /var/lib/pacman/db.lck exists - another transaction is in" >&2 - echo "flight (or a stale lock remains). Resolve it before running." >&2 - exit 1 - fi -} - -# Resolve the cached package file for the installed version of $PACKAGE. -resolve_package_file() { - local version - version="$("$PACMAN_BIN" -Q "$PACKAGE" 2>/dev/null | awk '{print $2}')" - [[ -n "$version" ]] || { - echo "Error: package '$PACKAGE' is not installed" >&2 - exit 1 - } - - local candidate - for candidate in "$CACHE_DIR/$PACKAGE-$version"-*.pkg.tar.zst; do - [[ -e "$candidate" ]] || continue - printf '%s\n' "$candidate" - return 0 - done - - echo "Error: no cached package for $PACKAGE-$version in $CACHE_DIR" >&2 - echo "Hint: run '$PACMAN_BIN -Sw $PACKAGE' first" >&2 - exit 1 -} - # ---------------------------------------------------------------------------- # Load generator (tmpfs allocation = real anonymous-ish memory, no extra deps) # ---------------------------------------------------------------------------- -mem_available_mb() { - local kb=0 key value rest - while read -r key value rest; do - if [[ $key == "MemAvailable:" ]]; then - kb="$value" - break - fi - done = ${LOAD_MIN_FREE_MB} MB" >&2 - echo "Close Chromium / builds first." >&2 - exit 1 - fi - - local target=$((avail - LOAD_FLOOR_MB)) - HOG_FILE="$(mktemp /dev/shm/hook-stall-hog.XXXXXX)" - echo "Applying memory pressure: allocating ${target} MB (floor ${LOAD_FLOOR_MB} MB available)" - dd if=/dev/zero of="$HOG_FILE" bs=1M count="$target" status=none 2>/dev/null || true - echo "MemAvailable now: $(mem_available_mb) MB" -} - -stop_load() { - if [[ -n "$LOAD_PID" ]] && kill -0 "$LOAD_PID" 2>/dev/null; then - kill "$LOAD_PID" 2>/dev/null || true - fi - if [[ -n "$HOG_FILE" && -e "$HOG_FILE" ]]; then - rm -f "$HOG_FILE" - HOG_FILE="" - fi -} - # ---------------------------------------------------------------------------- # Diagnostics capture # ---------------------------------------------------------------------------- -# Print every descendant PID of $1, including $1 itself. -descendant_pids() { - local root="$1" - local pids=("$root") - local i=0 - while ((i < ${#pids[@]})); do - local parent="${pids[i]}" - local child - while read -r child; do - [[ -n "$child" ]] && pids+=("$child") - done < <(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ') - i=$((i + 1)) - done - printf '%s\n' "${pids[@]}" -} - -kill_tree() { - local root="$1" - local pid - while read -r pid; do - kill -9 "$pid" 2>/dev/null || true - done < <(descendant_pids "$root" | tac) -} - -capture_stall() { - local root_pid="$1" - local run="$2" - local stamp - printf -v stamp '%(%Y%m%d-%H%M%S)T' -1 - - # Classify before capturing. The stall we are hunting leaves pacman.log - # ending on a hook's "running '...hook'..." line; anything else stalled - # somewhere unrelated and must not be counted as a reproduction. - local kind="other" - if tail -1 "$PACMAN_LOG" 2>/dev/null | grep -q "running '.*\.hook'"; then - kind="hook" - fi - - local dir="$OUT_DIR/${stamp}-run${run}-${kind}" - mkdir -p "$dir" - - echo " !! STALL ($kind) - capturing diagnostics to $dir" - tail -1 "$PACMAN_LOG" >"$dir/last-log-line.txt" 2>&1 || true - - ps -eo pid,ppid,stat,wchan:32,etime,args --forest >"$dir/ps-forest.txt" 2>&1 || true - - local pid - while read -r pid; do - [[ -d "/proc/$pid" ]] || continue - { - echo "=== pid $pid ===" - echo "--- cmdline ---" - tr '\0' ' ' <"/proc/$pid/cmdline" 2>/dev/null || true - echo - echo "--- wchan ---" - cat "/proc/$pid/wchan" 2>/dev/null || true - echo - echo "--- syscall ---" - cat "/proc/$pid/syscall" 2>/dev/null || true - echo "--- stack ---" - cat "/proc/$pid/stack" 2>/dev/null || echo "(unavailable - needs CONFIG_STACKTRACE)" - echo "--- status ---" - grep -E "^(Name|State|Threads|VmRSS|SigBlk|SigIgn|SigCgt)" "/proc/$pid/status" 2>/dev/null || true - echo "--- smaps_rollup (Pss) ---" - grep -E "^Pss:" "/proc/$pid/smaps_rollup" 2>/dev/null || true - echo - } >>"$dir/procs.txt" 2>&1 - done < <(descendant_pids "$root_pid") - - cp /proc/meminfo "$dir/meminfo.txt" 2>/dev/null || true - { - echo "--- pressure/memory ---" - cat /proc/pressure/memory 2>/dev/null || true - echo "--- pressure/io ---" - cat /proc/pressure/io 2>/dev/null || true - echo "--- pressure/cpu ---" - cat /proc/pressure/cpu 2>/dev/null || true - } >"$dir/pressure.txt" 2>&1 - - journalctl -k -n 50 --no-pager >"$dir/dmesg-tail.txt" 2>&1 || true - tail -30 "$PACMAN_LOG" >"$dir/pacman-log-tail.txt" 2>&1 || true - - STALLS=$((STALLS + 1)) -} - # ---------------------------------------------------------------------------- # The repro loop # ---------------------------------------------------------------------------- @@ -335,114 +158,47 @@ run_one() { LAST_ELAPSED=$((ended - started)) } -# Passive mode: watch OTHER people's transactions instead of driving our own. -# -# This exists because the active repro loop could not reproduce the stall: 65 -# transactions (40 idle + 25 under 2.6 GB of memory pressure) ran clean, against -# a historical rate of 4 in 56. At a uniform 7% the odds of 65 clean runs are -# under 1%, so the trigger is conditional on something a synthetic loop does not -# recreate - which means the evidence has to be captured from a real stall, when -# it next happens, rather than manufactured. -# -# Runs as a systemd service; costs one stat(2) per second and nothing else. -watch_forever() { - echo "Watching $PACMAN_LOG for hook stalls (>= ${STALL_TIMEOUT}s silent)" - echo "Dumps: $OUT_DIR" - - local last_size last_change now armed=0 seq=0 - last_size="$(log_size)" - printf -v last_change '%(%s)T' -1 - - while true; do - sleep 1 - local size - size="$(log_size)" - printf -v now '%(%s)T' -1 - - if [[ $size != "$last_size" ]]; then - last_size="$size" - last_change="$now" - armed=0 - continue - fi - - # Silence alone is normal - no transaction is running most of the time. - # Only a LIVE pacman sitting on a hook line is the signature we want. - ((armed == 1)) && continue - ((now - last_change >= STALL_TIMEOUT)) || continue - - local pacman_pid - pacman_pid="$(pgrep -x pacman.orig | head -1 || true)" - [[ -n $pacman_pid ]] || continue - - seq=$((seq + 1)) - capture_stall "$pacman_pid" "watch${seq}" - logger -t pacman-hook-stall "captured a stalled transaction (pid $pacman_pid)" - armed=1 - done -} - main() { validate_requirements if ((WATCH_MODE == 1)); then mkdir -p "$OUT_DIR" watch_forever - return - fi + else + local pkg_file + pkg_file="$(resolve_package_file)" - local pkg_file - pkg_file="$(resolve_package_file)" + mkdir -p "$OUT_DIR" - mkdir -p "$OUT_DIR" - - echo "============================================================================" - echo "pacman hook-stall diagnosis" - echo "============================================================================" - echo " package : $PACKAGE ($pkg_file)" - echo " runs : $RUNS" - echo " stall after : ${STALL_TIMEOUT}s of pacman.log silence" - echo " hard timeout: ${HARD_TIMEOUT}s" - local load_desc="idle" - ((WITH_LOAD == 1)) && load_desc="memory pressure ON" - echo " load : $load_desc" - echo " dumps : $OUT_DIR" - echo - - start_load - - local durations=() - while ((RUN_INDEX < RUNS)); do - RUN_INDEX=$((RUN_INDEX + 1)) - run_one "$pkg_file" "$RUN_INDEX" - durations+=("$LAST_ELAPSED") - local marker="" - ((LAST_ELAPSED >= STALL_TIMEOUT)) && marker=" <-- slow" - printf ' run %2d/%d: %3ds%s\n' "$RUN_INDEX" "$RUNS" "$LAST_ELAPSED" "$marker" - done - - stop_load - - echo - echo "============================================================================" - echo "Summary" - echo "============================================================================" - printf ' transactions : %d\n' "$RUNS" - printf ' stalls : %d\n' "$STALLS" - printf ' durations : %s\n' "${durations[*]}" - printf '%s\n' "${durations[@]}" | sort -n | awk ' - { d[NR] = $1 } - END { - if (NR == 0) exit - printf " min/median/max: %ds / %ds / %ds\n", d[1], d[int((NR+1)/2)], d[NR] - }' - if ((STALLS > 0)); then + echo "============================================================================" + echo "pacman hook-stall diagnosis" + echo "============================================================================" + echo " package : $PACKAGE ($pkg_file)" + echo " runs : $RUNS" + echo " stall after : ${STALL_TIMEOUT}s of pacman.log silence" + echo " hard timeout: ${HARD_TIMEOUT}s" + local load_desc="idle" + ((WITH_LOAD == 1)) && load_desc="memory pressure ON" + echo " load : $load_desc" + echo " dumps : $OUT_DIR" echo - echo " Captured dumps:" - find "$OUT_DIR" -mindepth 1 -maxdepth 1 -type d -printf ' %p\n' \ - 2>/dev/null | sort | tail -"$STALLS" + + start_load + + local durations=() + while ((RUN_INDEX < RUNS)); do + RUN_INDEX=$((RUN_INDEX + 1)) + run_one "$pkg_file" "$RUN_INDEX" + durations+=("$LAST_ELAPSED") + local marker="" + ((LAST_ELAPSED >= STALL_TIMEOUT)) && marker=" <-- slow" + printf ' run %2d/%d: %3ds%s\n' "$RUN_INDEX" "$RUNS" "$LAST_ELAPSED" "$marker" + done + + stop_load + + print_summary durations fi - echo "============================================================================" } # Escalate BEFORE parsing: after the parse loop "$@" has been shifted away, so diff --git a/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_capture.sh b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_capture.sh new file mode 100755 index 00000000..3bfafcb5 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_capture.sh @@ -0,0 +1,105 @@ +#!/bin/bash +# Stall diagnostics capture for diagnose_pacman_hook_stall.sh. +# Sourced, not executed; inherits the caller's strict mode and globals. + +# Print every descendant PID of $1, including $1 itself. +descendant_pids() { + local root="$1" + local pids=("$root") + local i=0 + while ((i < ${#pids[@]})); do + local parent="${pids[i]}" + local child + # Explicit fd, not `done < <(...)`: kcov's xtrace-based line + # instrumentation never emits a trace event for a `done ` + # line, leaving it permanently "uncovered". Can't become a pipe + # instead (`... | while read ...`) -- that runs the loop body in a + # subshell, so `pids+=("$child")` would never survive back into + # this function and the walk would silently stop at the root. + exec 9< <(ps -o pid= --ppid "$parent" 2>/dev/null | tr -d ' ') + while read -r child <&9; do + [[ -n "$child" ]] && pids+=("$child") + done + exec 9<&- + i=$((i + 1)) + done + printf '%s\n' "${pids[@]}" +} + +kill_tree() { + local root="$1" + local pid + # A pipe (not `done < <(...)`) is safe here: the loop body only calls + # kill, so running it in a subshell (which a pipe implies) changes + # nothing observable -- unlike descendant_pids above. + descendant_pids "$root" | tac | while read -r pid; do + kill -9 "$pid" 2>/dev/null || true + done +} + +capture_stall() { + local root_pid="$1" + local run="$2" + local stamp + printf -v stamp '%(%Y%m%d-%H%M%S)T' -1 + + # Classify before capturing. The stall we are hunting leaves pacman.log + # ending on a hook's "running '...hook'..." line; anything else stalled + # somewhere unrelated and must not be counted as a reproduction. + local kind="other" + if tail -1 "$PACMAN_LOG" 2>/dev/null | grep -q "running '.*\.hook'"; then + kind="hook" + fi + + local dir="$OUT_DIR/${stamp}-run${run}-${kind}" + mkdir -p "$dir" + + echo " !! STALL ($kind) - capturing diagnostics to $dir" + tail -1 "$PACMAN_LOG" >"$dir/last-log-line.txt" 2>&1 || true + + ps -eo pid,ppid,stat,wchan:32,etime,args --forest >"$dir/ps-forest.txt" 2>&1 || true + + local pid + # Pipes, not a done-with-redirect loop or a redirected group command: + # kcov's xtrace-based line instrumentation never emits a trace event for + # the loop-closing or group-closing line when it carries a trailing + # redirect, leaving it permanently "uncovered". Safe to pipe here -- + # unlike descendant_pids, nothing in this loop body needs to survive + # outside a subshell. + descendant_pids "$root_pid" | while read -r pid; do + [[ -d "/proc/$pid" ]] || continue + { + echo "=== pid $pid ===" + echo "--- cmdline ---" + tr '\0' ' ' <"/proc/$pid/cmdline" 2>/dev/null || true + echo + echo "--- wchan ---" + cat "/proc/$pid/wchan" 2>/dev/null || true + echo + echo "--- syscall ---" + cat "/proc/$pid/syscall" 2>/dev/null || true + echo "--- stack ---" + cat "/proc/$pid/stack" 2>/dev/null || echo "(unavailable - needs CONFIG_STACKTRACE)" + echo "--- status ---" + grep -E "^(Name|State|Threads|VmRSS|SigBlk|SigIgn|SigCgt)" "/proc/$pid/status" 2>/dev/null || true + echo "--- smaps_rollup (Pss) ---" + grep -E "^Pss:" "/proc/$pid/smaps_rollup" 2>/dev/null || true + echo + } 2>&1 | cat >>"$dir/procs.txt" + done + + cp /proc/meminfo "$dir/meminfo.txt" 2>/dev/null || true + { + echo "--- pressure/memory ---" + cat /proc/pressure/memory 2>/dev/null || true + echo "--- pressure/io ---" + cat /proc/pressure/io 2>/dev/null || true + echo "--- pressure/cpu ---" + cat /proc/pressure/cpu 2>/dev/null || true + } 2>&1 | cat >"$dir/pressure.txt" + + journalctl -k -n 50 --no-pager >"$dir/dmesg-tail.txt" 2>&1 || true + tail -30 "$PACMAN_LOG" >"$dir/pacman-log-tail.txt" 2>&1 || true + + STALLS=$((STALLS + 1)) +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_load.sh b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_load.sh new file mode 100755 index 00000000..a5555817 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_load.sh @@ -0,0 +1,49 @@ +#!/bin/bash +# Memory-pressure load generator for diagnose_pacman_hook_stall.sh. +# Sourced, not executed; inherits the caller's strict mode and globals. + +mem_available_mb() { + local kb=0 key value rest + # Explicit fd, not `done ` + # line, so it stays "uncovered" no matter how the loop body is + # exercised. An explicit fd keeps the redirect on a line of its own + # that kcov *does* trace. + exec 9= ${LOAD_MIN_FREE_MB} MB" >&2 + echo "Close Chromium / builds first." >&2 + exit 1 + fi + + local target=$((avail - LOAD_FLOOR_MB)) + HOG_FILE="$(mktemp /dev/shm/hook-stall-hog.XXXXXX)" + echo "Applying memory pressure: allocating ${target} MB (floor ${LOAD_FLOOR_MB} MB available)" + dd if=/dev/zero of="$HOG_FILE" bs=1M count="$target" status=none 2>/dev/null || true + echo "MemAvailable now: $(mem_available_mb) MB" +} + +stop_load() { + if [[ -n "$LOAD_PID" ]] && kill -0 "$LOAD_PID" 2>/dev/null; then + kill "$LOAD_PID" 2>/dev/null || true + fi + if [[ -n "$HOG_FILE" && -e "$HOG_FILE" ]]; then + rm -f "$HOG_FILE" + HOG_FILE="" + fi +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_setup.sh b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_setup.sh new file mode 100755 index 00000000..ab86e155 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_setup.sh @@ -0,0 +1,56 @@ +#!/bin/bash +# Preconditions/setup for diagnose_pacman_hook_stall.sh. +# Sourced, not executed; inherits the caller's strict mode and globals. + +require_root() { + if [[ $EUID -ne 0 ]]; then + exec sudo -E bash "$0" "$@" + fi +} + +validate_requirements() { + local tool + for tool in "$PACMAN_BIN" ps awk sed journalctl; do + command -v "$tool" >/dev/null 2>&1 || [[ -x "$tool" ]] || { + echo "Error: required tool '$tool' not found" >&2 + exit 1 + } + done + + [[ -r "$PACMAN_LOG" ]] || { + echo "Error: cannot read $PACMAN_LOG" >&2 + exit 1 + } + + # Refuse to run alongside another *transaction*: a collision would look + # exactly like the stall we are hunting. db.lck is the authoritative signal + # - pacman holds it for the whole transaction. Deliberately NOT a pgrep for + # "pacman": read-only queries (pacman -Qi) take no lock, are harmless, and + # run constantly here from background services. + if [[ -e "$PACMAN_LOCK" ]]; then + echo "Error: $PACMAN_LOCK exists - another transaction is in" >&2 + echo "flight (or a stale lock remains). Resolve it before running." >&2 + exit 1 + fi +} + +# Resolve the cached package file for the installed version of $PACKAGE. +resolve_package_file() { + local version + version="$("$PACMAN_BIN" -Q "$PACKAGE" 2>/dev/null | awk '{print $2}')" + [[ -n "$version" ]] || { + echo "Error: package '$PACKAGE' is not installed" >&2 + exit 1 + } + + local candidate + for candidate in "$CACHE_DIR/$PACKAGE-$version"-*.pkg.tar.zst; do + [[ -e "$candidate" ]] || continue + printf '%s\n' "$candidate" + return 0 + done + + echo "Error: no cached package for $PACKAGE-$version in $CACHE_DIR" >&2 + echo "Hint: run '$PACMAN_BIN -Sw $PACKAGE' first" >&2 + exit 1 +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_summary.sh b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_summary.sh new file mode 100755 index 00000000..1beff0f7 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_summary.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# Final run summary for diagnose_pacman_hook_stall.sh. +# Sourced, not executed; inherits the caller's strict mode and globals. + +# Print the final transaction summary: counts, per-run durations, and (if any +# stalls happened) the list of captured diagnostic dumps. $1 is a nameref to +# the caller's durations array. Reads RUNS/STALLS/OUT_DIR, all set by main +# before this runs. +print_summary() { + local -n _durations="$1" + + echo + echo "============================================================================" + echo "Summary" + echo "============================================================================" + printf ' transactions : %d\n' "$RUNS" + printf ' stalls : %d\n' "$STALLS" + printf ' durations : %s\n' "${_durations[*]}" + # Bash, not awk: an inline awk script is opaque to kcov's bash-xtrace + # based line instrumentation, so those lines stay permanently + # "uncovered" no matter how thoroughly the code path is exercised. + # Also guard the count in bash: `printf '%s\n' "${_durations[@]}"` on + # an empty array still emits one blank line (printf cycles its format + # at least once), so an awk-side "NR == 0" check was always dead code. + if ((${#_durations[@]} > 0)); then + local -a sorted + mapfile -t sorted < <(printf '%s\n' "${_durations[@]}" | sort -n) + local count=${#sorted[@]} + printf ' min/median/max: %ds / %ds / %ds\n' \ + "${sorted[0]}" "${sorted[(count + 1) / 2 - 1]}" "${sorted[count - 1]}" + fi + if ((STALLS > 0)); then + echo + echo " Captured dumps:" + find "$OUT_DIR" -mindepth 1 -maxdepth 1 -type d -printf ' %p\n' \ + 2>/dev/null | sort | tail -"$STALLS" + fi + echo "============================================================================" +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_usage.sh b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_usage.sh new file mode 100755 index 00000000..ee4d3f1c --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_usage.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Usage/help text for diagnose_pacman_hook_stall.sh. +# Sourced, not executed; inherits the caller's strict mode and globals. + +usage() { + cat < stall (default: $STALL_TIMEOUT) + --hard-timeout S Seconds before killing a stalled run (default: $HARD_TIMEOUT) + --with-load Also apply memory pressure (the hypothesis under test). + Run this with nothing else going - it deliberately + breaks the one-heavy-job-at-a-time rule. + --watch Passive mode: drive nothing, just watch pacman.log and + dump diagnostics when someone else's transaction stalls + on a hook. Intended to run as a systemd service. + -o, --out DIR Where to write stall dumps (default: $OUT_DIR) + -h, --help Show this help + +Exit status: 0 if the loop completed (with or without stalls), non-zero on a +setup failure. The stall count is reported in the summary. +EOF + exit 0 +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_watch.sh b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_watch.sh new file mode 100755 index 00000000..2b3d3f27 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_watch.sh @@ -0,0 +1,50 @@ +#!/bin/bash +# Passive watch mode for diagnose_pacman_hook_stall.sh. +# Sourced, not executed; inherits the caller's strict mode and globals. + +# Passive mode: watch OTHER people's transactions instead of driving our own. +# +# This exists because the active repro loop could not reproduce the stall: 65 +# transactions (40 idle + 25 under 2.6 GB of memory pressure) ran clean, against +# a historical rate of 4 in 56. At a uniform 7% the odds of 65 clean runs are +# under 1%, so the trigger is conditional on something a synthetic loop does not +# recreate - which means the evidence has to be captured from a real stall, when +# it next happens, rather than manufactured. +# +# Runs as a systemd service; costs one stat(2) per second and nothing else. +watch_forever() { + echo "Watching $PACMAN_LOG for hook stalls (>= ${STALL_TIMEOUT}s silent)" + echo "Dumps: $OUT_DIR" + + local last_size last_change now armed=0 seq=0 + last_size="$(log_size)" + printf -v last_change '%(%s)T' -1 + + while true; do + sleep 1 + local size + size="$(log_size)" + printf -v now '%(%s)T' -1 + + if [[ $size != "$last_size" ]]; then + last_size="$size" + last_change="$now" + armed=0 + continue + fi + + # Silence alone is normal - no transaction is running most of the time. + # Only a LIVE pacman sitting on a hook line is the signature we want. + ((armed == 1)) && continue + ((now - last_change >= STALL_TIMEOUT)) || continue + + local pacman_pid + pacman_pid="$(pgrep -x pacman.orig | head -1 || true)" + [[ -n $pacman_pid ]] || continue + + seq=$((seq + 1)) + capture_stall "$pacman_pid" "watch${seq}" + logger -t pacman-hook-stall "captured a stalled transaction (pid $pacman_pid)" + armed=1 + done +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_entry_harness.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_entry_harness.sh new file mode 100755 index 00000000..f9155c3c --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_entry_harness.sh @@ -0,0 +1,142 @@ +#!/usr/bin/env bash +# lib/tests/pacman_hook_stall_entry_harness.sh — shared setup for the entry +# script's subprocess tests: fake pacman.orig/ps/journalctl/pgrep/logger on +# PATH, and the run_entry helper that drives diagnose_pacman_hook_stall.sh +# under unshare -r. +# +# Sourced, not executed, by test_diagnose_pacman_hook_stall_*.sh files. +# +# Deliberately does NOT put a fake `sleep` on PATH: run_one's `while kill -0` +# loop compares real epoch timestamps, so a fake sleep that returns instantly +# turns it into a tight spin loop. Real sleep + small STALL_TIMEOUT/ +# HARD_TIMEOUT values keep tests fast while exercising the real timing logic. +set -uo pipefail + +if ! command -v unshare >/dev/null 2>&1 || ! unshare -r true 2>/dev/null; then + echo "SKIP: unshare -r not available in this environment" >&2 + exit 0 +fi + +PASS=0 +FAIL=0 +_t_pass() { + PASS=$((PASS + 1)) + printf ' OK: %s\n' "$1" +} +_t_fail() { + FAIL=$((FAIL + 1)) + printf ' FAIL: %s\n' "$1" +} + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FIXES_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" +ENTRY="${FIXES_DIR}/diagnose_pacman_hook_stall.sh" + +TMPDIR="$(mktemp -d)" +trap 'rm -rf "$TMPDIR"' EXIT + +FAKE_BIN="${TMPDIR}/fake_bin" +mkdir -p "$FAKE_BIN" + +# pacman.orig: -Q reports an installed version; -U hangs for +# $DEV/hang_seconds (default 0), appending a pacman.log line partway through +# if $DEV/log_advance_at is set, then exits (or dies cleanly on TERM/INT so +# the hard-timeout/SIGTERM tests don't leave an orphan). +cat >"${FAKE_BIN}/pacman.orig" <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +DEV="${STALL_ENTRY_TEST_DEV}" +printf '%s\n' "pacman.orig $*" >>"${DEV}/calls" +if [[ "$1" == "-Q" ]]; then + printf '%s %s\n' "$2" "1.0.0-1" + exit 0 +fi +if [[ "$1" == "-U" ]]; then + hang=0 + [[ -f "${DEV}/hang_seconds" ]] && hang="$(cat "${DEV}/hang_seconds")" + trap 'exit 0' TERM INT + if [[ -f "${DEV}/log_advance_at" ]]; then + sleep "$(cat "${DEV}/log_advance_at")" + echo "running 'some.hook'..." >>"${PACMAN_LOG}" + hang=$((hang - $(cat "${DEV}/log_advance_at"))) + ((hang < 0)) && hang=0 + fi + sleep "$hang" + exit 0 +fi +exit 0 +EOF +chmod +x "${FAKE_BIN}/pacman.orig" + +for tool in ps journalctl pgrep logger; do + cat >"${FAKE_BIN}/${tool}" <>"\${STALL_ENTRY_TEST_DEV}/calls" +exit 0 +EOF + chmod +x "${FAKE_BIN}/${tool}" +done + +export PATH="${FAKE_BIN}:${PATH}" +export PACMAN_BIN="${FAKE_BIN}/pacman.orig" +export CACHE_DIR="${TMPDIR}/cache" +export PACMAN_LOCK="${TMPDIR}/db.lck" +# Matches the fake pacman.orig's `-Q base-devel` -> "base-devel 1.0.0-1". +mkdir -p "$CACHE_DIR" +touch "${CACHE_DIR}/base-devel-1.0.0-1-x86_64.pkg.tar.zst" +# Also cache the package the -p/--package test overrides to. +touch "${CACHE_DIR}/some-other-pkg-1.0.0-1-x86_64.pkg.tar.zst" + +# run_entry [--hang N] [--log-advance-at N] [--background] ARGS... — runs the +# entry script under unshare -r with a fresh PACMAN_LOG/OUT_DIR/DEV, waits up +# to 15s (or runs detached if --background is given), and returns the run's +# directory (stdout/stderr/rc/dev/pacman.log/dumps all live under it). +run_entry() { + local hang=0 log_advance_at="" background=0 + local parsing_helper_flags=1 + while ((parsing_helper_flags == 1)) && [[ "$1" == --* ]]; do + case "$1" in + --hang) + hang="$2" + shift 2 + ;; + --log-advance-at) + log_advance_at="$2" + shift 2 + ;; + --background) + background=1 + shift + ;; + *) + # Not one of this helper's own flags -- it's a real argument + # for the entry script (e.g. --nope, --with-load, --watch). + parsing_helper_flags=0 + ;; + esac + done + + local out_dir="${TMPDIR}/run_$$_${RANDOM}" + mkdir -p "$out_dir" + local dev="${out_dir}/dev" + mkdir -p "$dev" + : >"${dev}/calls" + echo "$hang" >"${dev}/hang_seconds" + [[ -n "$log_advance_at" ]] && echo "$log_advance_at" >"${dev}/log_advance_at" + local pacman_log="${out_dir}/pacman.log" + : >"$pacman_log" + local run_out_dir="${out_dir}/dumps" + + if ((background == 1)); then + STALL_ENTRY_TEST_DEV="$dev" PACMAN_LOG="$pacman_log" \ + unshare -r bash "$ENTRY" -o "$run_out_dir" "$@" \ + >"${out_dir}/stdout" 2>"${out_dir}/stderr" & + echo "$!" >"${out_dir}/pid" + else + STALL_ENTRY_TEST_DEV="$dev" PACMAN_LOG="$pacman_log" \ + timeout 15 unshare -r bash "$ENTRY" -o "$run_out_dir" "$@" \ + >"${out_dir}/stdout" 2>"${out_dir}/stderr" + echo "$?" >"${out_dir}/rc" + fi + printf '%s' "$out_dir" +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_harness.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_harness.sh new file mode 100755 index 00000000..e67dbb4f --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/pacman_hook_stall_harness.sh @@ -0,0 +1,205 @@ +#!/usr/bin/env bash +# lib/tests/pacman_hook_stall_harness.sh — fake pacman/ps/journalctl/pgrep/ +# logger/sleep behind the diagnose_pacman_hook_stall.sh split. +# +# Sourced, not executed. Every external tool the script calls is a shim on +# PATH that records its invocation into $DEV and does the minimum real +# filesystem work the calling code depends on. PACMAN_BIN, PACMAN_LOG, +# CACHE_DIR and PACMAN_LOCK all point into a throwaway tmpdir (via the +# env-overridable globals the split introduced) so nothing touches the real +# machine. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +FIXES_DIR="$(cd "${SCRIPT_DIR}/../.." && pwd)" + +PASS=0 +FAIL=0 + +_t_pass() { + PASS=$((PASS + 1)) + printf ' OK: %s\n' "$1" +} + +_t_fail() { + FAIL=$((FAIL + 1)) + printf ' FAIL: %s\n' "$1" +} + +_t_eq() { + local want="$1" got="$2" what="$3" + if [[ "$got" == "$want" ]]; then + _t_pass "$what" + else + _t_fail "$what (want '${want}', got '${got}')" + fi +} + +TEST_TMPDIR="$(mktemp -d)" +trap 'rm -rf "${TEST_TMPDIR}"' EXIT + +readonly DEV="${TEST_TMPDIR}/device" +readonly FAKE_BIN="${TEST_TMPDIR}/fake_bin" +mkdir -p "${DEV}" "${FAKE_BIN}" + +# --- fake external tools ---------------------------------------------------- + +# Fake pacman.orig: `-Q ` reports an installed version (or "fail_query" +# to simulate not-installed), `-U --noconfirm ` simulates a transaction +# that either finishes fast or hangs until killed (via "$DEV/hang_seconds"). +cat >"${FAKE_BIN}/pacman.orig" <<'PACMANSHIM' +#!/usr/bin/env bash +set -euo pipefail +DEV="${STALL_TEST_DEV}" +printf '%s\n' "pacman.orig $*" >>"${DEV}/calls" +if [[ "$1" == "-Q" ]]; then + if [[ -f "${DEV}/fail_query" ]]; then + exit 1 + fi + printf '%s %s\n' "$2" "1.2.3-1" + exit 0 +fi +if [[ "$1" == "-U" ]]; then + hang=0 + [[ -f "${DEV}/hang_seconds" ]] && hang="$(cat "${DEV}/hang_seconds")" + trap 'exit 0' TERM INT + sleep "$hang" + exit 0 +fi +exit 0 +PACMANSHIM +chmod +x "${FAKE_BIN}/pacman.orig" + +# Fake ps: only the two invocation shapes the code uses. +# ps -o pid= --ppid -> children of , from $DEV/ps_tree +# ps -eo ... --forest -> a static one-line snapshot +cat >"${FAKE_BIN}/ps" <<'PSSHIM' +#!/usr/bin/env bash +set -euo pipefail +DEV="${STALL_TEST_DEV}" +if [[ "$1" == "-o" && "$2" == "pid=" && "$3" == "--ppid" ]]; then + parent="$4" + [[ -f "${DEV}/ps_tree" ]] || exit 0 + awk -F: -v p="$parent" '$1 == p { print $2 }' "${DEV}/ps_tree" + exit 0 +fi +echo " PID PPID STAT WCHAN ELAPSED COMMAND" +exit 0 +PSSHIM +chmod +x "${FAKE_BIN}/ps" + +cat >"${FAKE_BIN}/journalctl" <<'JOURNALSHIM' +#!/usr/bin/env bash +set -euo pipefail +printf '%s\n' "journalctl $*" >>"${STALL_TEST_DEV}/calls" +echo "fake kernel log line" +exit 0 +JOURNALSHIM +chmod +x "${FAKE_BIN}/journalctl" + +# Fake pgrep -x pacman.orig: reports the PID recorded in $DEV/pgrep_pid, or +# nothing (real pgrep exits 1 on no match). +cat >"${FAKE_BIN}/pgrep" <<'PGREPSHIM' +#!/usr/bin/env bash +set -euo pipefail +DEV="${STALL_TEST_DEV}" +printf '%s\n' "pgrep $*" >>"${DEV}/calls" +if [[ -f "${DEV}/pgrep_pid" ]]; then + cat "${DEV}/pgrep_pid" + exit 0 +fi +exit 1 +PGREPSHIM +chmod +x "${FAKE_BIN}/pgrep" + +cat >"${FAKE_BIN}/logger" <<'LOGGERSHIM' +#!/usr/bin/env bash +printf '%s\n' "logger $*" >>"${STALL_TEST_DEV}/calls" +exit 0 +LOGGERSHIM +chmod +x "${FAKE_BIN}/logger" + +# Fake sleep: records calls and, after $DEV/sleep_exit_after calls, exits +# non-zero so an infinite `while true; do sleep 1; ...; done` loop +# (watch_forever) terminates instead of hanging the test suite. +cat >"${FAKE_BIN}/sleep" <<'SLEEPSHIM' +#!/usr/bin/env bash +set -euo pipefail +DEV="${STALL_TEST_DEV}" +count_file="${DEV}/sleep_count" +[[ -f "$count_file" ]] || echo 0 >"$count_file" +n=$(($(cat "$count_file") + 1)) +echo "$n" >"$count_file" +if [[ -f "${DEV}/sleep_exit_after" ]]; then + limit="$(cat "${DEV}/sleep_exit_after")" + if ((n > limit)); then + exit 1 + fi +fi +exit 0 +SLEEPSHIM +chmod +x "${FAKE_BIN}/sleep" + +export STALL_TEST_DEV="${DEV}" +export PATH="${FAKE_BIN}:${PATH}" + +# --- subject under test ----------------------------------------------------- + +export PACMAN_BIN="${FAKE_BIN}/pacman.orig" +export PACMAN_LOG="${TEST_TMPDIR}/pacman.log" +export CACHE_DIR="${TEST_TMPDIR}/cache" +export PACMAN_LOCK="${TEST_TMPDIR}/db.lck" +: >"${PACMAN_LOG}" +mkdir -p "${CACHE_DIR}" + +# shellcheck source=../pacman_hook_stall_setup.sh +. "${FIXES_DIR}/lib/pacman_hook_stall_setup.sh" +# shellcheck source=../pacman_hook_stall_load.sh +. "${FIXES_DIR}/lib/pacman_hook_stall_load.sh" +# shellcheck source=../pacman_hook_stall_capture.sh +. "${FIXES_DIR}/lib/pacman_hook_stall_capture.sh" +# shellcheck source=../pacman_hook_stall_watch.sh +. "${FIXES_DIR}/lib/pacman_hook_stall_watch.sh" +# shellcheck source=../pacman_hook_stall_usage.sh +. "${FIXES_DIR}/lib/pacman_hook_stall_usage.sh" +# shellcheck source=../pacman_hook_stall_summary.sh +. "${FIXES_DIR}/lib/pacman_hook_stall_summary.sh" + +# log_size lives in the entry script itself (diagnose_pacman_hook_stall.sh), +# not in a lib -- moving it would reintroduce the LAST_ELAPSED/PACMAN_BIN +# cross-file-global seam the split was designed to avoid. Redefined here +# verbatim so run_one()/watch_forever() (called directly from tests) resolve +# it exactly as they do when sourced from the real entry script. +log_size() { stat -c %s "$PACMAN_LOG" 2>/dev/null || echo 0; } + +# Globals the six libs read. run_one()/main()/cleanup() (which additionally +# read WATCH_MODE/RUN_INDEX/LAST_ELAPSED/PACMAN_PID) live in the entry +# script, not a lib -- test_diagnose_pacman_hook_stall.sh exercises those +# directly as a subprocess instead, so this harness only needs what the libs +# themselves reference. +SCRIPT_NAME="diagnose_pacman_hook_stall.sh" +RUNS=40 +PACKAGE="base-devel" +STALL_TIMEOUT=20 +HARD_TIMEOUT=120 +OUT_DIR="${TEST_TMPDIR}/out" +WITH_LOAD=0 +LOAD_FLOOR_MB=800 +LOAD_MIN_FREE_MB=1500 +HOG_FILE="" +LOAD_PID="" +STALLS=0 + +# reset_state — wipe DEV/OUT_DIR/PACMAN_LOG between test groups so each group +# starts from "nothing has happened yet". +reset_state() { + rm -rf "${OUT_DIR}" "${DEV:?}/calls" "${DEV}/hang_seconds" "${DEV}/ps_tree" \ + "${DEV}/pgrep_pid" "${DEV}/sleep_count" "${DEV}/sleep_exit_after" \ + "${DEV}/fail_query" 2>/dev/null || true + : >"${PACMAN_LOG}" + : >"${DEV}/calls" + STALLS=0 + HOG_FILE="" + LOAD_PID="" +} +reset_state diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_diagnose_pacman_hook_stall_args.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_diagnose_pacman_hook_stall_args.sh new file mode 100755 index 00000000..2362bd70 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_diagnose_pacman_hook_stall_args.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +# Subprocess tests for diagnose_pacman_hook_stall.sh's arg parsing and basic +# run paths: -h, unknown flags, -n 0, -p, --with-load, and a single clean +# -n 1 run (plus the "log advances mid-run" branch of run_one). +# +# See pacman_hook_stall_entry_harness.sh for the fake pacman.orig/run_entry +# setup this sources. The heavier timing tests (stalls, hard-timeout, +# SIGTERM, --watch) live in test_diagnose_pacman_hook_stall_timing.sh -- +# split purely to stay under the repo's 250-line file cap. +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_entry_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_entry_harness.sh" + +# --- -h/--help: exits 0, prints usage, never touches pacman ----------------- + +out_dir="$(run_entry -h)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "^Usage:" "${out_dir}/stdout"; then + _t_pass "entry -h: exits 0 and prints usage" +else + _t_fail "entry -h: expected exit 0 and a usage line, got rc=$rc, stdout: $(cat "${out_dir}/stdout")" +fi + +# --- unknown flag: exits 1, no usage ---------------------------------------- + +out_dir="$(run_entry --nope)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" != "0" ]] && grep -q "Unknown option" "${out_dir}/stderr"; then + _t_pass "entry --nope: rejects an unknown flag" +else + _t_fail "entry --nope: expected a nonzero exit and 'Unknown option', got rc=$rc" +fi + +# --- -n 0: skip_load path, RUNS=0, print_summary with an empty array ------- + +out_dir="$(run_entry -n 0)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "transactions : 0" "${out_dir}/stdout"; then + _t_pass "entry -n 0: completes with zero transactions" +else + _t_fail "entry -n 0: expected exit 0 and 'transactions : 0', got rc=$rc, stdout: $(cat "${out_dir}/stdout")" +fi + +# --- -p/--package: value flows through to the summary banner --------------- + +out_dir="$(run_entry -n 0 -p some-other-pkg)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "package : some-other-pkg" "${out_dir}/stdout"; then + _t_pass "entry -p: overrides the package name" +else + _t_fail "entry -p: expected 'package : some-other-pkg' in stdout, got rc=$rc" +fi + +# --- --with-load: allocates a small amount and reports "memory pressure ON" - +# +# LOAD_FLOOR_MB defaults to 800 (allocate "everything down to 800 MB free"), +# which on a real box would dd multiple GB into tmpfs. Pin it well under +# whatever's actually available (a 100 MB margin, not the smallest possible) +# so this test allocates a small, positive amount even if MemAvailable drops +# slightly between this snapshot and start_load's own re-read of it. + +avail_mb="$(awk '/MemAvailable/ {print int($2 / 1024)}' /proc/meminfo)" +export LOAD_FLOOR_MB=$((avail_mb - 100)) +export LOAD_MIN_FREE_MB=1 +out_dir="$(run_entry -n 0 --with-load -t 1 --hard-timeout 2)" +unset LOAD_FLOOR_MB LOAD_MIN_FREE_MB +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "memory pressure ON" "${out_dir}/stdout" && + grep -qE "allocating [0-9]{1,3} MB" "${out_dir}/stdout"; then + _t_pass "entry --with-load: reports memory pressure ON and allocates a small amount" +else + _t_fail "entry --with-load: expected 'memory pressure ON' and a small allocation, got rc=$rc, stdout: $(cat "${out_dir}/stdout")" +fi + +# --- -n 1, fast pacman: one clean run, no stall ----------------------------- + +out_dir="$(run_entry -n 1 -t 5 --hard-timeout 8)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "stalls : 0" "${out_dir}/stdout"; then + _t_pass "entry -n 1 (fast pacman): completes with zero stalls" +else + _t_fail "entry -n 1 (fast pacman): expected exit 0 and zero stalls, got rc=$rc, stdout: $(cat "${out_dir}/stdout")" +fi + +# --- -n 1, pacman.log advances mid-run: run_one's "log advanced, reset the +# silence timer" branch fires and no stall is captured ----------------- + +out_dir="$(run_entry --log-advance-at 1 -n 1 -t 5 --hard-timeout 8)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "stalls : 0" "${out_dir}/stdout"; then + _t_pass "entry -n 1 (log advances mid-run): log activity resets the silence timer" +else + _t_fail "entry -n 1 (log advances): expected exit 0 and zero stalls, got rc=$rc, stdout: $(cat "${out_dir}/stdout")" +fi + +echo +echo "diagnose_pacman_hook_stall (args): ${PASS} passed, ${FAIL} failed" +((FAIL == 0)) diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_diagnose_pacman_hook_stall_timing.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_diagnose_pacman_hook_stall_timing.sh new file mode 100755 index 00000000..87647c9a --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_diagnose_pacman_hook_stall_timing.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# Subprocess tests for diagnose_pacman_hook_stall.sh's timing-sensitive +# paths: a real stall capture, the hard-timeout kill, cleanup's kill_tree +# branch on SIGTERM, and --watch mode. +# +# See pacman_hook_stall_entry_harness.sh for the fake pacman.orig/run_entry +# setup this sources. Arg-parsing and other basic run paths live in +# test_diagnose_pacman_hook_stall_args.sh -- split purely to stay under the +# repo's 250-line file cap. +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_entry_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_entry_harness.sh" + +# --- -n 1 with a real stall: pacman hangs past STALL_TIMEOUT but returns +# before HARD_TIMEOUT -- one stall captured, run still completes -------- + +out_dir="$(run_entry --hang 3 -n 1 -t 1 --hard-timeout 8)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "stalls : 1" "${out_dir}/stdout"; then + _t_pass "entry -n 1 (pacman hangs 3s, timeout 1s): captures exactly one stall" +else + _t_fail "entry -n 1 (stall): expected exit 0 and one stall, got rc=$rc, stdout: $(cat "${out_dir}/stdout")" +fi +if [[ -d "${out_dir}/dumps" ]] && [[ -n "$(find "${out_dir}/dumps" -mindepth 1 -maxdepth 1 -type d)" ]]; then + _t_pass "entry -n 1 (stall): writes a dump directory under -o" +else + _t_fail "entry -n 1 (stall): expected a dump directory under ${out_dir}/dumps" +fi + +# --- -n 1 with hard-timeout: pacman hangs past HARD_TIMEOUT, gets killed ---- + +out_dir="$(run_entry --hang 20 -n 1 -t 1 --hard-timeout 2)" +rc="$(cat "${out_dir}/rc")" +if [[ "$rc" == "0" ]] && grep -q "hard timeout" "${out_dir}/stdout"; then + _t_pass "entry -n 1 (hard timeout): kills the stuck transaction and reports it" +else + _t_fail "entry -n 1 (hard timeout): expected a 'hard timeout' line, got rc=$rc, stdout: $(cat "${out_dir}/stdout")" +fi + +# --- SIGTERM mid-run: cleanup's kill_tree branch fires on a live PACMAN_PID. +# Uses TERM, not INT: the entry script's `trap cleanup EXIT INT TERM` +# wires both to the same handler, and INT delivery to a backgrounded job +# was observed to be unreliable in this environment while TERM is not. + +out_dir="$(run_entry --background --hang 30 -n 5 -t 100 --hard-timeout 100)" +pid="$(cat "${out_dir}/pid")" +disown "$pid" 2>/dev/null || true +/usr/bin/sleep 1 +kill -TERM "$pid" 2>/dev/null || true +for _ in $(seq 1 30); do + grep -q "Cleaning up: killing in-flight pacman" "${out_dir}/stderr" 2>/dev/null && break + /usr/bin/sleep 0.2 +done +wait "$pid" 2>/dev/null || true +if grep -q "Cleaning up: killing in-flight pacman" "${out_dir}/stderr"; then + _t_pass "entry SIGTERM mid-run: cleanup kills the in-flight pacman transaction" +else + _t_fail "entry SIGTERM mid-run: expected the cleanup message on stderr, got: $(cat "${out_dir}/stderr")" +fi + +# --- --watch: runs the passive watcher; kill it and confirm it started ----- + +out_dir="$(run_entry --background --watch)" +pid="$(cat "${out_dir}/pid")" +/usr/bin/sleep 1.5 +if kill -0 "$pid" 2>/dev/null; then + _t_pass "entry --watch: stays running (doesn't exit early)" +else + _t_fail "entry --watch: exited early, stdout: $(cat "${out_dir}/stdout" 2>/dev/null)" +fi +kill -9 "$pid" 2>/dev/null || true +wait "$pid" 2>/dev/null || true +if grep -q "^Watching" "${out_dir}/stdout" 2>/dev/null; then + _t_pass "entry --watch: prints the watching banner" +else + _t_fail "entry --watch: expected a 'Watching' banner in stdout" +fi + +echo +echo "diagnose_pacman_hook_stall (timing): ${PASS} passed, ${FAIL} failed" +((FAIL == 0)) diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_capture.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_capture.sh new file mode 100755 index 00000000..b187fb6a --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_capture.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# Tests for lib/pacman_hook_stall_capture.sh: descendant_pids, kill_tree, +# capture_stall. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_harness.sh" + +# --- descendant_pids ----------------------------------------------------------- + +reset_state +# ps_tree format: "parent:child" lines, one per fake `ps -o pid= --ppid`. +cat >"${DEV}/ps_tree" <<'EOF' +100:200 +100:201 +200:300 +EOF +got="$(descendant_pids 100 | sort -n | tr '\n' ' ')" +_t_eq "100 200 201 300 " "$got" \ + "descendant_pids: walks the whole tree breadth-first, includes the root" + +reset_state +got="$(descendant_pids 500 | tr '\n' ' ')" +_t_eq "500 " "$got" \ + "descendant_pids: a childless root prints only itself" + +# --- kill_tree ------------------------------------------------------------- + +reset_state +/usr/bin/sleep 60 & +parent=$! +disown "$parent" 2>/dev/null || true +/usr/bin/sleep 60 & +child=$! +disown "$child" 2>/dev/null || true +printf '%s:%s\n' "$parent" "$child" >"${DEV}/ps_tree" +kill_tree "$parent" +still_alive=0 +for _ in $(seq 1 20); do + kill -0 "$parent" 2>/dev/null && still_alive=$((still_alive + 1)) + kill -0 "$child" 2>/dev/null && still_alive=$((still_alive + 1)) + ((still_alive == 0)) && break + still_alive=0 + /usr/bin/sleep 0.1 +done +if ! kill -0 "$parent" 2>/dev/null && ! kill -0 "$child" 2>/dev/null; then + _t_pass "kill_tree: kills the root and every descendant" +else + _t_fail "kill_tree: expected both root and child dead" + kill -9 "$parent" "$child" 2>/dev/null || true +fi + +reset_state +if (kill_tree 999999); then + _t_pass "kill_tree: a nonexistent root is a harmless no-op" +else + _t_fail "kill_tree: should not fail on a nonexistent PID" +fi + +# --- capture_stall ----------------------------------------------------------- + +reset_state +printf 'some earlier line\n' >>"${PACMAN_LOG}" +printf "running 'guard-lib.hook'...\n" >>"${PACMAN_LOG}" +STALLS=0 +capture_stall $$ 1 +dump_dir="$(find "${OUT_DIR}" -mindepth 1 -maxdepth 1 -type d | head -1)" +if [[ -n "$dump_dir" && "$dump_dir" == *-run1-hook ]]; then + _t_pass "capture_stall: classifies a hook-line stall and names the dump dir accordingly" +else + _t_fail "capture_stall: expected a *-run1-hook dump dir, got '$dump_dir'" +fi +_t_eq "1" "$STALLS" "capture_stall: increments STALLS" + +reset_state +printf 'unrelated log line\n' >>"${PACMAN_LOG}" +STALLS=0 +capture_stall $$ 2 +dump_dir="$(find "${OUT_DIR}" -mindepth 1 -maxdepth 1 -type d | head -1)" +if [[ -n "$dump_dir" && "$dump_dir" == *-run2-other ]]; then + _t_pass "capture_stall: classifies a non-hook stall as 'other'" +else + _t_fail "capture_stall: expected a *-run2-other dump dir, got '$dump_dir'" +fi + +reset_state +printf 'x\n' >>"${PACMAN_LOG}" +capture_stall $$ 3 +dump_dir="$(find "${OUT_DIR}" -mindepth 1 -maxdepth 1 -type d | head -1)" +if [[ -f "${dump_dir}/procs.txt" && -f "${dump_dir}/meminfo.txt" && + -f "${dump_dir}/pressure.txt" && -f "${dump_dir}/dmesg-tail.txt" && + -f "${dump_dir}/pacman-log-tail.txt" && -f "${dump_dir}/last-log-line.txt" && + -f "${dump_dir}/ps-forest.txt" ]]; then + _t_pass "capture_stall: writes every expected diagnostic file" +else + _t_fail "capture_stall: missing one or more expected diagnostic files in $dump_dir" + ls -la "$dump_dir" 2>&1 +fi + +echo +echo "pacman_hook_stall_capture: ${PASS} passed, ${FAIL} failed" +((FAIL == 0)) diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_load.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_load.sh new file mode 100755 index 00000000..63c1e95e --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_load.sh @@ -0,0 +1,111 @@ +#!/usr/bin/env bash +# Tests for lib/pacman_hook_stall_load.sh: mem_available_mb, start_load, +# stop_load. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_harness.sh" + +# --- mem_available_mb --------------------------------------------------------- + +reset_state +got="$(mem_available_mb)" +if [[ "$got" =~ ^[0-9]+$ && "$got" -gt 0 ]]; then + _t_pass "mem_available_mb: reads a positive integer from /proc/meminfo" +else + _t_fail "mem_available_mb: expected a positive integer, got '$got'" +fi + +# --- start_load / stop_load: WITH_LOAD=0 is a no-op --------------------------- + +reset_state +WITH_LOAD=0 +if (start_load) && [[ -z "$HOG_FILE" ]]; then + _t_pass "start_load: no-ops when WITH_LOAD=0" +else + _t_fail "start_load: should no-op (and not set HOG_FILE) when WITH_LOAD=0" +fi + +# --- start_load: refuses when available memory is below LOAD_MIN_FREE_MB ------ + +reset_state +WITH_LOAD=1 +avail="$(mem_available_mb)" +LOAD_MIN_FREE_MB=$((avail + 100000)) +if (start_load) 2>/dev/null; then + _t_fail "start_load: should refuse when available < LOAD_MIN_FREE_MB" +else + _t_pass "start_load: refuses when available < LOAD_MIN_FREE_MB" +fi + +# --- start_load: allocates (a tiny amount) and stop_load cleans it up --------- + +reset_state +WITH_LOAD=1 +avail="$(mem_available_mb)" +LOAD_MIN_FREE_MB=1 +LOAD_FLOOR_MB=$((avail - 2)) +( + start_load + [[ -n "$HOG_FILE" && -e "$HOG_FILE" ]] || exit 1 + echo "$HOG_FILE" >"${DEV}/hog_file_path" +) +if [[ -f "${DEV}/hog_file_path" ]]; then + _t_pass "start_load: allocates and sets HOG_FILE when under the floor" +else + _t_fail "start_load: should allocate and set HOG_FILE" +fi + +reset_state +WITH_LOAD=1 +avail="$(mem_available_mb)" +LOAD_MIN_FREE_MB=1 +LOAD_FLOOR_MB=$((avail - 2)) +start_load +hog="$HOG_FILE" +stop_load +if [[ -n "$hog" && ! -e "$hog" && -z "$HOG_FILE" ]]; then + _t_pass "stop_load: removes HOG_FILE and clears the variable" +else + _t_fail "stop_load: expected HOG_FILE removed and cleared, got '$HOG_FILE' (file existed: $([[ -e "$hog" ]] && echo yes || echo no))" +fi + +# --- stop_load: kills LOAD_PID if still alive ---------------------------------- + +reset_state +/usr/bin/sleep 60 & +LOAD_PID=$! +# Without job control (the default for a non-interactive script), SIGTERM to +# a background job can race the shell's own handling of it; disown detaches +# the job from shell bookkeeping so the kill below only affects the child. +disown "$LOAD_PID" 2>/dev/null || true +stop_load +still_alive=1 +for _ in $(seq 1 20); do + kill -0 "$LOAD_PID" 2>/dev/null || { + still_alive=0 + break + } + /usr/bin/sleep 0.1 +done +if ((still_alive == 0)); then + _t_pass "stop_load: kills a live LOAD_PID" +else + _t_fail "stop_load: LOAD_PID should have been killed" + kill -9 "$LOAD_PID" 2>/dev/null || true +fi +wait "$LOAD_PID" 2>/dev/null || true + +# --- stop_load: no-ops cleanly with no HOG_FILE/LOAD_PID set ------------------- + +reset_state +if (stop_load); then + _t_pass "stop_load: no-ops cleanly when nothing was started" +else + _t_fail "stop_load: should not fail with nothing to clean up" +fi + +echo +echo "pacman_hook_stall_load: ${PASS} passed, ${FAIL} failed" +((FAIL == 0)) diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_setup.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_setup.sh new file mode 100755 index 00000000..34a02731 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_setup.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# Tests for lib/pacman_hook_stall_setup.sh: require_root, validate_requirements, +# resolve_package_file. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_harness.sh" + +# --- require_root ------------------------------------------------------------ + +# Under unshare -r, EUID is 0 inside this shell, so require_root's `exec sudo` +# branch must never fire (a real invocation would replace this test process). +# "$@" matches the real call site (diagnose_pacman_hook_stall.sh: require_root "$@"). +if (require_root "$@"); then + _t_pass "require_root: no-ops when EUID is already 0" +else + _t_fail "require_root: no-op path failed as root" +fi + +# --- validate_requirements ---------------------------------------------------- + +reset_state +if (validate_requirements); then + _t_pass "validate_requirements: passes with fakes on PATH, readable log, no lock" +else + _t_fail "validate_requirements: unexpectedly failed the happy path" +fi + +reset_state +: >"${PACMAN_LOCK}" +if (validate_requirements) 2>/dev/null; then + _t_fail "validate_requirements: should reject when PACMAN_LOCK exists" +else + _t_pass "validate_requirements: rejects when PACMAN_LOCK exists" +fi +rm -f "${PACMAN_LOCK}" + +reset_state +rm -f "${PACMAN_LOG}" +if (validate_requirements) 2>/dev/null; then + _t_fail "validate_requirements: should reject an unreadable PACMAN_LOG" +else + _t_pass "validate_requirements: rejects an unreadable PACMAN_LOG" +fi +: >"${PACMAN_LOG}" + +reset_state +if (PATH="${TEST_TMPDIR}/empty_path" validate_requirements) 2>/dev/null; then + _t_fail "validate_requirements: should reject a missing required tool" +else + _t_pass "validate_requirements: rejects a missing required tool" +fi + +# --- resolve_package_file ----------------------------------------------------- + +reset_state +mkdir -p "${CACHE_DIR}" +touch "${CACHE_DIR}/base-devel-1.2.3-1-x86_64.pkg.tar.zst" +got="$(resolve_package_file)" +_t_eq "${CACHE_DIR}/base-devel-1.2.3-1-x86_64.pkg.tar.zst" "$got" \ + "resolve_package_file: finds the cached package for the installed version" + +reset_state +: >"${DEV}/fail_query" +if (resolve_package_file) 2>/dev/null; then + _t_fail "resolve_package_file: should reject a package that is not installed" +else + _t_pass "resolve_package_file: rejects a package that is not installed" +fi + +reset_state +rm -f "${CACHE_DIR}"/*.pkg.tar.zst 2>/dev/null || true +if (resolve_package_file) 2>/dev/null; then + _t_fail "resolve_package_file: should reject when the cache has no matching file" +else + _t_pass "resolve_package_file: rejects when the cache has no matching file" +fi + +echo +echo "pacman_hook_stall_setup: ${PASS} passed, ${FAIL} failed" +((FAIL == 0)) diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_summary.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_summary.sh new file mode 100755 index 00000000..47f37aec --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_summary.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# Tests for lib/pacman_hook_stall_summary.sh: print_summary. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_harness.sh" + +# --- no stalls: prints counts/durations, no "Captured dumps" section ------- + +reset_state +RUNS=3 +STALLS=0 +durations=(5 12 8) +out="$(print_summary durations)" + +if [[ "$out" == *"transactions : 3"* ]]; then + _t_pass "print_summary: reports the transaction count" +else + _t_fail "print_summary: expected 'transactions : 3' in output" +fi + +if [[ "$out" == *"stalls : 0"* ]]; then + _t_pass "print_summary: reports zero stalls" +else + _t_fail "print_summary: expected 'stalls : 0' in output" +fi + +if [[ "$out" == *"durations : 5 12 8"* ]]; then + _t_pass "print_summary: lists durations in the caller's order" +else + _t_fail "print_summary: expected 'durations : 5 12 8' in output" +fi + +if [[ "$out" == *"min/median/max: 5s / 8s / 12s"* ]]; then + _t_pass "print_summary: computes min/median/max for an odd count" +else + _t_fail "print_summary: expected min/median/max '5s / 8s / 12s' in output" +fi + +if [[ "$out" != *"Captured dumps"* ]]; then + _t_pass "print_summary: omits the dumps section when STALLS is 0" +else + _t_fail "print_summary: should not print 'Captured dumps' when STALLS is 0" +fi + +# --- with stalls: lists the captured dump directories ----------------------- + +reset_state +RUNS=2 +STALLS=2 +mkdir -p "${OUT_DIR}/dumpA" "${OUT_DIR}/dumpB" +durations=(21 30) +out="$(print_summary durations)" + +if [[ "$out" == *"Captured dumps:"* && "$out" == *"dumpA"* && "$out" == *"dumpB"* ]]; then + _t_pass "print_summary: lists captured dump directories when STALLS > 0" +else + _t_fail "print_summary: expected both dump dirs listed, got: $out" +fi + +if [[ "$out" == *"min/median/max: 21s / 21s / 30s"* ]]; then + _t_pass "print_summary: computes the lower-median for an even count" +else + _t_fail "print_summary: expected min/median/max '21s / 21s / 30s' in output" +fi + +# --- a single duration: min == median == max -------------------------------- + +reset_state +RUNS=1 +STALLS=0 +durations=(42) +out="$(print_summary durations)" +if [[ "$out" == *"min/median/max: 42s / 42s / 42s"* ]]; then + _t_pass "print_summary: min/median/max collapse to the same value with one run" +else + _t_fail "print_summary: expected min/median/max '42s / 42s / 42s' in output" +fi + +# --- empty durations: the bash-side count guard doesn't crash --------------- + +reset_state +RUNS=0 +STALLS=0 +durations=() +_t_eq "0" "${#durations[@]}" "print_summary: durations starts empty" +if out="$(print_summary durations 2>&1)"; then + _t_pass "print_summary: handles an empty durations array without crashing" +else + _t_fail "print_summary: should not fail on an empty durations array: $out" +fi +if [[ "$out" != *"min/median/max"* ]]; then + _t_pass "print_summary: omits min/median/max when there are no durations" +else + _t_fail "print_summary: should not print min/median/max with no durations" +fi + +echo +echo "pacman_hook_stall_summary: ${PASS} passed, ${FAIL} failed" +((FAIL == 0)) diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_usage.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_usage.sh new file mode 100755 index 00000000..9848eb2d --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_usage.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# Tests for lib/pacman_hook_stall_usage.sh: usage. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_harness.sh" + +reset_state +# usage() calls `exit`, not `return` -- command substitution already runs it +# in a subshell, so the exit only ends that subshell, not this test script. +out="$(usage 2>&1)" +rc=$? +_t_eq "0" "$rc" "usage: exits 0" + +if [[ "$out" == *"Usage: ${SCRIPT_NAME}"* ]]; then + _t_pass "usage: prints the usage header with SCRIPT_NAME" +else + _t_fail "usage: expected the usage header, got: $out" +fi + +if [[ "$out" == *"--runs"* && "$out" == *"--with-load"* && "$out" == *"--watch"* ]]; then + _t_pass "usage: documents the runs/with-load/watch flags" +else + _t_fail "usage: expected --runs, --with-load and --watch documented" +fi + +if [[ "$out" == *"default: ${RUNS}"* ]]; then + _t_pass "usage: interpolates the current RUNS default" +else + _t_fail "usage: expected 'default: ${RUNS}' in the output" +fi + +echo +echo "pacman_hook_stall_usage: ${PASS} passed, ${FAIL} failed" +((FAIL == 0)) diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_watch.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_watch.sh new file mode 100755 index 00000000..7ff779aa --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/test_pacman_hook_stall_watch.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +# Tests for lib/pacman_hook_stall_watch.sh: watch_forever. +# +# watch_forever is `while true`, and its bare `sleep 1` inside the loop +# doesn't check sleep's exit status -- so the fake sleep's exit-after-N-calls +# lever can't break the loop from the inside when the call is guarded by +# `|| true` (that suppresses `set -e` for the whole function, not just its +# last statement). Instead: background it, let it run for a fixed real-time +# window, then kill it and assert on what it left behind ($DEV/calls, +# $OUT_DIR dump dirs) -- STALLS itself doesn't propagate out of a background +# subshell. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=pacman_hook_stall_harness.sh +. "${SCRIPT_DIR}/pacman_hook_stall_harness.sh" + +# run_watch_for SECONDS — start watch_forever in the background, let it run +# for SECONDS of wall-clock time (using the real sleep, not the fake one), +# then kill it. Dump directories and $DEV/calls reflect whatever it did. +run_watch_for() { + local secs="$1" + watch_forever >/dev/null 2>&1 & + local wpid=$! + disown "$wpid" 2>/dev/null || true + /usr/bin/sleep "$secs" + kill -9 "$wpid" 2>/dev/null || true + wait "$wpid" 2>/dev/null || true +} + +dump_count() { find "${OUT_DIR}" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l; } + +# --- log stays silent, no live pacman -> no stall captured ------------------ + +reset_state +: >"${PACMAN_LOG}" +STALL_TIMEOUT=1 +run_watch_for 2 +_t_eq "0" "$(dump_count)" "watch_forever: no stall captured when pgrep finds nothing" + +# --- log silent AND a matching pacman pid -> captures, and logs it ---------- + +reset_state +: >"${PACMAN_LOG}" +STALL_TIMEOUT=1 +echo "$$" >"${DEV}/pgrep_pid" +run_watch_for 2 +count="$(dump_count)" +if ((count >= 1)); then + _t_pass "watch_forever: captures a stall when pgrep finds a live pacman" +else + _t_fail "watch_forever: expected at least one dump dir, got $count" +fi + +if grep -q "captured a stalled transaction" "${DEV}/calls" 2>/dev/null; then + _t_pass "watch_forever: logs the capture via logger" +else + _t_fail "watch_forever: expected a logger call recording the capture" +fi + +# --- once armed, stays armed while still silent: no re-capture ------------- + +reset_state +: >"${PACMAN_LOG}" +STALL_TIMEOUT=1 +echo "$$" >"${DEV}/pgrep_pid" +run_watch_for 3 +count="$(dump_count)" +_t_eq "1" "$count" "watch_forever: re-arms only after log activity, not every idle tick" + +# --- log activity resets the silence timer: no capture with a high timeout - + +reset_state +: >"${PACMAN_LOG}" +STALL_TIMEOUT=100 +echo "$$" >"${DEV}/pgrep_pid" +watch_forever >/dev/null 2>&1 & +wpid=$! +disown "$wpid" 2>/dev/null || true +/usr/bin/sleep 0.5 +echo "line" >>"${PACMAN_LOG}" +/usr/bin/sleep 1 +kill -9 "$wpid" 2>/dev/null || true +wait "$wpid" 2>/dev/null || true +_t_eq "0" "$(dump_count)" "watch_forever: log activity prevents a capture" + +echo +echo "pacman_hook_stall_watch: ${PASS} passed, ${FAIL} failed" +((FAIL == 0))