From 347222b92119d7f678989dccafd293ccb6719e5f Mon Sep 17 00:00:00 2001 From: Krzysztof kuhy Rudnicki Date: Sat, 22 Aug 2026 14:40:31 +0200 Subject: [PATCH] Add fixes/lib run_all.sh; the previous handoff was wrong about that directory The handoff recorded fixes/lib as having "no harness at all". It already had 8 test files and 2 harnesses. What was missing is run_all.sh -- the subject is_covered() actually builds, and the only thing the gate measures through. All 8 test files pass through it. No jail_args beside it, deliberately: the harness confines every write to its own `mktemp -d` and intercepts external commands with a PATH stub dir, and grep finds no absolute write to /etc, /var or /usr in any of the tests. So it runs bare, like the transcribe suite. First measurement through it: pacman_hook_stall_usage.sh is 1/1 = 100.00%. That denominator is honest, not broken -- the lib is a single usage() whose body is one `cat < --- NEXT_SESSION_PROMPT.md | 218 ++++++++---------- .../fixes-lib-run-all-2026-08-22.json | 39 ++++ .../single_use/fixes/lib/tests/run_all.sh | 32 +++ 3 files changed, 170 insertions(+), 119 deletions(-) create mode 100644 docs/superpowers/evidence/fixes-lib-run-all-2026-08-22.json create mode 100755 linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh diff --git a/NEXT_SESSION_PROMPT.md b/NEXT_SESSION_PROMPT.md index 4767aceb..11e6da07 100644 --- a/NEXT_SESSION_PROMPT.md +++ b/NEXT_SESSION_PROMPT.md @@ -1,21 +1,44 @@ -# Next session: fix the coverage INSTRUMENT, then keep clearing the allowlist +# Next session: the instrument is FIXED — go clear the allowlist > **Paste this whole file into a fresh Claude session opened in `~/testsAndMisc`.** > It is self-contained. Do not go looking for the previous session's context. -## Status in one table +## What changed: the bottleneck is gone -| Thing | Number | -| ------------------------------- | -------------------------------------- | -| In-scope shell libs | **201** | -| Passing the gate today | **98** | -| Still allowlisted (exempt) | **103** | -| Directories with a test harness | 7 | -| Cleared this session | 2 (`unity_handler`, `transcribe_venv`) | +The previous handoff said "the instrument is the bottleneck, not the tests." +That is **no longer true**. Both kcov defects are fixed and pushed +(`81191f40`, `29b8c5a7`). The two libs that were pinned at absurd numbers were +never test-limited: -The gate itself is **built and ready to arm**. `is_covered()` reads `jail_args` -and passes `--fail-on-case-error`. Only the allowlist stands between here and -wiring it into pre-commit, pre-push and CI. CI is green. +| lib | was | now | +| ------------------------- | --------------- | ------------------- | +| `dwm_config.sh` | 35/73 = 47.95% | **65/66 = 98.48%** | +| `rpi_nc_install.sh` | 10/88 = 11.36% | **85/88 = 96.59%** | +| `rpi_nc_ca.sh` | 72/73 = 98.63% | **73/73 = 100.00%** | +| `aw_autostart.sh` | 68/82 = 82.93% | **80/82 = 97.56%** | +| `clean_audio_filters.sh` | 83/93 = 89.25% | **85/93 = 91.40%** | +| `dot_resolver_install.sh` | 39/39 = 100.00% | 39/39 = 100.00% | + +Nothing went down. **Every allowlisted lib is worth re-measuring before you +write a single new test** — several are probably already at or near 100%. + +## How the instrument works now (read before changing it) + +Coverage = `(PS4-traced lines | kcov hits) & (kcov line set - continuation lines)` + +Two passes over the same cases, in two fresh namespaces. They **cannot** be +merged: kcov's ptrace and bash's xtrace are mutually exclusive — under +`SHELLOPTS=xtrace` every line kcov recorded as `hits="1"` comes back +`hits="0"`, measured A/B. + +**The trace must never supply the denominator.** A trace reports only lines +that RAN, so trace-as-denominator makes every subject 100.00% and gates on +nothing. kcov's line set is what keeps the gate meaningful. + +Full detail, including the three findings that each yield a silently wrong +trace, is in `docs/kcov-under-report.md`. The load-bearing one: **bash under +`unshare --map-root-user` runs privileged and DISCARDS an inherited `PS4`**, +so tracing is delivered through a `BASH_ENV` file instead. ## The scoping answers are SETTLED. Do not re-ask them. @@ -23,141 +46,104 @@ Repo-wide gate (not a ratchet) · **100%** line coverage · **ALL** `.sh` · pre-commit **AND** pre-push **AND** CI · tests first, arm the gate last · hardest first. -## START HERE — the instrument is the bottleneck, not the tests +## START HERE — re-measure the allowlist -**kcov has two separate defects. One is fixable today; the other caps libs at -numbers no test can move.** Full detail and the seven DISPROVEN hypotheses are -in `docs/kcov-under-report.md` — read it first and do not retry them. - -- **(a) Wrong denominator.** Continuation lines inside a multi-line quoted - argument (a `perl -0777 -pe '...'` block) are counted as statements. A - `bash -x` tracer never reports them as executable at all. This is what holds - `dwm_config.sh` down. -- **(b) Wrong numerator.** Ordinary statements execute but are not recorded. - **Proven against an independent instrument:** driving `rpi_nc_install.sh` - under `PS4='+PS4:${BASH_SOURCE##*/}:${LINENO} '` with `set -x`, 55 lines - trace as executed, and **45 of them are lines kcov calls uncovered** — - including 61, 62, 63, 66, 68, 102, 118, 146. This is why that lib sits at - 11.36% while all 29 of its assertions pass. - -### Step 1 — fix (a) in `meta/scripts/shell_coverage_report.py` - -Exclude continuation lines of multi-line quoted arguments from the -denominator. It already parses the kcov XML and has the source path from the -`filename` attribute, so this is a local change. - -### Step 2 — fix (b) by replacing kcov's numerator with a PS4 tracer - -Inside the jail the trace must land on a path the caller controls: `/root` and -`/var` are both bind-mounted to throwaway dirs, and `shell_coverage_jail.sh` -sends every case's stdout to `/dev/null`. Bind a dedicated output dir, or -teach the jail to copy the trace out before teardown. - -### Acceptance test for ANY instrument change - -```bash -# must still be 39/39 = 100.00% ---measure dot_resolver_install.sh --min 100 -# must not regress below 98.63% ---measure rpi_nc_ca.sh -``` - -If either moves the wrong way, the change is wrong. Both are measured through -`run_all.sh`, never a single test file. - -## If the instrument work stalls, there is always forward motion - -Two directories have libs and no harness at all: - -``` -22 libs linux_configuration/scripts/single_use/fixes/lib/ - 9 libs phone_focus_mode/lib/ -41 libs linux_configuration/scripts/single_use/features/lib/ (harness exists) -``` - -Building a harness for `fixes/lib` unblocks 22 libs at once, the way this -session's `testsAndMisc-bash` harness unblocked 5. - -## The invocation that works +103 libs are allowlisted on numbers produced by a broken instrument. Sweep +them with the fixed one before writing tests. The invocation: ```bash bash -c ' ja=(); while IFS= read -r l; do ja+=("$l"); done \ - < <(grep -v "^#\|^$" linux_configuration/scripts/single_use/features/lib/tests/jail_args) -timeout 300s bash meta/scripts/shell_coverage_jail.sh \ - --subject linux_configuration/scripts/single_use/features/lib/tests/run_all.sh \ + < <(grep -v "^#\|^$" /tests/jail_args) +timeout 900s bash meta/scripts/shell_coverage_jail.sh \ + --subject /tests/run_all.sh \ "${ja[@]}" --measure .sh --min 1 --fail-on-case-error -- ""' ``` +Omit the `ja` array entirely for suites with no `jail_args` (transcribe, +fixes) — those confine their writes to a `mktemp -d` and run bare. + **Always measure through `run_all.sh`, never a single test file.** That is the -subject `is_covered()` builds, and the runner's number can be LOWER than the -suite's own — `rpi_nc_ca.sh` is 100% alone and 98.63% through the runner. +subject `is_covered()` builds. -A suite declares its jail needs in a **`jail_args`** file beside its -`run_all.sh`; `ci_mirror.sh`, `shell-tests.yml` and `is_covered()` all read it. -No marker means the suite runs bare. +## Directories and their harness state -## Measured so far (all through `run_all.sh`) +``` +41 libs linux_configuration/scripts/single_use/features/lib/ harness + jail_args + 5 libs .../misc/testsAndMisc-bash/lib/ harness, no jail_args +22 libs .../single_use/fixes/lib/ run_all.sh ADDED this session; 6 of 22 tested + 9 libs phone_focus_mode/lib/ NO harness +``` -| lib | coverage | state | -| ------------------------- | ------------------- | ------------- | -| `dot_resolver_install.sh` | **39/39 = 100.00%** | off allowlist | -| `unity_handler.sh` | **32/32 = 100.00%** | off allowlist | -| `transcribe_venv.sh` | **53/53 = 100.00%** | off allowlist | -| `rpi_nc_ca.sh` | 72/73 = 98.63% | allowlisted | -| `nc_php.sh` | 84/85 = 98.82% | allowlisted | -| `transcribe_pkgmgr.sh` | 49/50 = 98.00% | allowlisted | -| `transcribe_deps.sh` | 48/52 = 92.31% | allowlisted | -| `clean_audio_filters.sh` | 83/93 = 89.25% | allowlisted | -| `aw_autostart.sh` | 68/82 = 82.93% | allowlisted | -| `dwm_config.sh` | 35/73 = 47.95% | defect (a) | -| `rpi_nc_install.sh` | 10/88 = 11.36% | defect (b) | +`fixes/lib` was listed as "no harness at all" in the previous handoff. That +was wrong — 8 test files and 2 harnesses already existed; what was missing was +`run_all.sh`, which is what `is_covered()` measures through. It now exists and +all 8 test files pass through it. **16 of the 22 libs there still have no +tests at all** — that is the largest remaining block of real test-writing work. -The last two are instrument-limited, not test-limited. **Writing more tests -for them will not move the number** — their assertions already pass and their -writes provably land on disk. +## Two instrument bounds you must know + +**Defect (c): kcov's LINE SET is incomplete too.** The report now prints, per +subject, any line the trace saw run that kcov never listed: + +``` +outside kcov's line set (ran, but counted in neither half): 68, 70, 73, ... +``` + +This **cannot inflate a percentage**. A line reaches that list only because +the trace saw it EXECUTE, so folding them in only ever raises the number +(`aw_autostart.sh` 80/82 = 97.56% -> 91/93 = 97.85%). It can never hide an +untested line. What it does mean: such a subject is measured over a SUBSET of +its statements. Read the list before clearing that lib. + +`aw_autostart.sh` is the known case — kcov lists nothing after the +`{ ... } >>"$file"` command group at lines 61-65. + +**`nc_php.sh` does not measure at all.** "kcov instrumented no lines", on the +fixed instrument AND on the pre-fix one (checked at `078d3463` in a throwaway +worktree). Not a regression. The 84/85 = 98.82% in older notes does not +reproduce — treat it as unverified. ## Traps that cost real time - **`--fail-on-case-error` or you are flying blind.** The jail suppresses a suite's stdout, so assertions fail invisibly without it. -- **To find WHERE a suite dies, use an `exit ` sentinel.** Temporarily end - the suite with `exit 42`; the jail surfaces the code. The only way to tell a - tracing failure from an aborted suite. +- **To find WHERE a suite dies, use an `exit ` sentinel.** The jail + surfaces the code. The only way to tell a tracing failure from an abort. - **`rm -f "$TEST_TMPDIR/bin/foo"` does NOT hide foo.** bash caches executable - locations. Both harnesses ship `_t_unstub`, which also runs `hash -r`. Three - assertions in this repo were asserting the opposite of their own names. -- **A prepended stub dir cannot hide a real binary.** If the host genuinely has - the tool (`pacman`, `aw-qt`, `/usr/lib/libsndfile.so`), REPLACE PATH or jail - it, or the "not installed" case tests nothing. -- **A stub must materialise what the real tool creates.** A record-only stub in - a `[[ ! -s $file ]]` fallback chain silently exercises the NEXT branch. + locations. Use the harnesses' `_t_unstub`, which also runs `hash -r`. +- **A prepended stub dir cannot hide a real binary.** If the host genuinely + has the tool, REPLACE PATH or jail it. +- **A stub must materialise what the real tool creates.** A record-only stub + in a `[[ ! -s $file ]]` chain silently exercises the NEXT branch. - **A top-level stub function SHADOWS the subject's own** for every later assertion. Put redefinitions in a subshell. -- **`2>/dev/null` swallows a PS4 trace too** — it cost a bogus "0 lines traced" - reading before the redirect was noticed. -- **`shfmt` reformats after your own check passes** — re-check the 250-line cap - after staging. -- **Commits exceed a 2-minute foreground timeout.** Use `run_in_background`. +- **`shfmt` and `prettier` reformat after your own check passes** — re-check + the 250-line cap after staging, and expect the pre-push prettier hook to + reject markdown you hand-formatted. +- **Commits and pushes exceed a 2-minute foreground timeout.** Background them. - **Two-strike rule.** Two failed attempts at the same thing -> stop, document, keep the working state. ## Rules that will bite you -- **No suppressions, ever.** Fix SC2016/SC2155 at the source. +- **No suppressions, ever.** Fix SC2016/SC2155/EXE001/PLR2004 at the source. - **Every commit touching code needs evidence** in `docs/superpowers/evidence/-.json` (validate with `python3 meta/scripts/validate_evidence.py `), **plus a contract** in `docs/superpowers/contracts/` once **>=4 code files** are staged. - **Put the MEASURED number in it**, never a rounded or hoped-for one. -- **Stage narrowly.** A concurrent session had 8 unrelated files in the shared - index this session; `git restore --staged` them. +- **Stage narrowly.** Another session may be working in this repo at the same + time; `git restore --staged` anything you did not touch. - **New test files need the exec bit** — `git add --chmod=+x` only changes the INDEX; `chmod +x` the working tree too or the shebang hook fails. -- **`CI_GREEN_SKIP=1`** is the documented escape hatch when THIS commit fixes a - red baseline. Export it for the pre-push mirror too. - Work directly on `main`. `git stash` and branch creation are blocked. -- **Another Claude session may be working in this repo simultaneously.** + +## When the allowlist reaches zero + +Wire `check_shell_coverage.sh` into pre-commit, pre-push and CI. The gate is +already correct. Note it now runs each suite TWICE per lib, so an `--all` +sweep is roughly double what it was and belongs in CI, not a commit hook. ## Not caused by this campaign @@ -165,9 +151,3 @@ writes provably land on disk. `error: resolution-too-deep` installing `meta/requirements.txt`. Needs dependency pinning; deliberately excluded from `check_ci_green.sh`. Do not fold it into this work. - -## When the allowlist reaches zero - -Wire `check_shell_coverage.sh` into pre-commit, pre-push and CI. The gate is -already correct; it runs a full jailed suite per lib, so an `--all` sweep is -minutes and belongs in CI, not a commit hook. diff --git a/docs/superpowers/evidence/fixes-lib-run-all-2026-08-22.json b/docs/superpowers/evidence/fixes-lib-run-all-2026-08-22.json new file mode 100644 index 00000000..80e1c07b --- /dev/null +++ b/docs/superpowers/evidence/fixes-lib-run-all-2026-08-22.json @@ -0,0 +1,39 @@ +{ + "intent": "Give linux_configuration/scripts/single_use/fixes/lib a run_all.sh so its six already-tested libs become measurable by the coverage gate, which measures through a suite runner and not through single test files.", + "scope": [ + "linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh - new suite runner", + "NEXT_SESSION_PROMPT.md - rewritten; the instrument is no longer the bottleneck", + "Non-goal: writing tests for the 16 fixes/lib libs that have none", + "Non-goal: clearing any lib off the allowlist" + ], + "changes": [ + "run_all.sh follows the transcribe suite's shape: iterate test_*.sh in one process tree, count failures, exit non-zero if any failed.", + "Deliberately no jail_args file beside it. The harness confines every write to its own `mktemp -d` and intercepts external commands with a PATH stub dir, and grep found no absolute writes to /etc, /var or /usr in any test, so the suite runs bare rather than inside the namespace jail.", + "Corrects the previous handoff, which recorded fixes/lib as having 'no harness at all'. It already had 8 test files and 2 harnesses; the missing piece was the runner that is_covered() builds as its subject." + ], + "verification": [ + { + "command": "timeout 300 bash linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh", + "result": "pass", + "evidence": "All 8 test files pass through the runner, ending 'All fixes lib test files passed.' -- e.g. pacman_hook_stall_usage 4 passed 0 failed, pacman_hook_stall_watch 5 passed 0 failed." + }, + { + "command": "shell_coverage_jail.sh --subject fixes/lib/tests/run_all.sh --measure pacman_hook_stall_usage.sh --min 1 --fail-on-case-error", + "result": "pass", + "evidence": "1/1 lines = 100.00%. The lib is a single usage() whose body is one `cat <[[:space:]]*/etc|>[[:space:]]*/var|>[[:space:]]*/usr' fixes/lib/tests/*.sh", + "result": "pass", + "evidence": "No matches, confirming the suite writes nothing outside its mktemp -d and so needs no jail_args." + } + ], + "risks": [ + "16 of the 22 libs in fixes/lib still have no tests, so the runner unblocks measurement for 6 libs rather than all 22.", + "This suite is noticeably slower under the jail than the features suite, because the watch_forever tests exercise timing loops; a future --all sweep should expect that." + ], + "rollback": [ + "git revert this commit, or delete tests/run_all.sh; the eight test files remain runnable individually and nothing else references the runner yet.", + "After rollback, confirm the individual test files still pass by running one directly." + ] +} diff --git a/linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh b/linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh new file mode 100755 index 00000000..54f92b83 --- /dev/null +++ b/linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# lib/tests/run_all.sh — run every fixes lib test in one process tree. +# +# Also the subject handed to shell_coverage_jail.sh when measuring coverage of +# a lib in this directory: coverage of a lib is only complete across the whole +# suite, and is_covered() builds exactly this subject. Never measure a single +# test file -- the runner's number is the one that counts, and it can be lower +# than a suite's own. +# +# No jail_args file sits beside this runner, and that is deliberate: the +# harness confines every write to its own `mktemp -d` and intercepts each +# external command with a PATH stub dir, so these suites touch nothing outside +# it. Both ci_mirror.sh and the shell-tests workflow therefore run this runner +# directly rather than wrapping it in the namespace jail. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +failed=0 +for test_script in "${SCRIPT_DIR}"/test_*.sh; do + printf '\n### %s\n' "$(basename "$test_script")" + if ! "$test_script"; then + failed=$((failed + 1)) + fi +done + +if [[ $failed -gt 0 ]]; then + printf '\n%d test file(s) FAILED\n' "$failed" + exit 1 +fi + +printf '\nAll fixes lib test files passed.\n'