mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-19 01:34:22 +01:00
Add fixes/lib run_all.sh; the previous handoff was wrong about that directory
The handoff recorded fixes/lib as having "no harness at all". It already had
8 test files and 2 harnesses. What was missing is run_all.sh -- the subject
is_covered() actually builds, and the only thing the gate measures through.
All 8 test files pass through it.
No jail_args beside it, deliberately: the harness confines every write to its
own `mktemp -d` and intercepts external commands with a PATH stub dir, and
grep finds no absolute write to /etc, /var or /usr in any of the tests. So it
runs bare, like the transcribe suite.
First measurement through it: pacman_hook_stall_usage.sh is 1/1 = 100.00%.
That denominator is honest, not broken -- the lib is a single usage() whose
body is one `cat <<EOF` heredoc plus `exit 0`, and heredoc content is data
rather than statements.
16 of the 22 libs there still have no tests at all. That is now the largest
block of real test-writing work left in the campaign.
Also rewrites NEXT_SESSION_PROMPT.md, whose central claim ("the instrument is
the bottleneck, not the tests") stopped being true this session. The new one
leads with re-measuring the 103 allowlisted libs on the fixed instrument
before writing any new test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
d58910d916
commit
347222b921
@ -1,21 +1,44 @@
|
||||
# Next session: fix the coverage INSTRUMENT, then keep clearing the allowlist
|
||||
# Next session: the instrument is FIXED — go clear the allowlist
|
||||
|
||||
> **Paste this whole file into a fresh Claude session opened in `~/testsAndMisc`.**
|
||||
> It is self-contained. Do not go looking for the previous session's context.
|
||||
|
||||
## Status in one table
|
||||
## What changed: the bottleneck is gone
|
||||
|
||||
| Thing | Number |
|
||||
| ------------------------------- | -------------------------------------- |
|
||||
| In-scope shell libs | **201** |
|
||||
| Passing the gate today | **98** |
|
||||
| Still allowlisted (exempt) | **103** |
|
||||
| Directories with a test harness | 7 |
|
||||
| Cleared this session | 2 (`unity_handler`, `transcribe_venv`) |
|
||||
The previous handoff said "the instrument is the bottleneck, not the tests."
|
||||
That is **no longer true**. Both kcov defects are fixed and pushed
|
||||
(`81191f40`, `29b8c5a7`). The two libs that were pinned at absurd numbers were
|
||||
never test-limited:
|
||||
|
||||
The gate itself is **built and ready to arm**. `is_covered()` reads `jail_args`
|
||||
and passes `--fail-on-case-error`. Only the allowlist stands between here and
|
||||
wiring it into pre-commit, pre-push and CI. CI is green.
|
||||
| lib | was | now |
|
||||
| ------------------------- | --------------- | ------------------- |
|
||||
| `dwm_config.sh` | 35/73 = 47.95% | **65/66 = 98.48%** |
|
||||
| `rpi_nc_install.sh` | 10/88 = 11.36% | **85/88 = 96.59%** |
|
||||
| `rpi_nc_ca.sh` | 72/73 = 98.63% | **73/73 = 100.00%** |
|
||||
| `aw_autostart.sh` | 68/82 = 82.93% | **80/82 = 97.56%** |
|
||||
| `clean_audio_filters.sh` | 83/93 = 89.25% | **85/93 = 91.40%** |
|
||||
| `dot_resolver_install.sh` | 39/39 = 100.00% | 39/39 = 100.00% |
|
||||
|
||||
Nothing went down. **Every allowlisted lib is worth re-measuring before you
|
||||
write a single new test** — several are probably already at or near 100%.
|
||||
|
||||
## How the instrument works now (read before changing it)
|
||||
|
||||
Coverage = `(PS4-traced lines | kcov hits) & (kcov line set - continuation lines)`
|
||||
|
||||
Two passes over the same cases, in two fresh namespaces. They **cannot** be
|
||||
merged: kcov's ptrace and bash's xtrace are mutually exclusive — under
|
||||
`SHELLOPTS=xtrace` every line kcov recorded as `hits="1"` comes back
|
||||
`hits="0"`, measured A/B.
|
||||
|
||||
**The trace must never supply the denominator.** A trace reports only lines
|
||||
that RAN, so trace-as-denominator makes every subject 100.00% and gates on
|
||||
nothing. kcov's line set is what keeps the gate meaningful.
|
||||
|
||||
Full detail, including the three findings that each yield a silently wrong
|
||||
trace, is in `docs/kcov-under-report.md`. The load-bearing one: **bash under
|
||||
`unshare --map-root-user` runs privileged and DISCARDS an inherited `PS4`**,
|
||||
so tracing is delivered through a `BASH_ENV` file instead.
|
||||
|
||||
## The scoping answers are SETTLED. Do not re-ask them.
|
||||
|
||||
@ -23,141 +46,104 @@ Repo-wide gate (not a ratchet) · **100%** line coverage · **ALL** `.sh` ·
|
||||
pre-commit **AND** pre-push **AND** CI · tests first, arm the gate last ·
|
||||
hardest first.
|
||||
|
||||
## START HERE — the instrument is the bottleneck, not the tests
|
||||
## START HERE — re-measure the allowlist
|
||||
|
||||
**kcov has two separate defects. One is fixable today; the other caps libs at
|
||||
numbers no test can move.** Full detail and the seven DISPROVEN hypotheses are
|
||||
in `docs/kcov-under-report.md` — read it first and do not retry them.
|
||||
|
||||
- **(a) Wrong denominator.** Continuation lines inside a multi-line quoted
|
||||
argument (a `perl -0777 -pe '...'` block) are counted as statements. A
|
||||
`bash -x` tracer never reports them as executable at all. This is what holds
|
||||
`dwm_config.sh` down.
|
||||
- **(b) Wrong numerator.** Ordinary statements execute but are not recorded.
|
||||
**Proven against an independent instrument:** driving `rpi_nc_install.sh`
|
||||
under `PS4='+PS4:${BASH_SOURCE##*/}:${LINENO} '` with `set -x`, 55 lines
|
||||
trace as executed, and **45 of them are lines kcov calls uncovered** —
|
||||
including 61, 62, 63, 66, 68, 102, 118, 146. This is why that lib sits at
|
||||
11.36% while all 29 of its assertions pass.
|
||||
|
||||
### Step 1 — fix (a) in `meta/scripts/shell_coverage_report.py`
|
||||
|
||||
Exclude continuation lines of multi-line quoted arguments from the
|
||||
denominator. It already parses the kcov XML and has the source path from the
|
||||
`filename` attribute, so this is a local change.
|
||||
|
||||
### Step 2 — fix (b) by replacing kcov's numerator with a PS4 tracer
|
||||
|
||||
Inside the jail the trace must land on a path the caller controls: `/root` and
|
||||
`/var` are both bind-mounted to throwaway dirs, and `shell_coverage_jail.sh`
|
||||
sends every case's stdout to `/dev/null`. Bind a dedicated output dir, or
|
||||
teach the jail to copy the trace out before teardown.
|
||||
|
||||
### Acceptance test for ANY instrument change
|
||||
|
||||
```bash
|
||||
# must still be 39/39 = 100.00%
|
||||
--measure dot_resolver_install.sh --min 100
|
||||
# must not regress below 98.63%
|
||||
--measure rpi_nc_ca.sh
|
||||
```
|
||||
|
||||
If either moves the wrong way, the change is wrong. Both are measured through
|
||||
`run_all.sh`, never a single test file.
|
||||
|
||||
## If the instrument work stalls, there is always forward motion
|
||||
|
||||
Two directories have libs and no harness at all:
|
||||
|
||||
```
|
||||
22 libs linux_configuration/scripts/single_use/fixes/lib/
|
||||
9 libs phone_focus_mode/lib/
|
||||
41 libs linux_configuration/scripts/single_use/features/lib/ (harness exists)
|
||||
```
|
||||
|
||||
Building a harness for `fixes/lib` unblocks 22 libs at once, the way this
|
||||
session's `testsAndMisc-bash` harness unblocked 5.
|
||||
|
||||
## The invocation that works
|
||||
103 libs are allowlisted on numbers produced by a broken instrument. Sweep
|
||||
them with the fixed one before writing tests. The invocation:
|
||||
|
||||
```bash
|
||||
bash -c '
|
||||
ja=(); while IFS= read -r l; do ja+=("$l"); done \
|
||||
< <(grep -v "^#\|^$" linux_configuration/scripts/single_use/features/lib/tests/jail_args)
|
||||
timeout 300s bash meta/scripts/shell_coverage_jail.sh \
|
||||
--subject linux_configuration/scripts/single_use/features/lib/tests/run_all.sh \
|
||||
< <(grep -v "^#\|^$" <suite>/tests/jail_args)
|
||||
timeout 900s bash meta/scripts/shell_coverage_jail.sh \
|
||||
--subject <suite>/tests/run_all.sh \
|
||||
"${ja[@]}" --measure <lib>.sh --min 1 --fail-on-case-error -- ""'
|
||||
```
|
||||
|
||||
Omit the `ja` array entirely for suites with no `jail_args` (transcribe,
|
||||
fixes) — those confine their writes to a `mktemp -d` and run bare.
|
||||
|
||||
**Always measure through `run_all.sh`, never a single test file.** That is the
|
||||
subject `is_covered()` builds, and the runner's number can be LOWER than the
|
||||
suite's own — `rpi_nc_ca.sh` is 100% alone and 98.63% through the runner.
|
||||
subject `is_covered()` builds.
|
||||
|
||||
A suite declares its jail needs in a **`jail_args`** file beside its
|
||||
`run_all.sh`; `ci_mirror.sh`, `shell-tests.yml` and `is_covered()` all read it.
|
||||
No marker means the suite runs bare.
|
||||
## Directories and their harness state
|
||||
|
||||
## Measured so far (all through `run_all.sh`)
|
||||
```
|
||||
41 libs linux_configuration/scripts/single_use/features/lib/ harness + jail_args
|
||||
5 libs .../misc/testsAndMisc-bash/lib/ harness, no jail_args
|
||||
22 libs .../single_use/fixes/lib/ run_all.sh ADDED this session; 6 of 22 tested
|
||||
9 libs phone_focus_mode/lib/ NO harness
|
||||
```
|
||||
|
||||
| lib | coverage | state |
|
||||
| ------------------------- | ------------------- | ------------- |
|
||||
| `dot_resolver_install.sh` | **39/39 = 100.00%** | off allowlist |
|
||||
| `unity_handler.sh` | **32/32 = 100.00%** | off allowlist |
|
||||
| `transcribe_venv.sh` | **53/53 = 100.00%** | off allowlist |
|
||||
| `rpi_nc_ca.sh` | 72/73 = 98.63% | allowlisted |
|
||||
| `nc_php.sh` | 84/85 = 98.82% | allowlisted |
|
||||
| `transcribe_pkgmgr.sh` | 49/50 = 98.00% | allowlisted |
|
||||
| `transcribe_deps.sh` | 48/52 = 92.31% | allowlisted |
|
||||
| `clean_audio_filters.sh` | 83/93 = 89.25% | allowlisted |
|
||||
| `aw_autostart.sh` | 68/82 = 82.93% | allowlisted |
|
||||
| `dwm_config.sh` | 35/73 = 47.95% | defect (a) |
|
||||
| `rpi_nc_install.sh` | 10/88 = 11.36% | defect (b) |
|
||||
`fixes/lib` was listed as "no harness at all" in the previous handoff. That
|
||||
was wrong — 8 test files and 2 harnesses already existed; what was missing was
|
||||
`run_all.sh`, which is what `is_covered()` measures through. It now exists and
|
||||
all 8 test files pass through it. **16 of the 22 libs there still have no
|
||||
tests at all** — that is the largest remaining block of real test-writing work.
|
||||
|
||||
The last two are instrument-limited, not test-limited. **Writing more tests
|
||||
for them will not move the number** — their assertions already pass and their
|
||||
writes provably land on disk.
|
||||
## Two instrument bounds you must know
|
||||
|
||||
**Defect (c): kcov's LINE SET is incomplete too.** The report now prints, per
|
||||
subject, any line the trace saw run that kcov never listed:
|
||||
|
||||
```
|
||||
outside kcov's line set (ran, but counted in neither half): 68, 70, 73, ...
|
||||
```
|
||||
|
||||
This **cannot inflate a percentage**. A line reaches that list only because
|
||||
the trace saw it EXECUTE, so folding them in only ever raises the number
|
||||
(`aw_autostart.sh` 80/82 = 97.56% -> 91/93 = 97.85%). It can never hide an
|
||||
untested line. What it does mean: such a subject is measured over a SUBSET of
|
||||
its statements. Read the list before clearing that lib.
|
||||
|
||||
`aw_autostart.sh` is the known case — kcov lists nothing after the
|
||||
`{ ... } >>"$file"` command group at lines 61-65.
|
||||
|
||||
**`nc_php.sh` does not measure at all.** "kcov instrumented no lines", on the
|
||||
fixed instrument AND on the pre-fix one (checked at `078d3463` in a throwaway
|
||||
worktree). Not a regression. The 84/85 = 98.82% in older notes does not
|
||||
reproduce — treat it as unverified.
|
||||
|
||||
## Traps that cost real time
|
||||
|
||||
- **`--fail-on-case-error` or you are flying blind.** The jail suppresses a
|
||||
suite's stdout, so assertions fail invisibly without it.
|
||||
- **To find WHERE a suite dies, use an `exit <n>` sentinel.** Temporarily end
|
||||
the suite with `exit 42`; the jail surfaces the code. The only way to tell a
|
||||
tracing failure from an aborted suite.
|
||||
- **To find WHERE a suite dies, use an `exit <n>` sentinel.** The jail
|
||||
surfaces the code. The only way to tell a tracing failure from an abort.
|
||||
- **`rm -f "$TEST_TMPDIR/bin/foo"` does NOT hide foo.** bash caches executable
|
||||
locations. Both harnesses ship `_t_unstub`, which also runs `hash -r`. Three
|
||||
assertions in this repo were asserting the opposite of their own names.
|
||||
- **A prepended stub dir cannot hide a real binary.** If the host genuinely has
|
||||
the tool (`pacman`, `aw-qt`, `/usr/lib/libsndfile.so`), REPLACE PATH or jail
|
||||
it, or the "not installed" case tests nothing.
|
||||
- **A stub must materialise what the real tool creates.** A record-only stub in
|
||||
a `[[ ! -s $file ]]` fallback chain silently exercises the NEXT branch.
|
||||
locations. Use the harnesses' `_t_unstub`, which also runs `hash -r`.
|
||||
- **A prepended stub dir cannot hide a real binary.** If the host genuinely
|
||||
has the tool, REPLACE PATH or jail it.
|
||||
- **A stub must materialise what the real tool creates.** A record-only stub
|
||||
in a `[[ ! -s $file ]]` chain silently exercises the NEXT branch.
|
||||
- **A top-level stub function SHADOWS the subject's own** for every later
|
||||
assertion. Put redefinitions in a subshell.
|
||||
- **`2>/dev/null` swallows a PS4 trace too** — it cost a bogus "0 lines traced"
|
||||
reading before the redirect was noticed.
|
||||
- **`shfmt` reformats after your own check passes** — re-check the 250-line cap
|
||||
after staging.
|
||||
- **Commits exceed a 2-minute foreground timeout.** Use `run_in_background`.
|
||||
- **`shfmt` and `prettier` reformat after your own check passes** — re-check
|
||||
the 250-line cap after staging, and expect the pre-push prettier hook to
|
||||
reject markdown you hand-formatted.
|
||||
- **Commits and pushes exceed a 2-minute foreground timeout.** Background them.
|
||||
- **Two-strike rule.** Two failed attempts at the same thing -> stop, document,
|
||||
keep the working state.
|
||||
|
||||
## Rules that will bite you
|
||||
|
||||
- **No suppressions, ever.** Fix SC2016/SC2155 at the source.
|
||||
- **No suppressions, ever.** Fix SC2016/SC2155/EXE001/PLR2004 at the source.
|
||||
- **Every commit touching code needs evidence** in
|
||||
`docs/superpowers/evidence/<slug>-<date>.json` (validate with
|
||||
`python3 meta/scripts/validate_evidence.py <file>`), **plus a contract** in
|
||||
`docs/superpowers/contracts/` once **>=4 code files** are staged.
|
||||
- **Put the MEASURED number in it**, never a rounded or hoped-for one.
|
||||
- **Stage narrowly.** A concurrent session had 8 unrelated files in the shared
|
||||
index this session; `git restore --staged` them.
|
||||
- **Stage narrowly.** Another session may be working in this repo at the same
|
||||
time; `git restore --staged` anything you did not touch.
|
||||
- **New test files need the exec bit** — `git add --chmod=+x` only changes the
|
||||
INDEX; `chmod +x` the working tree too or the shebang hook fails.
|
||||
- **`CI_GREEN_SKIP=1`** is the documented escape hatch when THIS commit fixes a
|
||||
red baseline. Export it for the pre-push mirror too.
|
||||
- Work directly on `main`. `git stash` and branch creation are blocked.
|
||||
- **Another Claude session may be working in this repo simultaneously.**
|
||||
|
||||
## When the allowlist reaches zero
|
||||
|
||||
Wire `check_shell_coverage.sh` into pre-commit, pre-push and CI. The gate is
|
||||
already correct. Note it now runs each suite TWICE per lib, so an `--all`
|
||||
sweep is roughly double what it was and belongs in CI, not a commit hook.
|
||||
|
||||
## Not caused by this campaign
|
||||
|
||||
@ -165,9 +151,3 @@ writes provably land on disk.
|
||||
`error: resolution-too-deep` installing `meta/requirements.txt`. Needs
|
||||
dependency pinning; deliberately excluded from `check_ci_green.sh`. Do not
|
||||
fold it into this work.
|
||||
|
||||
## When the allowlist reaches zero
|
||||
|
||||
Wire `check_shell_coverage.sh` into pre-commit, pre-push and CI. The gate is
|
||||
already correct; it runs a full jailed suite per lib, so an `--all` sweep is
|
||||
minutes and belongs in CI, not a commit hook.
|
||||
|
||||
39
docs/superpowers/evidence/fixes-lib-run-all-2026-08-22.json
Normal file
39
docs/superpowers/evidence/fixes-lib-run-all-2026-08-22.json
Normal file
@ -0,0 +1,39 @@
|
||||
{
|
||||
"intent": "Give linux_configuration/scripts/single_use/fixes/lib a run_all.sh so its six already-tested libs become measurable by the coverage gate, which measures through a suite runner and not through single test files.",
|
||||
"scope": [
|
||||
"linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh - new suite runner",
|
||||
"NEXT_SESSION_PROMPT.md - rewritten; the instrument is no longer the bottleneck",
|
||||
"Non-goal: writing tests for the 16 fixes/lib libs that have none",
|
||||
"Non-goal: clearing any lib off the allowlist"
|
||||
],
|
||||
"changes": [
|
||||
"run_all.sh follows the transcribe suite's shape: iterate test_*.sh in one process tree, count failures, exit non-zero if any failed.",
|
||||
"Deliberately no jail_args file beside it. The harness confines every write to its own `mktemp -d` and intercepts external commands with a PATH stub dir, and grep found no absolute writes to /etc, /var or /usr in any test, so the suite runs bare rather than inside the namespace jail.",
|
||||
"Corrects the previous handoff, which recorded fixes/lib as having 'no harness at all'. It already had 8 test files and 2 harnesses; the missing piece was the runner that is_covered() builds as its subject."
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"command": "timeout 300 bash linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh",
|
||||
"result": "pass",
|
||||
"evidence": "All 8 test files pass through the runner, ending 'All fixes lib test files passed.' -- e.g. pacman_hook_stall_usage 4 passed 0 failed, pacman_hook_stall_watch 5 passed 0 failed."
|
||||
},
|
||||
{
|
||||
"command": "shell_coverage_jail.sh --subject fixes/lib/tests/run_all.sh --measure pacman_hook_stall_usage.sh --min 1 --fail-on-case-error",
|
||||
"result": "pass",
|
||||
"evidence": "1/1 lines = 100.00%. The lib is a single usage() whose body is one `cat <<EOF` heredoc plus `exit 0`; heredoc content is data, not statements, so a denominator of 1 is the honest reading. Its one off-set line is the `usage() {` declaration -- the benign class."
|
||||
},
|
||||
{
|
||||
"command": "grep -rn '>[[:space:]]*/etc|>[[:space:]]*/var|>[[:space:]]*/usr' fixes/lib/tests/*.sh",
|
||||
"result": "pass",
|
||||
"evidence": "No matches, confirming the suite writes nothing outside its mktemp -d and so needs no jail_args."
|
||||
}
|
||||
],
|
||||
"risks": [
|
||||
"16 of the 22 libs in fixes/lib still have no tests, so the runner unblocks measurement for 6 libs rather than all 22.",
|
||||
"This suite is noticeably slower under the jail than the features suite, because the watch_forever tests exercise timing loops; a future --all sweep should expect that."
|
||||
],
|
||||
"rollback": [
|
||||
"git revert this commit, or delete tests/run_all.sh; the eight test files remain runnable individually and nothing else references the runner yet.",
|
||||
"After rollback, confirm the individual test files still pass by running one directly."
|
||||
]
|
||||
}
|
||||
32
linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh
Executable file
32
linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh
Executable file
@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
# lib/tests/run_all.sh — run every fixes lib test in one process tree.
|
||||
#
|
||||
# Also the subject handed to shell_coverage_jail.sh when measuring coverage of
|
||||
# a lib in this directory: coverage of a lib is only complete across the whole
|
||||
# suite, and is_covered() builds exactly this subject. Never measure a single
|
||||
# test file -- the runner's number is the one that counts, and it can be lower
|
||||
# than a suite's own.
|
||||
#
|
||||
# No jail_args file sits beside this runner, and that is deliberate: the
|
||||
# harness confines every write to its own `mktemp -d` and intercepts each
|
||||
# external command with a PATH stub dir, so these suites touch nothing outside
|
||||
# it. Both ci_mirror.sh and the shell-tests workflow therefore run this runner
|
||||
# directly rather than wrapping it in the namespace jail.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
failed=0
|
||||
for test_script in "${SCRIPT_DIR}"/test_*.sh; do
|
||||
printf '\n### %s\n' "$(basename "$test_script")"
|
||||
if ! "$test_script"; then
|
||||
failed=$((failed + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ $failed -gt 0 ]]; then
|
||||
printf '\n%d test file(s) FAILED\n' "$failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf '\nAll fixes lib test files passed.\n'
|
||||
Loading…
Reference in New Issue
Block a user