Krzysztof kuhy Rudnicki
347222b921
The handoff recorded fixes/lib as having "no harness at all". It already had
8 test files and 2 harnesses. What was missing is run_all.sh -- the subject
is_covered() actually builds, and the only thing the gate measures through.
All 8 test files pass through it.
No jail_args beside it, deliberately: the harness confines every write to its
own `mktemp -d` and intercepts external commands with a PATH stub dir, and
grep finds no absolute write to /etc, /var or /usr in any of the tests. So it
runs bare, like the transcribe suite.
First measurement through it: pacman_hook_stall_usage.sh is 1/1 = 100.00%.
That denominator is honest, not broken -- the lib is a single usage() whose
body is one `cat <<EOF` heredoc plus `exit 0`, and heredoc content is data
rather than statements.
16 of the 22 libs there still have no tests at all. That is now the largest
block of real test-writing work left in the campaign.
Also rewrites NEXT_SESSION_PROMPT.md, whose central claim ("the instrument is
the bottleneck, not the tests") stopped being true this session. The new one
leads with re-measuring the 103 allowlisted libs on the fixed instrument
before writing any new test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7.2 KiB
Next session: the instrument is FIXED — go clear the allowlist
Paste this whole file into a fresh Claude session opened in
~/testsAndMisc. It is self-contained. Do not go looking for the previous session's context.
What changed: the bottleneck is gone
The previous handoff said "the instrument is the bottleneck, not the tests."
That is no longer true. Both kcov defects are fixed and pushed
(81191f40, 29b8c5a7). The two libs that were pinned at absurd numbers were
never test-limited:
| lib | was | now |
|---|---|---|
dwm_config.sh |
35/73 = 47.95% | 65/66 = 98.48% |
rpi_nc_install.sh |
10/88 = 11.36% | 85/88 = 96.59% |
rpi_nc_ca.sh |
72/73 = 98.63% | 73/73 = 100.00% |
aw_autostart.sh |
68/82 = 82.93% | 80/82 = 97.56% |
clean_audio_filters.sh |
83/93 = 89.25% | 85/93 = 91.40% |
dot_resolver_install.sh |
39/39 = 100.00% | 39/39 = 100.00% |
Nothing went down. Every allowlisted lib is worth re-measuring before you write a single new test — several are probably already at or near 100%.
How the instrument works now (read before changing it)
Coverage = (PS4-traced lines | kcov hits) & (kcov line set - continuation lines)
Two passes over the same cases, in two fresh namespaces. They cannot be
merged: kcov's ptrace and bash's xtrace are mutually exclusive — under
SHELLOPTS=xtrace every line kcov recorded as hits="1" comes back
hits="0", measured A/B.
The trace must never supply the denominator. A trace reports only lines that RAN, so trace-as-denominator makes every subject 100.00% and gates on nothing. kcov's line set is what keeps the gate meaningful.
Full detail, including the three findings that each yield a silently wrong
trace, is in docs/kcov-under-report.md. The load-bearing one: bash under
unshare --map-root-user runs privileged and DISCARDS an inherited PS4,
so tracing is delivered through a BASH_ENV file instead.
The scoping answers are SETTLED. Do not re-ask them.
Repo-wide gate (not a ratchet) · 100% line coverage · ALL .sh ·
pre-commit AND pre-push AND CI · tests first, arm the gate last ·
hardest first.
START HERE — re-measure the allowlist
103 libs are allowlisted on numbers produced by a broken instrument. Sweep them with the fixed one before writing tests. The invocation:
bash -c '
ja=(); while IFS= read -r l; do ja+=("$l"); done \
< <(grep -v "^#\|^$" <suite>/tests/jail_args)
timeout 900s bash meta/scripts/shell_coverage_jail.sh \
--subject <suite>/tests/run_all.sh \
"${ja[@]}" --measure <lib>.sh --min 1 --fail-on-case-error -- ""'
Omit the ja array entirely for suites with no jail_args (transcribe,
fixes) — those confine their writes to a mktemp -d and run bare.
Always measure through run_all.sh, never a single test file. That is the
subject is_covered() builds.
Directories and their harness state
41 libs linux_configuration/scripts/single_use/features/lib/ harness + jail_args
5 libs .../misc/testsAndMisc-bash/lib/ harness, no jail_args
22 libs .../single_use/fixes/lib/ run_all.sh ADDED this session; 6 of 22 tested
9 libs phone_focus_mode/lib/ NO harness
fixes/lib was listed as "no harness at all" in the previous handoff. That
was wrong — 8 test files and 2 harnesses already existed; what was missing was
run_all.sh, which is what is_covered() measures through. It now exists and
all 8 test files pass through it. 16 of the 22 libs there still have no
tests at all — that is the largest remaining block of real test-writing work.
Two instrument bounds you must know
Defect (c): kcov's LINE SET is incomplete too. The report now prints, per subject, any line the trace saw run that kcov never listed:
outside kcov's line set (ran, but counted in neither half): 68, 70, 73, ...
This cannot inflate a percentage. A line reaches that list only because
the trace saw it EXECUTE, so folding them in only ever raises the number
(aw_autostart.sh 80/82 = 97.56% -> 91/93 = 97.85%). It can never hide an
untested line. What it does mean: such a subject is measured over a SUBSET of
its statements. Read the list before clearing that lib.
aw_autostart.sh is the known case — kcov lists nothing after the
{ ... } >>"$file" command group at lines 61-65.
nc_php.sh does not measure at all. "kcov instrumented no lines", on the
fixed instrument AND on the pre-fix one (checked at 078d3463 in a throwaway
worktree). Not a regression. The 84/85 = 98.82% in older notes does not
reproduce — treat it as unverified.
Traps that cost real time
--fail-on-case-erroror you are flying blind. The jail suppresses a suite's stdout, so assertions fail invisibly without it.- To find WHERE a suite dies, use an
exit <n>sentinel. The jail surfaces the code. The only way to tell a tracing failure from an abort. rm -f "$TEST_TMPDIR/bin/foo"does NOT hide foo. bash caches executable locations. Use the harnesses'_t_unstub, which also runshash -r.- A prepended stub dir cannot hide a real binary. If the host genuinely has the tool, REPLACE PATH or jail it.
- A stub must materialise what the real tool creates. A record-only stub
in a
[[ ! -s $file ]]chain silently exercises the NEXT branch. - A top-level stub function SHADOWS the subject's own for every later assertion. Put redefinitions in a subshell.
shfmtandprettierreformat after your own check passes — re-check the 250-line cap after staging, and expect the pre-push prettier hook to reject markdown you hand-formatted.- Commits and pushes exceed a 2-minute foreground timeout. Background them.
- Two-strike rule. Two failed attempts at the same thing -> stop, document, keep the working state.
Rules that will bite you
- No suppressions, ever. Fix SC2016/SC2155/EXE001/PLR2004 at the source.
- Every commit touching code needs evidence in
docs/superpowers/evidence/<slug>-<date>.json(validate withpython3 meta/scripts/validate_evidence.py <file>), plus a contract indocs/superpowers/contracts/once >=4 code files are staged. - Put the MEASURED number in it, never a rounded or hoped-for one.
- Stage narrowly. Another session may be working in this repo at the same
time;
git restore --stagedanything you did not touch. - New test files need the exec bit —
git add --chmod=+xonly changes the INDEX;chmod +xthe working tree too or the shebang hook fails. - Work directly on
main.git stashand branch creation are blocked.
When the allowlist reaches zero
Wire check_shell_coverage.sh into pre-commit, pre-push and CI. The gate is
already correct. Note it now runs each suite TWICE per lib, so an --all
sweep is roughly double what it was and belongs in CI, not a commit hook.
Not caused by this campaign
Python tests CI has been failing since 2026-08-17 — pip hits
error: resolution-too-deep installing meta/requirements.txt. Needs
dependency pinning; deliberately excluded from check_ci_green.sh. Do not
fold it into this work.