mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 20:07:04 +01:00
Handoff: the 16 untested fixes/lib libs, ordered smallest-first
Retargets the handoff from 're-measure the allowlist' (done) to the work that is actually left: writing test suites for the 16 libs in fixes/lib that have none, to 100% line coverage. Includes the size-ordered table, the family groupings (bluetooth four, arch perf six, hosts guard two) so shared stubs get reused, the two entry scripts that source them, and what they actually call -- systemctl, bluetoothctl, pacman, sysctl, nvidia-*, apt-get. Says explicitly NOT to build a new harness: fixes/lib/tests already has a 205-line lib harness, a 142-line entry harness and eight test files to copy from, and jscpd fails the commit above 2% duplication. Records the three-step bar the five cleared libs met -- measure 100.00%, re-measure at --min 100, then run the real gate with no exemption -- so the next session does not have to re-derive why one reading is not enough. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
105d869776
commit
503ba1fcca
@ -1,201 +1,208 @@
|
||||
# Next session: the instrument is FIXED — go clear the allowlist
|
||||
# Next session: test the 16 untested `fixes/lib` libs to 100%
|
||||
|
||||
> **Paste this whole file into a fresh Claude session opened in `~/testsAndMisc`.**
|
||||
> It is self-contained. Do not go looking for the previous session's context.
|
||||
|
||||
## What changed: the bottleneck is gone
|
||||
## The job
|
||||
|
||||
The previous handoff said "the instrument is the bottleneck, not the tests."
|
||||
That is **no longer true**. Both kcov defects are fixed and pushed
|
||||
(`81191f40`, `29b8c5a7`). The two libs that were pinned at absurd numbers were
|
||||
never test-limited:
|
||||
`linux_configuration/scripts/single_use/fixes/lib/` has 22 libs. Six are done
|
||||
and off the allowlist. **Sixteen have no tests at all.** Write them, to 100%
|
||||
line coverage, and take each one off `meta/shell-coverage-allowlist.txt`.
|
||||
|
||||
| lib | was | now |
|
||||
| ------------------------- | --------------- | ------------------- |
|
||||
| `dwm_config.sh` | 35/73 = 47.95% | **65/66 = 98.48%** |
|
||||
| `rpi_nc_install.sh` | 10/88 = 11.36% | **85/88 = 96.59%** |
|
||||
| `rpi_nc_ca.sh` | 72/73 = 98.63% | **73/73 = 100.00%** |
|
||||
| `aw_autostart.sh` | 68/82 = 82.93% | **80/82 = 97.56%** |
|
||||
| `clean_audio_filters.sh` | 83/93 = 89.25% | **85/93 = 91.40%** |
|
||||
| `dot_resolver_install.sh` | 39/39 = 100.00% | 39/39 = 100.00% |
|
||||
That is the largest remaining block of real test-writing work in the campaign.
|
||||
|
||||
Nothing went down. **Every allowlisted lib is worth re-measuring before you
|
||||
write a single new test** — several are probably already at or near 100%.
|
||||
| lines | lib | family |
|
||||
| ----- | ------------------------- | -------------------------------------- |
|
||||
| 215 | `ubuntu_perf_fixes.sh` | ubuntu perf |
|
||||
| 214 | `bt_audio.sh` | bluetooth (`fix_bluetooth.sh`) |
|
||||
| 202 | `arch_hardware.sh` | arch perf (`optimize_arch_desktop.sh`) |
|
||||
| 180 | `nvidia_config.sh` | arch perf |
|
||||
| 179 | `thorium_repairs.sh` | standalone |
|
||||
| 166 | `arch_perf_probes.sh` | arch perf |
|
||||
| 161 | `hosts_guard_migrate.sh` | hosts guard |
|
||||
| 150 | `bt_adapter.sh` | bluetooth |
|
||||
| 141 | `hosts_guard_rollback.sh` | hosts guard |
|
||||
| 128 | `bt_pairing.sh` | bluetooth |
|
||||
| 126 | `arch_cpu.sh` | arch perf |
|
||||
| 122 | `arch_sysctl.sh` | arch perf |
|
||||
| 121 | `arch_perf_fixes.sh` | arch perf |
|
||||
| 103 | `arch_perf_report.sh` | arch perf |
|
||||
| 80 | `ubuntu_perf_more.sh` | ubuntu perf |
|
||||
| 76 | `bt_report.sh` | bluetooth |
|
||||
|
||||
## How the instrument works now (read before changing it)
|
||||
**Suggested order: smallest first, by family.** `bt_report.sh` (76) then the
|
||||
rest of the bluetooth four; the arch-perf six share `optimize_arch_desktop.sh`
|
||||
and will share stubs. Do not start with `ubuntu_perf_fixes.sh`.
|
||||
|
||||
## Everything you need already exists — do NOT build a new harness
|
||||
|
||||
`fixes/lib/tests/` is a working suite. It has:
|
||||
|
||||
- `run_all.sh` — the runner. **This is the subject the gate measures through.**
|
||||
- `pacman_hook_stall_harness.sh` (205 lines) — lib-level harness: `mktemp -d`
|
||||
sandbox, a `FAKE_BIN` PATH stub dir, `_t_eq` / `_t_pass` / `_t_fail`,
|
||||
and `_t_unstub` (which also runs `hash -r`).
|
||||
- `pacman_hook_stall_entry_harness.sh` (142 lines) — same for entry scripts.
|
||||
- Eight `test_*.sh` files to copy the shape from.
|
||||
|
||||
**jscpd fails the commit above 2% duplication**, so extend/parameterise these
|
||||
rather than cloning a harness per family. If a family genuinely needs
|
||||
different setup, add a small harness that _sources_ the existing one.
|
||||
|
||||
No `jail_args` file sits beside `run_all.sh`, deliberately: the harness
|
||||
confines every write to its own `mktemp -d` and stubs external commands on
|
||||
PATH. Keep it that way — the new tests must not write to `/etc`, `/var` or
|
||||
`/usr`. If a lib insists on writing an absolute path, stub the writer, do not
|
||||
add a bind mount.
|
||||
|
||||
## What these libs touch (drives your stubs)
|
||||
|
||||
`systemctl` (most common), `bluetoothctl`, `pacman`, `sysctl`, `sudo`,
|
||||
`nvidia-*`, `apt-get`. `systemctl`, `pacman` and `sudo` are already in the
|
||||
jail's `DEFAULT_SHIMS`; the rest you stub in `FAKE_BIN`.
|
||||
|
||||
Two entry scripts source these libs, and are the place to look for how each
|
||||
lib is really called:
|
||||
|
||||
- `linux_configuration/scripts/single_use/fixes/fix_bluetooth.sh`
|
||||
- `linux_configuration/scripts/single_use/fixes/optimize_arch_desktop.sh`
|
||||
|
||||
## The loop, per lib
|
||||
|
||||
```bash
|
||||
# 1. write tests/test_<lib>.sh, chmod +x it
|
||||
# 2. run the suite bare first -- much faster than the jail
|
||||
bash linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh
|
||||
|
||||
# 3. then measure through the runner, in the jail
|
||||
timeout 900s bash meta/scripts/shell_coverage_jail.sh \
|
||||
--subject linux_configuration/scripts/single_use/fixes/lib/tests/run_all.sh \
|
||||
--measure <lib>.sh --min 1 --fail-on-case-error -- ""
|
||||
|
||||
# 4. close the gap, repeat until 100.00%
|
||||
```
|
||||
|
||||
**Always measure through `run_all.sh`, never a single test file.** That is the
|
||||
subject `is_covered()` builds, and the runner's number can be LOWER than a
|
||||
suite's own.
|
||||
|
||||
## The bar for taking a lib off the allowlist
|
||||
|
||||
Do all three. This is the standard the five cleared libs met:
|
||||
|
||||
1. Measure 100.00% through `run_all.sh`.
|
||||
2. **Measure again at `--min 100`** so a shortfall is a non-zero exit rather
|
||||
than a number you might misread.
|
||||
3. Remove the line from `meta/shell-coverage-allowlist.txt`, then run the real
|
||||
gate with no exemption and confirm exit 0:
|
||||
```bash
|
||||
bash meta/scripts/check_shell_coverage.sh <path to lib>.sh
|
||||
# expect: "every checked shell library meets the coverage bar"
|
||||
```
|
||||
|
||||
Two matching runs matters: this directory's suites drive timing loops, and a
|
||||
timing-dependent suite can vary its line set between runs (`dwm_config.sh` was
|
||||
observed doing exactly that). A flaky 100% becomes a red CI job whose cause is
|
||||
invisible.
|
||||
|
||||
Allowlist is at **98** entries. Each lib you finish takes it down by one.
|
||||
|
||||
## The instrument is FIXED — trust its numbers now
|
||||
|
||||
Coverage = `(PS4-traced lines | kcov hits) & (kcov line set - continuation lines)`
|
||||
|
||||
Two passes over the same cases, in two fresh namespaces. They **cannot** be
|
||||
merged: kcov's ptrace and bash's xtrace are mutually exclusive — under
|
||||
`SHELLOPTS=xtrace` every line kcov recorded as `hits="1"` comes back
|
||||
`hits="0"`, measured A/B.
|
||||
Two passes, two namespaces, because kcov's ptrace and bash's xtrace cannot
|
||||
share a process. Full detail in `docs/kcov-under-report.md`. You should not
|
||||
need to touch it. If you do, the acceptance test is
|
||||
`dot_resolver_install.sh` = 39/39 and `rpi_nc_ca.sh` = 73/73, and **no lib's
|
||||
percentage may go DOWN** — a drop means a lost trace, not a stricter
|
||||
instrument.
|
||||
|
||||
**The trace must never supply the denominator.** A trace reports only lines
|
||||
that RAN, so trace-as-denominator makes every subject 100.00% and gates on
|
||||
nothing. kcov's line set is what keeps the gate meaningful.
|
||||
|
||||
Full detail, including the three findings that each yield a silently wrong
|
||||
trace, is in `docs/kcov-under-report.md`. The load-bearing one: **bash under
|
||||
`unshare --map-root-user` runs privileged and DISCARDS an inherited `PS4`**,
|
||||
so tracing is delivered through a `BASH_ENV` file instead.
|
||||
|
||||
## The scoping answers are SETTLED. Do not re-ask them.
|
||||
|
||||
Repo-wide gate (not a ratchet) · **100%** line coverage · **ALL** `.sh` ·
|
||||
pre-commit **AND** pre-push **AND** CI · tests first, arm the gate last ·
|
||||
hardest first.
|
||||
|
||||
## START HERE — re-measure the allowlist
|
||||
|
||||
98 libs are allowlisted on numbers produced by a broken instrument. Sweep
|
||||
them with the fixed one before writing tests. The invocation:
|
||||
|
||||
```bash
|
||||
bash -c '
|
||||
ja=(); while IFS= read -r l; do ja+=("$l"); done \
|
||||
< <(grep -v "^#\|^$" <suite>/tests/jail_args)
|
||||
timeout 900s bash meta/scripts/shell_coverage_jail.sh \
|
||||
--subject <suite>/tests/run_all.sh \
|
||||
"${ja[@]}" --measure <lib>.sh --min 1 --fail-on-case-error -- ""'
|
||||
```
|
||||
|
||||
Omit the `ja` array entirely for suites with no `jail_args` (transcribe,
|
||||
fixes) — those confine their writes to a `mktemp -d` and run bare.
|
||||
|
||||
**Always measure through `run_all.sh`, never a single test file.** That is the
|
||||
subject `is_covered()` builds.
|
||||
|
||||
## Directories and their harness state
|
||||
One bound to know: the report may print
|
||||
|
||||
```
|
||||
41 libs linux_configuration/scripts/single_use/features/lib/ harness + jail_args
|
||||
5 libs .../misc/testsAndMisc-bash/lib/ harness, no jail_args
|
||||
22 libs .../single_use/fixes/lib/ run_all.sh ADDED this session; 6 of 22 tested
|
||||
9 libs phone_focus_mode/lib/ NO harness
|
||||
outside kcov's line set (ran, but counted in neither half): 68, 70, ...
|
||||
```
|
||||
|
||||
`fixes/lib` was listed as "no harness at all" in the previous handoff. That
|
||||
was wrong — 8 test files and 2 harnesses already existed; what was missing was
|
||||
`run_all.sh`, which is what `is_covered()` measures through. It now exists and
|
||||
all 8 test files pass through it. **16 of the 22 libs there still have no
|
||||
tests at all** — that is the largest remaining block of real test-writing work.
|
||||
|
||||
## Already measured this session — do NOT redo these
|
||||
|
||||
All through `run_all.sh` on the fixed instrument. `fixes/lib` needs no
|
||||
`jail_args`; the features libs use the one beside their `run_all.sh`.
|
||||
|
||||
| lib | coverage | note |
|
||||
| ------------------------------ | ------------------- | ------------------------- |
|
||||
| `rpi_nc_ca.sh` | **73/73 = 100.00%** | measured 3x, solid |
|
||||
| `dot_resolver_install.sh` | **39/39 = 100.00%** | acceptance baseline |
|
||||
| `pacman_hook_stall_capture.sh` | **59/59 = 100.00%** | fixes/lib |
|
||||
| `pacman_hook_stall_load.sh` | **25/25 = 100.00%** | fixes/lib |
|
||||
| `pacman_hook_stall_summary.sh` | **18/18 = 100.00%** | fixes/lib; off-set: 36 |
|
||||
| `pacman_hook_stall_watch.sh` | **24/24 = 100.00%** | fixes/lib |
|
||||
| `pacman_hook_stall_usage.sh` | **1/1 = 100.00%** | body is one heredoc |
|
||||
| `dwm_config.sh` | 65/66 = 98.48% | line 39 is `done < <(…)` |
|
||||
| `pacman_hook_stall_setup.sh` | 26/27 = 96.30% | uncovered: 7 |
|
||||
| `aw_autostart.sh` | 80/82 = 97.56% | 11 off-set lines |
|
||||
| `rpi_nc_install.sh` | 85/88 = 96.59% | uncovered: 70, 202, 216 |
|
||||
| `transcribe_pkgmgr.sh` | 49/50 = 98.00% | uncovered: 42 |
|
||||
| `clean_audio_filters.sh` | 85/93 = 91.40% | uncovered incl. 52-58 |
|
||||
| `transcribe_deps.sh` | 48/52 = 92.31% | uncovered: 19, 52, 53, 55 |
|
||||
|
||||
**Five were CLEARED this session** (`83de41f4`): the pacman_hook_stall libs
|
||||
`capture`, `load`, `summary`, `usage` and `watch`. Each was measured twice,
|
||||
the second time gating at `--min 100`, then confirmed through the real
|
||||
`check_shell_coverage.sh` with no exemption — all exit 0. **Allowlist:
|
||||
103 -> 98.**
|
||||
|
||||
`rpi_nc_ca.sh` and `dot_resolver_install.sh` are also at 100.00% and measured
|
||||
3x / 2x, but they live in `features/lib`, where 39 other libs are still
|
||||
allowlisted. Removing them is safe by the same standard whenever you want.
|
||||
|
||||
The bar to reuse: **two matching runs, the second at `--min 100`, then the
|
||||
real gate.** `fixes/lib`'s suites drive `watch_forever` timing loops, and a
|
||||
timing-dependent suite can vary its line set run to run (`dwm_config.sh` was
|
||||
seen to). A flaky 100% becomes a red CI job whose cause is invisible.
|
||||
|
||||
Allowlist entries are **static per-file lines**, verified: adding
|
||||
`fixes/lib/tests/run_all.sh` did NOT start failing the 16 untested libs in
|
||||
that directory — `check_shell_coverage.sh <one of them>` still exits 0,
|
||||
because each is listed explicitly. The allowlist header's "every file in that
|
||||
directory is enforced" describes intent, not current mechanics.
|
||||
|
||||
## Two instrument bounds you must know
|
||||
|
||||
**Defect (c): kcov's LINE SET is incomplete too.** The report now prints, per
|
||||
subject, any line the trace saw run that kcov never listed:
|
||||
|
||||
```
|
||||
outside kcov's line set (ran, but counted in neither half): 68, 70, 73, ...
|
||||
```
|
||||
|
||||
This **cannot inflate a percentage**. A line reaches that list only because
|
||||
the trace saw it EXECUTE, so folding them in only ever raises the number
|
||||
(`aw_autostart.sh` 80/82 = 97.56% -> 91/93 = 97.85%). It can never hide an
|
||||
untested line. What it does mean: such a subject is measured over a SUBSET of
|
||||
its statements. Read the list before clearing that lib.
|
||||
|
||||
`aw_autostart.sh` is the known case — kcov lists nothing after the
|
||||
`{ ... } >>"$file"` command group at lines 61-65.
|
||||
|
||||
**`nc_php.sh` does not measure at all.** "kcov instrumented no lines", on the
|
||||
fixed instrument AND on the pre-fix one (checked at `078d3463` in a throwaway
|
||||
worktree). Not a regression. The 84/85 = 98.82% in older notes does not
|
||||
reproduce — treat it as unverified.
|
||||
That **cannot inflate a number** — such a line is by construction one the
|
||||
trace saw execute, so folding it in only ever raises the figure. It does mean
|
||||
the lib is measured over a subset of its statements. Read the list before
|
||||
clearing that lib; if the lines are ordinary statements, say so in the
|
||||
evidence.
|
||||
|
||||
## Traps that cost real time
|
||||
|
||||
- **`--fail-on-case-error` or you are flying blind.** The jail suppresses a
|
||||
- **Never edit the working tree while a measurement runs.** The jail
|
||||
fingerprints `git status --porcelain` and aborts with "a write escaped the
|
||||
jail". Editing an unrelated file trips it. FALSE alarm; cost two runs last
|
||||
session, the second time after it had already been diagnosed. Commit before
|
||||
starting a sweep; use the scratchpad for notes.
|
||||
- **A sweep's output file buffers.** `pgrep -x kcov` is the liveness signal,
|
||||
not file growth. A measurement silent for minutes is usually still running;
|
||||
this directory's suites take 30s+ per kcov pass, and there are two passes.
|
||||
- **Do not `pgrep -f shell_coverage_jail` in a wait loop** — the loop's own
|
||||
command line matches, so it never exits.
|
||||
- **`--fail-on-case-error` or you are flying blind.** The jail suppresses the
|
||||
suite's stdout, so assertions fail invisibly without it.
|
||||
- **To find WHERE a suite dies, use an `exit <n>` sentinel.** The jail
|
||||
surfaces the code. The only way to tell a tracing failure from an abort.
|
||||
- **`rm -f "$TEST_TMPDIR/bin/foo"` does NOT hide foo.** bash caches executable
|
||||
locations. Use the harnesses' `_t_unstub`, which also runs `hash -r`.
|
||||
locations. Use `_t_unstub`, which also runs `hash -r`.
|
||||
- **A prepended stub dir cannot hide a real binary.** If the host genuinely
|
||||
has the tool, REPLACE PATH or jail it.
|
||||
has the tool, REPLACE PATH or jail it, or the "not installed" case tests
|
||||
nothing.
|
||||
- **A stub must materialise what the real tool creates.** A record-only stub
|
||||
in a `[[ ! -s $file ]]` chain silently exercises the NEXT branch.
|
||||
- **A top-level stub function SHADOWS the subject's own** for every later
|
||||
assertion. Put redefinitions in a subshell.
|
||||
- **`shfmt` and `prettier` reformat after your own check passes** — re-check
|
||||
the 250-line cap after staging, and expect the pre-push prettier hook to
|
||||
reject markdown you hand-formatted.
|
||||
- **`shfmt` and `prettier` reformat after your check passes** — re-check the
|
||||
250-line cap after staging, and expect the pre-push prettier hook to reject
|
||||
hand-formatted markdown.
|
||||
- **Commits and pushes exceed a 2-minute foreground timeout.** Background them.
|
||||
- **Never edit the working tree while a measurement runs.** The jail
|
||||
fingerprints `git status --porcelain` before each case and aborts with
|
||||
"the subject modified the working tree; a write escaped the jail" if it
|
||||
changed. Editing an unrelated file mid-run produces that error and it is a
|
||||
FALSE alarm — cost two runs this session, twice.
|
||||
- **A sweep's output file buffers.** `pgrep -f shell_coverage_jail` is the
|
||||
liveness signal, not file growth. A measurement that has printed nothing for
|
||||
minutes is usually still running; `fixes/lib`'s timing-loop suites take 30s+
|
||||
per kcov pass.
|
||||
- **Two-strike rule.** Two failed attempts at the same thing -> stop, document,
|
||||
keep the working state.
|
||||
|
||||
## Rules that will bite you
|
||||
|
||||
- **No suppressions, ever.** Fix SC2016/SC2155/EXE001/PLR2004 at the source.
|
||||
- **Never weaken an assertion to move a number.** If a percentage looks absurd
|
||||
while assertions pass, suspect the instrument and re-measure.
|
||||
- **Every commit touching code needs evidence** in
|
||||
`docs/superpowers/evidence/<slug>-<date>.json` (validate with
|
||||
`python3 meta/scripts/validate_evidence.py <file>`), **plus a contract** in
|
||||
`docs/superpowers/contracts/` once **>=4 code files** are staged.
|
||||
- **Put the MEASURED number in it**, never a rounded or hoped-for one.
|
||||
- **Stage narrowly.** Another session may be working in this repo at the same
|
||||
time; `git restore --staged` anything you did not touch.
|
||||
- **New test files need the exec bit** — `git add --chmod=+x` only changes the
|
||||
INDEX; `chmod +x` the working tree too or the shebang hook fails.
|
||||
- **Stage narrowly.** Another session may be working in this repo at the same
|
||||
time; `git restore --staged` anything you did not touch.
|
||||
- Work directly on `main`. `git stash` and branch creation are blocked.
|
||||
|
||||
## When the allowlist reaches zero
|
||||
## State as of 2026-08-22
|
||||
|
||||
Wire `check_shell_coverage.sh` into pre-commit, pre-push and CI. The gate is
|
||||
already correct. Note it now runs each suite TWICE per lib, so an `--all`
|
||||
sweep is roughly double what it was and belongs in CI, not a commit hook.
|
||||
Allowlist **98**. Instrument fixed and pushed (`81191f40`, `29b8c5a7`). Five
|
||||
libs cleared (`83de41f4`). `fixes/lib/tests/run_all.sh` added (`347222b9`).
|
||||
|
||||
Already measured — do NOT redo:
|
||||
|
||||
| lib | coverage |
|
||||
| ---------------------------- | ------------------- |
|
||||
| `rpi_nc_ca.sh` | **73/73 = 100.00%** |
|
||||
| `dot_resolver_install.sh` | **39/39 = 100.00%** |
|
||||
| `dwm_config.sh` | 65/66 = 98.48% |
|
||||
| `pacman_hook_stall_setup.sh` | 26/27 = 96.30% |
|
||||
| `aw_autostart.sh` | 80/82 = 97.56% |
|
||||
| `rpi_nc_install.sh` | 85/88 = 96.59% |
|
||||
| `transcribe_pkgmgr.sh` | 49/50 = 98.00% |
|
||||
| `clean_audio_filters.sh` | 85/93 = 91.40% |
|
||||
| `transcribe_deps.sh` | 48/52 = 92.31% |
|
||||
|
||||
`rpi_nc_ca.sh` and `dot_resolver_install.sh` are at 100% and could come off
|
||||
the allowlist whenever you want, by the three-step bar above.
|
||||
|
||||
**`nc_php.sh` does not measure at all** — "kcov instrumented no lines", on the
|
||||
pre-fix instrument too (checked at `078d3463` in a throwaway worktree). Not a
|
||||
regression. Its old 84/85 = 98.82% does not reproduce; treat as unverified.
|
||||
|
||||
## Not caused by this campaign
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user