mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-22 03:42:19 +01:00
The instrument, not the tests, was the bottleneck. Coverage is now
(PS4-traced lines | kcov hits) & (kcov line set - continuation lines).
The trace deliberately does NOT supply the denominator. A trace reports only
lines that RAN, so using it for both halves would make every subject 100.00%
and arm a repo-wide gate that checks nothing.
Three measured findings, each of which yields a silently wrong trace:
1. kcov's ptrace and bash's xtrace cannot share a process. Under
SHELLOPTS=xtrace every line kcov recorded as hits="1" comes back hits="0"
-- reproduced A/B on one subject. Hence two passes, two namespaces.
2. SHELLOPTS is readonly inside bash, so `export SHELLOPTS=xtrace` fails and
tracing never turns on; and `set -x` does not reach child processes.
3. Decisive: bash under `unshare --map-root-user` runs PRIVILEGED and
DISCARDS an inherited PS4, falling back to "+ " while still honouring
xtrace. The trace loses its file:line prefix and every subject reads 0%.
Fixed by delivering PS4 and `set -x` through a BASH_ENV file, which also
solves child-process propagation.
A fourth trap, in the report: kcov records filename="dwm_config.sh", a bare
basename. The source path comes from the trace's ${BASH_SOURCE} or a unique
repo search; an ambiguous basename yields no exclusion rather than a guess.
Measured through run_all.sh, nothing down:
dot_resolver_install.sh 39/39 = 100.00% (acceptance, unchanged)
rpi_nc_ca.sh 73/73 = 100.00% (was 72/73 = 98.63%)
dwm_config.sh 65/66 = 98.48% (was 35/73 = 47.95%)
rpi_nc_install.sh 85/88 = 96.59% (was 10/88 = 11.36%)
rpi_nc_ca reaching 100% is the load-bearing check: its missing line 141 was
cross-process contagion, and the trace pass has no kcov in it to contaminate.
dwm_config's remaining line 39 is a `done < <(...)`. Left in the denominator
on purpose -- removing lines inflates coverage, the fail-open direction, so a
second exclusion rule needs per-line justification, not a pattern from one lib.
Both new Python modules hold 100% branch coverage (29 tests).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
71 lines
6.1 KiB
JSON
71 lines
6.1 KiB
JSON
{
|
|
"intent": "Make shell coverage measure the tests instead of the instrument: correct kcov's denominator (continuation lines of multi-line quoted arguments are not statements) and replace its unreliable numerator with a PS4 xtrace, so libs stop being capped at numbers no test can move.",
|
|
"scope": [
|
|
"meta/scripts/shell_coverage_jail.sh - runs two passes and reconciles their failures",
|
|
"meta/scripts/lib/shell_coverage_case_runner.sh - per-pass case execution; BASH_ENV trace delivery",
|
|
"meta/scripts/lib/shell_coverage_jail_setup.sh - trace dir, xtrace_env.sh, reconcile_pass_failures",
|
|
"meta/scripts/lib/shell_coverage_lines.py - new: statement/executed line classification",
|
|
"meta/scripts/shell_coverage_report.py - union arithmetic and source resolution",
|
|
"python_pkg/tests/test_shell_coverage_lines.py, python_pkg/tests/test_shell_coverage_report.py - new tests",
|
|
"docs/kcov-under-report.md - records the fix and the three root causes",
|
|
"Non-goal: wiring the gate into pre-commit/pre-push/CI (waits for the allowlist to reach zero)",
|
|
"Non-goal: excluding `done < <(...)` from the denominator (fail-open direction; needs per-line justification)"
|
|
],
|
|
"changes": [
|
|
"Two passes over the same cases in two fresh namespaces: kcov's ptrace and bash's xtrace are mutually exclusive, measured A/B -- under SHELLOPTS=xtrace every kcov hit collapses from 1 to 0. Pass one supplies the instrumentable line set, pass two the executed set.",
|
|
"Coverage = (PS4-traced lines UNION kcov hits) INTERSECT (kcov line set MINUS continuation lines). The trace deliberately does not supply the denominator: it can only report lines that ran, so trace-as-denominator would make every subject 100% and gate on nothing.",
|
|
"Trace delivered via a BASH_ENV file that assigns PS4 and runs `set -x` inside each shell. Required because bash under `unshare --map-root-user` runs privileged and DISCARDS an inherited PS4 (falling back to the default '+ ', losing the file:line prefix entirely), and because `set -x` does not propagate to child processes while SHELLOPTS is readonly inside bash.",
|
|
"Continuation lines detected by tracking single/double quote state across the source, honouring backslash escapes in double quotes only and treating an unquoted # as end-of-line.",
|
|
"Source path resolved from the trace's ${BASH_SOURCE} or a unique repo search, because kcov records only a bare basename; an ambiguous basename yields no exclusion rather than a wrong guess.",
|
|
"The trace pass failing more cases than the kcov pass is a hard error: a partially-traced case under-reports and looks exactly like the defect being fixed.",
|
|
"Traced lines absent from kcov's line set are counted and reported rather than silently dropped -- a non-zero count is a third defect the design cannot correct for."
|
|
],
|
|
"verification": [
|
|
{
|
|
"command": "shell_coverage_jail.sh --measure dot_resolver_install.sh --min 100 --fail-on-case-error (through run_all.sh)",
|
|
"result": "pass",
|
|
"evidence": "dot_resolver_install.sh: 39/39 lines = 100.00%, exit 0. Acceptance baseline unchanged by the instrument change."
|
|
},
|
|
{
|
|
"command": "shell_coverage_jail.sh --measure rpi_nc_ca.sh --min 100 --fail-on-case-error (through run_all.sh)",
|
|
"result": "pass",
|
|
"evidence": "rpi_nc_ca.sh: 73/73 lines = 100.00%, exit 0 -- was 72/73 = 98.63%. Line 141 was lost to cross-process contagion from test_dwm_config.sh; the trace pass has no kcov in it to contaminate. This is the discriminating check that the numerator fix landed."
|
|
},
|
|
{
|
|
"command": "shell_coverage_jail.sh --measure dwm_config.sh (through run_all.sh)",
|
|
"result": "pass",
|
|
"evidence": "35/73 = 47.95% -> 65/66 = 98.48%. Denominator lost exactly the 7 continuation lines 64-66 and 98-101; numerator rose from 35 to 65. Remaining uncovered line 39 is a `done < <(...)`, left in the denominator deliberately."
|
|
},
|
|
{
|
|
"command": "shell_coverage_jail.sh --measure rpi_nc_install.sh (through run_all.sh)",
|
|
"result": "pass",
|
|
"evidence": "10/88 = 11.36% -> 85/88 = 96.59%. The lib was never test-limited; its 29 assertions already passed."
|
|
},
|
|
{
|
|
"command": "python3 -m pytest python_pkg/tests/test_shell_coverage_lines.py python_pkg/tests/test_shell_coverage_report.py",
|
|
"result": "pass",
|
|
"evidence": "29 passed."
|
|
},
|
|
{
|
|
"command": "python3 -m coverage run --branch --source=meta/scripts -m pytest (the two new test modules); coverage report -m",
|
|
"result": "pass",
|
|
"evidence": "meta/scripts/lib/shell_coverage_lines.py 53 stmts 38 branches 100%; meta/scripts/shell_coverage_report.py 61 stmts 28 branches 100%."
|
|
},
|
|
{
|
|
"command": "pre-commit run --files (the seven touched code files)",
|
|
"result": "pass",
|
|
"evidence": "All hooks pass. Three real lint findings were fixed at source, not suppressed: EXE001 (shebang without exec bit), PLR2004 (magic argv index named _ARGV_WITH_TRACE), E501 (long test signatures wrapped)."
|
|
}
|
|
],
|
|
"risks": [
|
|
"Every measurement now runs the suite twice, so an --all sweep takes roughly double the wall time. That is why the gate belongs in CI rather than a commit hook.",
|
|
"xtrace is slower than an untraced run, so a case near its --timeout under kcov can exceed it under trace. This fails loudly via the pass-failure reconciliation rather than silently under-reporting.",
|
|
"A subject that runs `set +x`, inspects $-, or opens fd 199 would corrupt its own trace. Neither construct exists in any current subject (grepped), and fd 199 was chosen over a low fd for this reason.",
|
|
"The continuation-line detector is a quote-state scanner, not a bash parser. Over-detection would inflate coverage; it was validated against dwm_config.sh, where it flags exactly the 7 known lines and nothing else."
|
|
],
|
|
"rollback": [
|
|
"git revert the commit; the previous instrument is self-contained in the same five files.",
|
|
"After rollback, re-measure dot_resolver_install.sh (expect 39/39) and rpi_nc_ca.sh (expect the old 72/73 = 98.63%) to confirm the old behaviour is back, and restore the allowlist entries for any lib cleared on the strength of the new numbers."
|
|
]
|
|
}
|