mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 20:27:02 +01:00
Correct the cause of rpi_nc_ca.sh's missing line: it is cross-process
The previous commit attributed rpi_nc_ca.sh's 72/73 to a heredoc BODY being counted in the denominator. That was wrong. Line 141 is `cat <<'EOF'` -- the heredoc's OPENING line, an ordinary statement. The body starts at 142. The real cause is that the kcov under-report is contagious across processes sharing one jail. Bisected over the five sibling suites: run test_dwm_config.sh before test_rpi_nc_ca.sh and line 141 is lost; run test_aw_autostart.sh or test_dot_resolver_install.sh first and it is kept. Alone the CA suite reports 73/73 = 100.00%; through run_all.sh it reports 72/73 = 98.63%. Both figures reproduce twice, so this is deterministic contamination rather than flakiness. Nothing is untested: with dwm_config running first, the CA suite still exits 0 with every assertion passing, show_help included. The line executes and is not traced. A heredoc-body detector was prototyped and deleted once it showed line 141 is not a body line -- a fix aimed at the wrong defect. The operational rule this implies is now written down: is_covered() measures through run_all.sh, so the runner's number is the lib's coverage, and a suite's own single-file figure must never be reported in its place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016sxxsDHWgdRhmdmzNMA6XM
This commit is contained in:
parent
92678c9506
commit
b830734cad
@ -216,6 +216,20 @@ Note the failure is _silent and one-directional_: it under-reports, so it can
|
||||
only ever keep a lib on the allowlist that deserves to come off. It cannot
|
||||
promote an untested lib. That is why the campaign can continue around it.
|
||||
|
||||
**The under-report is contagious across processes in one jail.** Measured on
|
||||
`rpi_nc_ca.sh`: run alone its suite reports **73/73 = 100.00%**, reproduced
|
||||
twice. Run through `run_all.sh` it reports **72/73 = 98.63%**, also twice --
|
||||
and bisecting the five sibling suites shows a single culprit, `test_dwm_config.sh`.
|
||||
Run `dwm_config` first and `rpi_nc_ca` loses line 141 (`cat <<'EOF'`, an
|
||||
ordinary statement); run any other sibling first and it keeps it. The CA
|
||||
suite's assertions all pass either way, confirmed with an exit sentinel, so
|
||||
nothing is actually untested.
|
||||
|
||||
This matters for the gate: `is_covered()` measures through `run_all.sh`, so
|
||||
the _runner's_ number is the one that counts, and it can be lower than the
|
||||
same suite's own number for reasons that have nothing to do with the tests.
|
||||
Never report the single-file figure as the lib's coverage.
|
||||
|
||||
**Use an `exit <n>` sentinel to tell the two apart.** The jail discards a
|
||||
case's stdout, so a suite's own report is invisible; but a non-zero exit is
|
||||
surfaced by `--fail-on-case-error`. Temporarily ending the suite with
|
||||
|
||||
@ -0,0 +1,43 @@
|
||||
{
|
||||
"intent": "Correct the recorded explanation for rpi_nc_ca.sh's 98.63%. The commit that landed it attributed the missing line to a heredoc BODY being counted in the denominator. That was wrong: line 141 is `cat <<'EOF'`, an ordinary statement, and the real cause is that the under-report is contagious across processes sharing one jail.",
|
||||
"scope": [
|
||||
"docs/shell-split-verification.md: record the cross-process finding and the rule that the runner's number is the authoritative one",
|
||||
"Non-goal: no code change -- the 98.63% figure and the allowlist state committed earlier are both correct, only the stated reason was wrong",
|
||||
"Non-goal: a heredoc-body filter in shell_coverage_report.py; it would not have fixed this line and is not justified by this evidence"
|
||||
],
|
||||
"changes": [
|
||||
"Bisected the five sibling suites: running test_dwm_config.sh before test_rpi_nc_ca.sh costs line 141; running test_aw_autostart.sh or test_dot_resolver_install.sh first does not. The culprit is a specific preceding process, not the construct on the line.",
|
||||
"A prototype heredoc-body detector was written, tested against the real file, and DELETED once it showed line 141 is the heredoc's OPENING line (a real statement at line 141; the body starts at 142). Keeping it would have been a fix aimed at the wrong defect.",
|
||||
"Recorded the operational rule this implies: is_covered() measures through run_all.sh, so the runner's figure is the lib's coverage. The same suite's single-file figure can be higher for reasons unrelated to the tests and must never be reported as the lib's number."
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"command": "measure rpi_nc_ca.sh alone vs through run_all.sh, twice each",
|
||||
"result": "pass",
|
||||
"evidence": "Alone: 73/73 = 100.00% both times. Through the runner: 72/73 = 98.63% both times. Stable in both modes, so this is deterministic contamination and not flakiness."
|
||||
},
|
||||
{
|
||||
"command": "run each preceding sibling suite, then the CA suite, measuring rpi_nc_ca.sh",
|
||||
"result": "pass",
|
||||
"evidence": "test_aw_autostart -> 100.00%; test_dot_resolver_install -> 100.00%; test_dwm_config -> 98.63%. A single culprit, isolated by bisection."
|
||||
},
|
||||
{
|
||||
"command": "run test_dwm_config.sh then test_rpi_nc_ca.sh under --fail-on-case-error",
|
||||
"result": "pass",
|
||||
"evidence": "No 'case exited' line: the CA suite exits 0 with every assertion passing, including the show_help assertion. Line 141 therefore EXECUTES and is simply not traced -- nothing is untested."
|
||||
},
|
||||
{
|
||||
"command": "each of the six suite files run individually under --fail-on-case-error",
|
||||
"result": "pass",
|
||||
"evidence": "failures=0 for all six. The interaction is purely in the measurement, not in the tests."
|
||||
}
|
||||
],
|
||||
"risks": [
|
||||
"Any lib's runner figure can be depressed by an unrelated sibling suite, so a lib may sit below the bar for reasons its own tests cannot fix. Until the root cause is found this caps what the campaign can certify -- but only downward, so no untested lib is ever promoted.",
|
||||
"The bisection identifies test_dwm_config.sh as today's culprit; adding another suite to the directory could change which files are affected. Re-measure every lib in a directory after adding a suite to it, as was done for the jail_args change."
|
||||
],
|
||||
"rollback": [
|
||||
"git revert the commit; the doc returns to the heredoc-body explanation, which is incorrect and would send the next session after the wrong fix.",
|
||||
"After rollback, re-run the bisection above to re-derive the correct cause before trusting the doc."
|
||||
]
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user