mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 15:34:06 +01:00
Krzysztof kuhy Rudnicki
170f843481
The jail recorded each case's exit code but reported only timeouts, discarding every other non-zero status. A suite with a failing assertion exited 0 through the push gate -- the same blind spot that let an earlier session ship 2 failing assertions in the nc_php suite. Found by negative control, not by reading: a deliberately failing assertion still exited 0. Propagating that status unconditionally was wrong, and caught before landing: the jail's own usage example passes a `bogus` case that exits non-zero to cover an error path, so a clean 5/5 = 100.00% run started failing. That is the mode the 105-lib campaign runs in. So the behaviour sits behind --fail-on-case-error, default off, passed only by ci_mirror.sh and shell-tests.yml. Verified in all four quadrants: measurement + error case = 0, gate + error case = 1, gate + clean suite = 0, gate + broken assertion = 1. Both gates also pass --min 0: with no --measure the jail was measuring run_all.sh itself against its default --min 100 and failing the push at 54.55%, a coverage verdict neither gate asked for. ci_mirror.sh now prints the jail's output on failure, so a red gate can be diagnosed. The added flag pushed the jail over the 250-line cap, so the in-namespace case runner moved to lib/shell_coverage_case_runner.sh, following the existing lib/shell_coverage_jail_setup.sh split. All four quadrants were re-run after the extraction. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016sxxsDHWgdRhmdmzNMA6XM
22 lines
2.3 KiB
JSON
22 lines
2.3 KiB
JSON
{
|
|
"title": "Make the lib/tests gate run jail-requiring suites, and make it able to fail",
|
|
"objective": "Unblock the push gate and CI for suites that must execute inside shell_coverage_jail.sh, and close the fail-open hole that let a suite with a failing assertion exit 0 through that gate -- without changing the jail's behaviour in the measurement mode the 105-lib shell-coverage campaign depends on.",
|
|
"acceptance_criteria": [
|
|
"A suite declares its jail requirement as data (a jail_args file beside run_all.sh); neither ci_mirror.sh nor shell-tests.yml names any suite path, so a future jail-requiring suite needs no edit to either runner.",
|
|
"Both runners fail closed on a jail_args file that is present but empty: reported as a configuration error, never silently skipped.",
|
|
"Gate mode fails on a failing assertion: the features/lib suite with a deliberately broken _t_eq injected before _t_report exits non-zero.",
|
|
"Gate mode does not false-positive: the unmodified features/lib suite exits 0.",
|
|
"Measurement mode is unchanged: a subject with an error-path case that exits non-zero still reports its coverage and exits 0, so --fail-on-case-error is off by default.",
|
|
"The extracted jail_args flags reproduce the independently measured dot_resolver_install.sh 39/39 = 100.00%.",
|
|
"The five lib/tests suites that do not require the jail still run bare and pass.",
|
|
"Every touched shell file is within the 250-line cap and shellcheck-clean, with no suppressions added."
|
|
],
|
|
"out_of_scope": [
|
|
"The REFUSING guards in the individual test files: they are what stop a test run from writing the developer's real /etc, and are deliberately untouched.",
|
|
"Wiring check_shell_coverage.sh into any hook -- that arms only when the allowlist reaches zero.",
|
|
"The kcov mis-measurement of rpi_nc_install.sh, which is tracked separately as an instrument defect.",
|
|
"The unrelated python_pkg/token_audit/ files a concurrent session staged into the shared index."
|
|
],
|
|
"verifier": "Four quadrants via meta/scripts/shell_coverage_jail.sh: measurement+error-case = 0 with coverage printed; gate+error-case = 1; gate+clean features/lib suite = 0; gate+injected-failing-assertion = 1. Plus bash meta/scripts/check_file_length.sh --all showing no violation, shellcheck on the three shell files, and a successful git push through the ci-mirror pre-push hook."
|
|
}
|