testsAndMisc/docs/superpowers/evidence/jail-propagates-case-failure-2026-08-22.json

{
  "intent": "Make the jailed lib/tests gate actually able to fail. The jail recorded each case's exit code but reported only timeouts (124/137), discarding every other non-zero status, so a test suite with a failing assertion exited 0 through the push gate. A gate that cannot fail reports safe while checking nothing.",
  "scope": [
    "meta/scripts/shell_coverage_jail.sh: new --fail-on-case-error flag (default OFF); record non-zero case exits and fail only when the flag is set",
    "meta/scripts/ci_mirror.sh: pass --min 0 to the suite-running invocation",
    ".github/workflows/shell-tests.yml: same --min 0",
    "Non-goal: the coverage bar stays check_shell_coverage.sh's job, per-lib and --measure'd; this gate asks only 'did the assertions pass?'",
    "meta/scripts/lib/shell_coverage_case_runner.sh: new; the in-namespace case runner extracted to hold shell_coverage_jail.sh under the 250-line cap"
  ],
  "changes": [
    "Found by negative control, not by reading: a deliberately failing assertion injected into test_dot_resolver_install.sh still exited 0. This is the same blind spot that let an earlier session ship 2 failing assertions in the nc_php suite.",
    "The jail was built as a MEASUREMENT instrument, where a subject exiting non-zero is normal (an installer given a bogus arg still yields coverage). Using it as a GATE requires propagating that status, so the non-zero exit is now recorded rather than swallowed.",
    "Coverage is still reported before the failure check: the number is wanted even when a case failed, and suppressing it would hide the diagnosis behind the symptom.",
    "--min 0 on both gate invocations: with no --measure, the jail measured run_all.sh itself against its default --min 100 and failed the push at 54.55% -- a number neither gate asked for.",
    "Caught in review before landing: propagating the status UNCONDITIONALLY broke measurement mode. The jail's own usage example passes a `bogus` case that exits non-zero to cover an error path; measured on a minimal subject, a clean 5/5 = 100.00% run started exiting 1. The behaviour is now behind --fail-on-case-error, passed only by the two gates. Measurement mode -- what the 105-lib campaign runs -- is unchanged.",
    "ci_mirror.sh captures the jail's output and prints it on the failure path only. Discarding it left a red gate reporting 'suite failed: <runner>' with no warn: or FAIL: lines to diagnose from.",
    "The added flag pushed shell_coverage_jail.sh to 285 lines and the file-length-cap hook rejected the commit. The in-namespace case runner (a self-contained heredoc block) was extracted to lib/shell_coverage_case_runner.sh, following the existing lib/shell_coverage_jail_setup.sh split. Jail 240 lines, new lib 64. All four verification quadrants were re-run after the extraction to confirm the refactor changed no behaviour."
  ],
  "verification": [
    {
      "command": "shell_coverage_jail.sh --subject errsubj.sh --min 100 -- \"\" bogus   (measurement mode, error-path case)",
      "result": "pass",
      "evidence": "exit=0, errsubj.sh: 5/5 lines = 100.00%. Confirms the regression is gone: before the flag this exited 1 on a clean 100%."
    },
    {
      "command": "shell_coverage_jail.sh --subject errsubj.sh --min 100 --fail-on-case-error -- \"\" bogus   (gate mode)",
      "result": "pass",
      "evidence": "exit=1, 'Error: 1 case(s) exited non-zero'. Same subject, opposite verdict -- the flag is what separates the modes."
    },
    {
      "command": "jailed features/lib run_all.sh --min 0 --fail-on-case-error, unmodified suite",
      "result": "pass",
      "evidence": "exit=0. No false positive; a green suite passes the gate. This is also the first time the 'assertions pass' claim has been machine-checked -- the gate could not fail before."
    },
    {
      "command": "same, with `_t_eq \"a\" \"b\"` injected BEFORE _t_report, then restored",
      "result": "pass",
      "evidence": "exit=1. The negative control that proves the gate can fail. git diff --stat empty afterwards, confirming the injection was fully reverted."
    },
    {
      "command": "shellcheck meta/scripts/shell_coverage_jail.sh meta/scripts/ci_mirror.sh",
      "result": "pass",
      "evidence": "no findings; no suppressions added."
    },
    {
      "command": "git diff --cached --name-only",
      "result": "pass",
      "evidence": "only my files. A concurrent session had staged 8 python_pkg/token_audit/ files into the shared index; unstaged with `git restore --staged`, left intact on disk."
    },
    {
      "command": "bash meta/scripts/check_file_length.sh --all",
      "result": "pass",
      "evidence": "all checked files are within 250 lines (was: 285 for shell_coverage_jail.sh, +35 over)."
    },
    {
      "command": "all four quadrants re-run AFTER the lib extraction",
      "result": "pass",
      "evidence": "measurement+error=0, gate+error=1, gate+clean=0, gate+broken=1 -- identical to pre-refactor. shellcheck clean on all three shell files."
    }
  ],
  "risks": [
    "The two modes now differ by a flag. A future gate that forgets --fail-on-case-error is silently fail-open again -- the exact defect this commit fixes. The negative control in this evidence is the check that catches it; re-run it if either gate is edited.",
    "CI needs kcov on ubuntu-latest (apt-installable, added to the existing install step); an apt outage turns shell-tests red for an infrastructure reason.",
    "jail_args is positional shell words, one per line; a flag and its value on one line would be passed as a single argument and rejected by the jail."
  ],
  "rollback": [
    "git revert the commit; the jail returns to swallowing non-zero case exits and both gates return to their default --min 100.",
    "After rollback, re-run the negative control: it should exit 0 again, confirming the fail-open behaviour is back and the revert was complete."
  ]
}