mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 17:34:04 +01:00
Krzysztof kuhy Rudnicki
7f706adb11
63a5124 claimed "12 tests, 12 killed by at least one mutation", but the harness
that proved it lived in a session scratchpad. That left an unfalsifiable claim
about tests whose entire purpose is to not be vacuous -- and the facts it
encodes are exactly the ones a future edit to either hook would invalidate.
The mutations are per guard, not one whole-body gutting. Gutting a hook body
kills only the three positive tests, because a gutted hook emits neither a
launch nor any output, which is precisely what the nine negative tests expect.
Run that way the negative tests look vacuous when they are not; dropping each
guard in turn is what actually exercises them.
Two failure modes of the harness itself are handled. A mutation that leaves the
hook text unchanged now raises instead of quietly reporting "killed 0": a
behaviour-preserving edit to a hook, such as renaming a variable, would
otherwise turn a mutation into a no-op and weaken the proof without a word.
Verified by renaming count to n in a copy of the start hook -- the copy still
printed the correct banner, and the tool still refused it. And where the hooks
are not deployed at all the tool exits 0 rather than failing, since a CI runner
has no ~/.claude to mutate.
Wired into pre-commit scoped to the suite and the mutation definitions, the only
two files that can invalidate the proof. It costs ~40s, so it stays off
unrelated commits. It gates rather than warns: survivors mean exit 1.
One finding stands recorded: start-no-readable-guard kills nothing on its own.
The start hook's [[ ! -r "$STATE" ]] guard is redundant, since jq's stderr is
suppressed and an empty count already fails the arithmetic, so only the compound
mutation reaches those cases. The guard is left in place as defence in depth.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
48 lines
4.6 KiB
JSON
48 lines
4.6 KiB
JSON
{
|
|
"intent": "Make the non-vacuity of the session_autopsy hook tests re-verifiable and enforced, instead of a claim in a commit message. 63a5124 asserted '12 tests, 12 killed by at least one mutation', but the harness that proved it lived in a session scratchpad and would have vanished with the session -- leaving an unfalsifiable claim about tests whose whole purpose is to not be vacuous.",
|
|
"scope": [
|
|
"meta/scripts/mutate_session_autopsy_hooks.py (new)",
|
|
".pre-commit-config.yaml (one new local hook)",
|
|
"Non-goal: changing the hooks or the tests. Both are unmodified by this commit."
|
|
],
|
|
"changes": [
|
|
"Promoted the scratchpad harness to a repo tool: 8 named mutations, each breaking ONE guard in a copy of the hooks, run against the suite through AUTOPSY_HOOKS_DIR. Fails if any test survives every mutation.",
|
|
"Per-guard rather than whole-body mutation, because gutting a hook body kills only the 3 positive tests -- a gutted hook emits neither a launch nor output, which is exactly what the 9 negative tests expect. Whole-body gutting alone would falsely brand them vacuous.",
|
|
"Added a staleness guard: if a mutation leaves the hook text unchanged it raises rather than silently reporting 'killed 0'. A behaviour-preserving edit to a hook (renaming a variable, say) would otherwise turn a mutation into a no-op and quietly weaken the proof.",
|
|
"Exits 0 when the hooks are not deployed, so a CI runner with no ~/.claude skips rather than fails.",
|
|
"Wired as a pre-commit hook scoped by files: regex to the suite and the mutation definitions -- the only two files that can invalidate the proof. It takes ~40s, so it must not run on unrelated commits.",
|
|
"The tool is a gate, not a report: it returns non-zero on survivors, per the 'self-checks must gate, not warn' rule."
|
|
],
|
|
"verification": [
|
|
{
|
|
"command": "python3 meta/scripts/mutate_session_autopsy_hooks.py",
|
|
"result": "pass",
|
|
"evidence": "8 mutations; 'PASS: all 12 tests killed by at least one mutation'. Per-mutation kills: gut-end 2, gut-start 1, end-no-transcript-guard 4, start-count-ge-zero 1, start-no-numeric-guard 1, start-no-readable-guard 0, start-always-announce 3, start-unguarded-announce 5."
|
|
},
|
|
{
|
|
"command": "AUTOPSY_HOOKS_DIR=/nonexistent/hooks python3 meta/scripts/mutate_session_autopsy_hooks.py",
|
|
"result": "pass",
|
|
"evidence": "'session_autopsy hooks are not deployed under /nonexistent/hooks -- skipping', exit 0. This is the CI path."
|
|
},
|
|
{
|
|
"command": "staleness guard: copy the hooks, sed 's/\\bcount\\b/n/g' the start hook (behaviour-identical rename), run against the copy",
|
|
"result": "pass",
|
|
"evidence": "The drifted hook still printed the correct banner, confirming the rename preserved behaviour; the tool then failed with \"mutation 'start-count-ge-zero' no longer applies to session_autopsy_start.sh -- the hook text changed, so this mutation tests nothing.\" Text drift is detected without a behaviour break."
|
|
},
|
|
{
|
|
"command": "pre-commit run --files meta/scripts/mutate_session_autopsy_hooks.py .pre-commit-config.yaml linux_configuration/tests/test_session_autopsy_hooks.py",
|
|
"result": "pass",
|
|
"evidence": "All hooks pass, including the new 'session_autopsy hook tests are non-vacuous (mutation)'. Two real lint findings were fixed rather than suppressed: an import inside a function (ruff PLC0415 / pylint C0415) moved to module scope, and unsorted imports. No noqa or type: ignore added."
|
|
}
|
|
],
|
|
"risks": [
|
|
"The gate adds ~40s to any commit touching either of the two scoped files. Deliberate: it is the only thing that can catch the tests silently decaying into exit-status assertions.",
|
|
"Mutations are string transforms over the hook text, so they are coupled to that text by construction. The staleness guard converts that coupling from a silent weakening into a loud failure, but it does mean editing a hook can require updating a mutation.",
|
|
"start-no-readable-guard kills nothing on its own; this is a recorded finding, not a gap. The start hook's readability guard is redundant -- jq's stderr is suppressed and an empty count already fails the arithmetic -- so only the compound start-unguarded-announce mutation reaches those cases."
|
|
],
|
|
"rollback": [
|
|
"Remove the session-autopsy-hook-mutations hook from .pre-commit-config.yaml and delete meta/scripts/mutate_session_autopsy_hooks.py. No production code or test behaviour is affected; the 12 tests keep running under the ordinary pytest gate.",
|
|
"After rollback, confirm 'pre-commit run --files linux_configuration/tests/test_session_autopsy_hooks.py' still passes and the suite still reports 12 passed."
|
|
]
|
|
}
|