The session-autopsy hook tests claimed to be non-vacuous; this commit turns that claim into a pre-commit gate that re-proves it by mutation, every time the proof could go stale.
+
+
+
+
Background
+
The session-autopsy system analyzes Claude Code transcripts after each session. Two small bash hooks wire it into the harness: session_autopsy_end.sh feeds the ended session's transcript to the analyzer, and session_autopsy_start.sh prints a one-line nudge when unreviewed automation candidates exist.
+
Both hooks share an unusual contract: they always exit 0, deliberately, so a broken analyzer can never block a session from starting or ending. That contract has a testing consequence — any test that asserts the hook's exit status passes even if the hook's entire body is deleted. The twelve tests in test_session_autopsy_hooks.py therefore assert side effects: that the end hook launches the analyzer with the right transcript path (captured via a shimmed setsid on PATH), and that the start hook prints exactly the expected banner, or exactly nothing.
+
"Asserts a side effect" is itself just a claim. The previous commit proved it once with a throwaway mutation harness in a session scratchpad — and that proof would have evaporated with the session, leaving an unfalsifiable assertion about tests whose whole purpose is to not be vacuous.
+
+
Intuition
+
Mutation testing asks: if I deliberately break the code, do the tests notice? A test that survives every plausible break is asserting nothing. The tool copies both hooks, breaks one guard per copy, points the suite at the broken copy through AUTOPSY_HOOKS_DIR, and records which tests fail:
+
+
copy hooksdeployed files untouched
+
+
apply 1 of 8 mutationse.g. drop the transcript guard
+
+
run the 12 testsAUTOPSY_HOOKS_DIR=copy
+
+
record killssurvivor of all 8 ⇒ FAIL
+
+
The subtlety that shaped the design: gutting a whole hook is the wrong mutation. A gutted hook emits no launch and no output — which is exactly what the nine negative tests ("given junk input, stay silent") already expect. Gut-only mutation brands those nine tests vacuous when they are not:
+
+
Gut the whole body
Kills 3 positive tests. The 9 negative tests still pass — silence is what they wanted. Misleading
+
Break one guard at a time
An always-announcing start hook, a dropped numeric check… each negative test dies to the mutation that removes the guard it exercises. All 12 killed
+
+
One more failure mode is closed: mutations are string edits over the hook text, so a harmless rename inside a hook would make a mutation silently stop matching — testing nothing. The staleness guard makes that a loud error instead.
+
+
Code
+
The tool — meta/scripts/mutate_session_autopsy_hooks.py (new, 253 lines). Eight named mutations, each a targeted string transform breaking one guard; a runner that copies the hooks, applies one mutation, and executes the suite against the copy; and a verdict that fails on any never-killed test:
+
+
Skip with exit 0 when ~/.claude/hooks has no deployed hooks — a CI runner has nothing to mutate, and a skip must not fail the build.
+
Raise if a mutation leaves the hook text unchanged (staleness guard): a no-op mutation would quietly weaken the proof rather than report a survivor.
+
Gate, don't warn: survivors mean a non-zero exit, per the house rule that self-checks must fail, not print advice.
+
+
The wiring — one new local pre-commit hook, scoped by a files: regex to the only two files that can invalidate the proof: the test suite and the mutation definitions. The run costs ~40 seconds, so it must not fire on unrelated commits:
Recorded honestly rather than papered over: the start-no-readable-guard mutation kills nothing on its own, because that guard is provably redundant (jq's stderr is muted and an empty count already fails the arithmetic) — only the compound always-announce mutation reaches those cases.
+
+
Quiz
+
Click an answer to check it — rationale appears after you pick.