hook test mutation gate

2026-07-25 · testsAndMisc

The session-autopsy hook tests claimed to be non-vacuous; this commit turns that claim into a pre-commit gate that re-proves it by mutation, every time the proof could go stale.

Background

The session-autopsy system analyzes Claude Code transcripts after each session. Two small bash hooks wire it into the harness: session_autopsy_end.sh feeds the ended session's transcript to the analyzer, and session_autopsy_start.sh prints a one-line nudge when unreviewed automation candidates exist.

Both hooks share an unusual contract: they always exit 0, deliberately, so a broken analyzer can never block a session from starting or ending. That contract has a testing consequence — any test that asserts the hook's exit status passes even if the hook's entire body is deleted. The twelve tests in test_session_autopsy_hooks.py therefore assert side effects: that the end hook launches the analyzer with the right transcript path (captured via a shimmed setsid on PATH), and that the start hook prints exactly the expected banner, or exactly nothing.

"Asserts a side effect" is itself just a claim. The previous commit proved it once with a throwaway mutation harness in a session scratchpad — and that proof would have evaporated with the session, leaving an unfalsifiable assertion about tests whose whole purpose is to not be vacuous.

Intuition

Mutation testing asks: if I deliberately break the code, do the tests notice? A test that survives every plausible break is asserting nothing. The tool copies both hooks, breaks one guard per copy, points the suite at the broken copy through AUTOPSY_HOOKS_DIR, and records which tests fail:

copy hooksdeployed files untouched
apply 1 of 8 mutationse.g. drop the transcript guard
run the 12 testsAUTOPSY_HOOKS_DIR=copy
record killssurvivor of all 8 ⇒ FAIL

The subtlety that shaped the design: gutting a whole hook is the wrong mutation. A gutted hook emits no launch and no output — which is exactly what the nine negative tests ("given junk input, stay silent") already expect. Gut-only mutation brands those nine tests vacuous when they are not:

Gut the whole body

Kills 3 positive tests. The 9 negative tests still pass — silence is what they wanted. Misleading

Break one guard at a time

An always-announcing start hook, a dropped numeric check… each negative test dies to the mutation that removes the guard it exercises. All 12 killed

One more failure mode is closed: mutations are string edits over the hook text, so a harmless rename inside a hook would make a mutation silently stop matching — testing nothing. The staleness guard makes that a loud error instead.

Code

The tool — meta/scripts/mutate_session_autopsy_hooks.py (new, 253 lines). Eight named mutations, each a targeted string transform breaking one guard; a runner that copies the hooks, applies one mutation, and executes the suite against the copy; and a verdict that fails on any never-killed test:

  1. Skip with exit 0 when ~/.claude/hooks has no deployed hooks — a CI runner has nothing to mutate, and a skip must not fail the build.
  2. Raise if a mutation leaves the hook text unchanged (staleness guard): a no-op mutation would quietly weaken the proof rather than report a survivor.
  3. Gate, don't warn: survivors mean a non-zero exit, per the house rule that self-checks must fail, not print advice.

The wiring — one new local pre-commit hook, scoped by a files: regex to the only two files that can invalidate the proof: the test suite and the mutation definitions. The run costs ~40 seconds, so it must not fire on unrelated commits:

- id: session-autopsy-hook-mutations
  entry: python3 meta/scripts/mutate_session_autopsy_hooks.py
  files: ^(linux_configuration/tests/test_session_autopsy_hooks\.py|meta/scripts/mutate_session_autopsy_hooks\.py)$

Recorded honestly rather than papered over: the start-no-readable-guard mutation kills nothing on its own, because that guard is provably redundant (jq's stderr is muted and an empty count already fails the arithmetic) — only the compound always-announce mutation reaches those cases.

Quiz

Click an answer to check it — rationale appears after you pick.