mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 21:34:05 +01:00
Krzysztof kuhy Rudnicki
35ddf4114f
Four evidence files committed today cite mutation runs -- 18, 13, 17 and
17 killed -- that lived in a scratch directory. Evidence pointing at a run
nobody can repeat is not evidence, so the lists move into the repo behind a
spec-driven runner.
Mutations are literal find/replace strings in JSON, not sed expressions.
The shell version this replaces escaped each pattern twice and used | as
its field separator, which collided with || and reported six mutations as
SURVIVED when they had never been applied at all -- a false negative in the
one tool whose entire job is catching false negatives.
Two deliberate choices follow from that:
- a no-op (find matched nothing) counts as a FAILURE, not a skip. A stale
spec entry otherwise stops testing its behaviour without saying so.
- a timeout counts as killed: a mutation that hangs the suite has still
been detected by it.
The conversion itself exercised the first rule -- it reported 8 no-ops from
mangled escapes rather than quietly scoring them as killed. Each was fixed
against the real source line.
All four specs now reproduce their committed figures exactly, and the
working tree is byte-clean after every run.
Not wired into pre-commit: it runs the whole suite once per mutation, which
is minutes, not seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011w2Pj4Ek3ZNMQVYcFHgXDY
62 lines
4.4 KiB
JSON
62 lines
4.4 KiB
JSON
{
|
|
"intent": "Make the mutation-testing claims in this session's four evidence files reproducible from the repo. They cited runs from a scratch directory that will not survive the session, leaving committed evidence pointing at something nobody could repeat.",
|
|
"scope": [
|
|
"meta/scripts/mutate_shell.py - new: a spec-driven mutation runner",
|
|
"meta/scripts/fixtures/mutations/{monitor,backup_capture,curfew_net,hosts_libs}.json - new: the 65 mutations already used",
|
|
"Non-goal: wiring this into pre-commit. It runs the full shell suite once per mutation, which is far too slow for a commit hook.",
|
|
"Non-goal: changing any subject or test; this only relocates and formalises checks already performed."
|
|
],
|
|
"changes": [
|
|
"Declared mutations in JSON with literal find/replace strings rather than sed expressions. The shell version escaped each pattern twice and used | as a field separator, which collided with || and silently reported six mutations as survivors when they had never been applied — a false negative in the exact tool meant to catch false negatives.",
|
|
"Counted a no-op (find matched nothing) as a failure rather than ignoring it. A stale spec entry otherwise stops testing its behaviour without ever saying so, which is the same silent-pass failure mode the tool exists to detect.",
|
|
"Restored the subject in a finally block, so an interrupted run cannot leave a deliberately broken file in the working tree.",
|
|
"Treated a timeout as killed: a mutation that makes the suite hang has still been detected by it."
|
|
],
|
|
"verification": [
|
|
{
|
|
"command": "python3 meta/scripts/mutate_shell.py meta/scripts/fixtures/mutations/monitor.json",
|
|
"result": "pass",
|
|
"evidence": "18 killed, 0 not killed — matching the figure in monitor-coverage-backfill-2026-08-17.json."
|
|
},
|
|
{
|
|
"command": "python3 meta/scripts/mutate_shell.py meta/scripts/fixtures/mutations/backup_capture.json",
|
|
"result": "pass",
|
|
"evidence": "13 killed, 0 not killed — matching phone-backup-capture-split-2026-08-17.json."
|
|
},
|
|
{
|
|
"command": "python3 meta/scripts/mutate_shell.py meta/scripts/fixtures/mutations/curfew_net.json",
|
|
"result": "pass",
|
|
"evidence": "17 killed, 0 not killed — matching curfew-net-split-2026-08-17.json."
|
|
},
|
|
{
|
|
"command": "python3 meta/scripts/mutate_shell.py meta/scripts/fixtures/mutations/hosts_libs.json",
|
|
"result": "pass",
|
|
"evidence": "17 killed, 0 not killed — matching hosts-enforcer-split-2026-08-17.json."
|
|
},
|
|
{
|
|
"command": "git status --short after every run",
|
|
"result": "pass",
|
|
"evidence": "Clean. The restore path returns each subject byte-for-byte; an earlier shell version left a mode-only diff behind, which is why this one uses copy2 and an unconditional finally."
|
|
},
|
|
{
|
|
"command": "pre-commit run --files meta/scripts/mutate_shell.py meta/scripts/fixtures/mutations/*.json",
|
|
"result": "pass",
|
|
"evidence": "Clean, including ruff, mypy and pylint. Ruff's autofixer removed a suppression during the first run; the tool was re-run afterwards and still reports 13 killed on backup_capture, so the autofix changed nothing behavioural."
|
|
},
|
|
{
|
|
"command": "first conversion run of all four specs",
|
|
"result": "pass",
|
|
"evidence": "Reported 8 no-ops rather than silently passing, caused by the sed-to-literal conversion mangling escapes. Each was corrected against the actual source line and re-run. That the tool surfaced them rather than counting them as killed is the behaviour being relied on."
|
|
}
|
|
],
|
|
"risks": [
|
|
"The specs pin literal source text, so any edit to a mutated line makes its entry a no-op. That is reported loudly rather than passing quietly, but it does mean the specs need updating alongside the code they target.",
|
|
"This is not enforced by any hook. Nothing prevents a future shell library from being added with a coverage number and no mutation spec; the discipline remains a convention.",
|
|
"A full spec run takes minutes, since it executes the whole suite once per mutation. It is a deliberate, occasional check, not something to put in a tight loop."
|
|
],
|
|
"rollback": [
|
|
"git revert the commit. The four evidence files then cite mutation runs with no reproducible path, which is the state this commit exists to fix — prefer fixing forward.",
|
|
"After rollback, nothing else breaks: no hook, test or script references the tool."
|
|
]
|
|
}
|