testsAndMisc/docs/superpowers/evidence/ci-mirror-serialise-under-pressure-2026-08-23.json
Krzysztof kuhy Rudnicki 2a1bb60d9f
Some checks are pending
Pre-commit checks / pre-commit (push) Waiting to run
Python tests / test (push) Waiting to run
Shell tests / shell-tests (push) Waiting to run
Shell tests / arch-tests (push) Waiting to run
Serialise the heavy ci-mirror gates when memory is tight
The push kept failing with "1798 passed" immediately followed by "Killed".
The suite was never the problem: it peaks at ~0.29 GiB against its own 2G
cap. ci_mirror.sh runs pre-commit --all-files and the pytest gate
concurrently, and the pre-commit tree peaks at ~1.75 GiB (mypy and pylint
repo-wide, invisible to /usr/bin/time because it measures only the parent).
The pair wants ~2 GiB at once; the pytest scope has MemorySwapMax=0, so on
a loaded machine it is the one that dies.

Concurrency is now conditional on MemAvailable. Above 4 GiB the fast path
is unchanged. The probe lives in ci_mirror_mem.sh, following the existing
ci_mirror_range/ci_mirror_shell split, which also keeps ci_mirror.sh under
the line cap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PJrqKERAo5QUeEHWS5Asgp
2026-08-23 20:02:59 +02:00

45 lines
3.5 KiB
JSON

{
"intent": "Stop the pre-push gate failing with 'tests passed, then Killed' when the machine is under memory pressure, so a push is blocked by the code and not by unrelated load.",
"scope": [
"meta/scripts/ci_mirror.sh - run the two heavy gates serially when memory is tight",
"meta/scripts/ci_mirror_mem.sh - new lib holding the probe, the threshold and the predicate",
"Non-goal: the 2G pytest cap in pytest_changed_packages.py, which measurement showed is correct and was left alone."
],
"changes": [
"Diagnosed the real cause: ci_mirror.sh runs `pre-commit run --all-files` and the pytest gate concurrently with & and wait. Measured tree peaks are ~1.75 GiB for pre-commit (mypy/pylint repo-wide) and ~0.29 GiB for pytest, so the pair wants ~2 GiB at the same instant. The pytest scope runs under MemoryMax=2G with MemorySwapMax=0, so under pressure it is the one the kernel kills - the log reads '1798 passed' then 'Killed', which the gate reports as a test failure.",
"Added available_mem_mib (reads MemAvailable from /proc, no fork) and should_serialise_gates, and made the concurrency conditional on it. Above 4 GiB available the fast parallel path is unchanged.",
"Extracted the probe into ci_mirror_mem.sh following the existing ci_mirror_range.sh / ci_mirror_shell.sh pattern, which also kept ci_mirror.sh under the 250-line cap (245).",
"Exposed the threshold as a predicate rather than a bare constant so SC2034 is resolved by giving the value a real use in its own file, not by a suppression."
],
"verification": [
{
"command": "systemd-run --scope -p MemoryMax=2G -p MemorySwapMax=0 /usr/bin/time -v python -m pytest ... (live tree and clean worktree)",
"result": "pass",
"evidence": "1798 passed, Maximum RSS 293928 KB (~0.29 GiB) both times. Proves the suite is nowhere near its own 2G ceiling, so the ceiling was not the problem and did not need raising."
},
{
"command": "systemd-run --scope -p MemoryMax=8G bash -c 'pre-commit run --all-files; cat .../memory.peak'",
"result": "pass",
"evidence": "1751576576 bytes = ~1.75 GiB across the process tree. /usr/bin/time reported only ~0.29 GiB because it measures the parent, not the forked mypy/pylint children - the cgroup peak is the honest number."
},
{
"command": "Stubbed available_mem_mib at 1000 and 99999 MiB against should_serialise_gates",
"result": "pass",
"evidence": "1000 -> SERIAL, 99999 -> PARALLEL. Boundary table also checked: 4095 SERIAL, 4096 PARALLEL. Confirms the new branch is reachable and not dead code."
},
{
"command": "pre-commit run --files meta/scripts/ci_mirror.sh meta/scripts/ci_mirror_mem.sh",
"result": "pass",
"evidence": "No failures. file-length-cap satisfied at 245 lines; SC2034 resolved without a suppression."
}
],
"risks": [
"Under 4 GiB available the gate is slower, since the stages no longer overlap. That is the intended trade against a false failure.",
"MemAvailable is a snapshot: memory can be consumed after the check by another process, so this reduces the failure window rather than closing it absolutely. The 2G cap with MemorySwapMax=0 still bounds the damage to a clean kill rather than zram thrashing."
],
"rollback": [
"git revert the commit and delete meta/scripts/ci_mirror_mem.sh; the gate returns to unconditional concurrency.",
"After rollback, confirm ci_mirror.sh still sources cleanly (bash -n) and that a push runs the gate."
]
}