mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 20:07:04 +01:00
Bound every coverage-jail case with a timeout and closed stdin
Investigated whether this session caused the 11:33 hard freeze. It did not, and the evidence is recorded in the evidence artifact: - No OOM kill, no oom_reaper, no hung-task or soft-lockup message, and no kernel message AT ALL after 11:20. The log stops mid-line at 11:33:15 -- a hard hang, not a process the kernel was fighting. - Last commit 11:30:32; last jail trace 11:11:57, 21 minutes earlier. The session was idle when the machine died. - Steam wrote a crash minidump at 11:31:10 and at-spi2 disabled an unresponsive app at 11:32:06, on an RTX 3090. Correlation only -- a hard hang leaves no log to prove cause. - Nothing survived the reboot: no stray unshare/kcov, no leftover tmpdirs, no leaked mounts, and the live /etc and /usr/local/bin files keep their original dates. Hardened anyway, because the jail had one genuinely unbounded behaviour: a case could run forever. Two hangs yesterday came from a `crontab -` fallback inheriting the caller's stdin. Each case now runs under `timeout --kill-after=10s` (default 120s, --timeout to override) with stdin from /dev/null, and a timed-out case WARNS by name instead of being swallowed -- a silent timeout looks identical to an unreachable line in the report. Verified: a `sleep 600` subject now ends in 5s with --timeout 5s. dot_resolver_install.sh still measures 39/39 = 100.00%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DbD4nco3EgL1UFoZKmKsLE
This commit is contained in:
parent
61ff5eacac
commit
b7523d1691
53
docs/superpowers/evidence/jail-case-timeout-2026-08-22.json
Normal file
53
docs/superpowers/evidence/jail-case-timeout-2026-08-22.json
Normal file
@ -0,0 +1,53 @@
|
||||
{
|
||||
"intent": "Investigate whether this session's work caused the 2026-08-22 11:33 hard freeze, and bound the one unbounded thing the coverage jail could do regardless of the verdict.",
|
||||
"scope": [
|
||||
"meta/scripts/shell_coverage_jail.sh: per-case timeout and stdin closed",
|
||||
"Non-goal: changing anything else about the jail; the freeze investigation cleared it"
|
||||
],
|
||||
"changes": [
|
||||
"Each kcov case now runs under `timeout --kill-after=10s` with a default 120s ceiling, overridable with --timeout.",
|
||||
"Each case runs with stdin redirected from /dev/null, so a subject that reads stdin cannot block on the caller's terminal.",
|
||||
"A timed-out case prints a warning naming the case, rather than being swallowed by the existing `|| true`."
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"command": "journalctl --list-boots; journalctl -b -1 for OOM, hung tasks, lockups and late kernel messages",
|
||||
"result": "pass - this session's work is not implicated",
|
||||
"evidence": "Boot -1 ended 11:33:15 with the log cut mid-line. No OOM kill, no oom_reaper, no hung-task or soft-lockup message anywhere, and NO kernel message at all after 11:20 -- a hard hang signature rather than a process the kernel was fighting."
|
||||
},
|
||||
{
|
||||
"command": "correlate this session's activity with the final minutes of boot -1",
|
||||
"result": "pass - the session was idle when the freeze happened",
|
||||
"evidence": "Last commit 11:30:32; last system-level trace from a jail run at 11:11:57 (a failed sudo from an early attempt), over 21 minutes before the freeze. Meanwhile Steam wrote a crash minidump at 11:31:10 and at-spi2 disabled an unresponsive app (pid 4848) at 11:32:06, on an RTX 3090 with the proprietary driver."
|
||||
},
|
||||
{
|
||||
"command": "ps for unshare/kcov; ls /tmp/tmp.*; mount | grep /tmp/tmp.; stat the live system files",
|
||||
"result": "pass - no residue survived the reboot",
|
||||
"evidence": "No stray processes, no leftover jail tmpdirs, zero leaked mounts. /etc/modules-load.d/night-lockdown-i2c.conf still dated 2026-07-23 and /usr/local/bin/sync-dot-cert.sh still dated 2026-08-11, both untouched."
|
||||
},
|
||||
{
|
||||
"command": "bash meta/scripts/shell_coverage_jail.sh ... --measure dot_resolver_install.sh --min 100",
|
||||
"result": "pass - no regression from the timeout change",
|
||||
"evidence": "dot_resolver_install.sh: 39/39 lines = 100.00%, both before and after."
|
||||
},
|
||||
{
|
||||
"command": "run a subject whose body is `sleep 600` with --timeout 5s",
|
||||
"result": "pass - the bound fires",
|
||||
"evidence": "'warn: case timed out after 5s' and the whole run finished in 5 seconds instead of 600."
|
||||
},
|
||||
{
|
||||
"command": "shellcheck meta/scripts/shell_coverage_jail.sh; check_file_length.sh",
|
||||
"result": "pass",
|
||||
"evidence": "Clean, no suppressions; the file is exactly at the 250-line cap."
|
||||
}
|
||||
],
|
||||
"risks": [
|
||||
"The freeze cause is NOT positively identified. A hard hang leaves no log by construction, so the Steam minidump two minutes prior is a correlation, not a proven cause. What is established is the negative: nothing from this session was running at the time, and no OOM occurred.",
|
||||
"The jail still runs real installers as uid 0 inside a namespace. That is the most invasive thing this tooling does, and while containment was verified by canary, escape assertion and mtime checks, it remains the part with any plausible blast radius.",
|
||||
"The 120s default is a guess, not a measured figure. A legitimately slow suite could now be cut off; the warning names the case so that failure is visible rather than silent."
|
||||
],
|
||||
"rollback": [
|
||||
"git revert this commit; the jail returns to unbounded per-case execution.",
|
||||
"After rollback, confirm dot_resolver_install.sh still measures 39/39 through the runner."
|
||||
]
|
||||
}
|
||||
@ -43,6 +43,9 @@ MEASURE=""
|
||||
SEED_DIRS=()
|
||||
SEED_FILES=()
|
||||
MIN_PERCENT=100
|
||||
# Per-case wall-clock ceiling. A hung case cannot be distinguished from a slow
|
||||
# one without a bound, and an unbounded run blocks a hook indefinitely.
|
||||
CASE_TIMEOUT="120s"
|
||||
BIND_PATHS=()
|
||||
EXTRA_SHIMS=()
|
||||
CASES=()
|
||||
@ -64,6 +67,7 @@ Usage: $SCRIPT_NAME --subject <file> [options] -- <case> [<case>...]
|
||||
script writing /etc/php/8.2/apache2/ finds nothing to
|
||||
mirror on an Arch box.
|
||||
--seed-file <path> create an empty file at <path> inside the jail
|
||||
--timeout <dur> per-case wall-clock ceiling (default 120s)
|
||||
--min <percent> fail below this (default 100)
|
||||
-- <case>... one invocation each; "" means "no arguments"
|
||||
|
||||
@ -101,6 +105,10 @@ while [[ $# -gt 0 ]]; do
|
||||
SEED_FILES+=("$2")
|
||||
shift 2
|
||||
;;
|
||||
--timeout)
|
||||
CASE_TIMEOUT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--min)
|
||||
MIN_PERCENT="$2"
|
||||
shift 2
|
||||
@ -183,6 +191,7 @@ jail="$1"
|
||||
subject_dir="$2"
|
||||
subject_base="$3"
|
||||
measure_base="$4"
|
||||
case_timeout="$5"
|
||||
. "$jail/mounts.sh"
|
||||
export PATH="$jail/bin:/usr/bin:/bin"
|
||||
export HOME="$jail/home"
|
||||
@ -192,9 +201,20 @@ export SUDO_USER="$USER"
|
||||
cd "$subject_dir" || exit 1
|
||||
while IFS= read -r line; do
|
||||
read -r -a case_args <<<"$line"
|
||||
kcov --include-pattern="$measure_base" "$jail/cov" \
|
||||
# Bounded, and with stdin closed. Both matter: a subject that reads stdin
|
||||
# blocks forever on the caller's terminal (a `crontab -` fallback did
|
||||
# exactly this and hung a run for the full two minutes with no output),
|
||||
# and a subject that spins never yields at all. A timed-out case is
|
||||
# reported rather than swallowed, because a silent timeout looks
|
||||
# identical to an unreachable line in the coverage report.
|
||||
timeout --kill-after=10s "$case_timeout" \
|
||||
kcov --include-pattern="$measure_base" "$jail/cov" \
|
||||
"./$subject_base" ${case_args[@]+"${case_args[@]}"} \
|
||||
>/dev/null 2>&1 || true
|
||||
>/dev/null 2>&1 </dev/null
|
||||
rc=$?
|
||||
if [[ $rc -eq 124 || $rc -eq 137 ]]; then
|
||||
printf 'warn: case timed out after %s: %s\n' "$case_timeout" "$line" >&2
|
||||
fi
|
||||
done <"$jail/cases"
|
||||
INNER
|
||||
chmod +x "$JAIL/run_cases.sh"
|
||||
@ -212,7 +232,8 @@ INNER
|
||||
} >>"$JAIL/mounts.sh"
|
||||
|
||||
unshare --user --map-root-user --mount --fork \
|
||||
"$JAIL/run_cases.sh" "$JAIL" "$subject_dir" "$subject_base" "$measure_base"
|
||||
"$JAIL/run_cases.sh" "$JAIL" "$subject_dir" "$subject_base" "$measure_base" \
|
||||
"$CASE_TIMEOUT"
|
||||
|
||||
local tree_after
|
||||
tree_after="$(cd "$REPO_ROOT" && git status --porcelain)"
|
||||
|
||||
Loading…
Reference in New Issue
Block a user