CI: skip the kcov-jail suites instead of failing to install kcov

The previous commit added kcov to the shell-tests apt step on the assumption
it was packaged for Ubuntu. It is not -- CI went red in 34s on "E: Unable to
locate package kcov" -- and upstream ships no prebuilt binary either, so it
would have to be built from source on every run.

CI now skips jail_args-marked suites with a printed reason, the same trade
the arch-tests job already makes for Arch-only tests: declare what this
runner cannot do rather than pretend to check it. Enforcement is not lost --
ci_mirror.sh runs those suites through the real jail on every push, and that
gate is already proven to fail on a broken assertion. An empty jail_args is
still a hard failure.

Simulated over all six discovered runners: 5 PASS, 1 SKIP, none dropped.
Committed with CI_GREEN_SKIP=1, the ci-baseline-green hook's documented
escape hatch for a commit that IS the fix for the red baseline.

Also records the kcov instrument defect in docs/shell-split-verification.md.
rpi_nc_install.sh measures 11.36% while its 29 assertions pass and its writes
land on disk. Five hypotheses are now disproven against minimal
reproductions -- the three from the previous session, plus `cd`-relocation
and a heredoc piped into a stubbed external. The cause is still unknown, so
the rule is documented instead: a suspicious percentage means re-measure, not
re-write the test. The failure only ever under-reports, so it can keep a lib
on the allowlist but can never promote an untested one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016sxxsDHWgdRhmdmzNMA6XM
This commit is contained in:
Krzysztof Rudnicki 2026-08-22 12:27:33 +02:00
parent 170f843481
commit 6ebb3462ab
3 changed files with 95 additions and 18 deletions

View File

@ -12,11 +12,8 @@ jobs:
steps:
- uses: actions/checkout@v4
# kcov + unshare back shell_coverage_jail.sh, which the jail_args-marked
# lib/tests suites below run under. util-linux (unshare) is preinstalled;
# kcov is not.
- name: Install bats and kcov
run: sudo apt-get update && sudo apt-get install -y bats kcov
- name: Install bats
run: sudo apt-get update && sudo apt-get install -y bats
- name: Run bats suites
run: bats linux_configuration/tests/*.bats
@ -73,26 +70,31 @@ jobs:
for r in "${runners[@]}"; do
printf '\n=== %s ===\n' "$r"
# A suite that writes to real system paths ships a `jail_args`
# file and REFUSES to run bare; run it through the namespace jail
# with the flags it declares. Empty marker = configuration error.
# file and REFUSES to run bare, because running it bare would
# rewrite the runner's own /etc. It needs shell_coverage_jail.sh,
# which needs kcov -- and kcov is NOT packaged for Ubuntu 24.04
# ("E: Unable to locate package kcov", measured) and upstream
# ships no prebuilt binary, so it would have to be built from
# source on every run.
#
# These suites are therefore SKIPPED here and run for real by the
# ci-mirror pre-push hook on the developer machine, which has
# kcov. That is the same trade the arch-tests job below makes:
# declare what this runner cannot do rather than pretend to check
# it. A skip is announced, never silent.
#
# Empty marker = configuration error, still a hard failure.
args_file="${r%/run_all.sh}/jail_args"
if [ -f "$args_file" ]; then
mapfile -t jail_args < <(grep -v '^#\|^$' "$args_file" || true)
if [ "${#jail_args[@]}" -eq 0 ]; then
if ! grep -qv '^#\|^$' "$args_file"; then
printf 'FAIL %s (jail_args present but empty)\n' "$r"
failed=$((failed + 1))
continue
fi
# --min 0: this step gates on the ASSERTIONS, not on coverage.
# The jail's default --min 100 would otherwise measure the
# runner itself and fail on a number this step never asked for;
# the coverage bar belongs to check_shell_coverage.sh.
cmd=(bash meta/scripts/shell_coverage_jail.sh \
--subject "$r" "${jail_args[@]}" --min 0 --fail-on-case-error -- "")
else
cmd=("$r")
printf 'SKIP %s (needs the kcov jail; enforced by pre-push)\n' "$r"
continue
fi
if "${cmd[@]}"; then
if "$r"; then
printf 'PASS %s\n' "$r"
else
printf 'FAIL %s\n' "$r"

View File

@ -164,3 +164,38 @@ and `main`, which between them assign and read the same configuration globals.
Getting the entry under the cap needs `parse_args` restructured rather than
relocated — a refactor, not a verbatim move, so it is out of scope for this
method. Attempt it with a test harness, or accept an entry that stays over.
### kcov silently under-reports some subjects: never trust a percentage alone
`rpi_nc_install.sh` measures **10/88 = 11.36%** under
`meta/scripts/shell_coverage_jail.sh`, reporting lines 13–60 as covered and
everything from 61 on as not. The code past line 61 provably runs: the suite's
29 assertions pass, and both `/root/.nextcloud_db_password` (line 61) and
`/etc/apache2/sites-available/nextcloud.conf` (a heredoc ending at line 100)
exist on disk after a run. The two sibling libs in the same directory measure
sensibly, so this is not universal — and nobody has bounded which subjects it
affects.
**Five hypotheses have each been tested against a minimal reproduction and
DISPROVEN. Do not retry them:**
1. kcov traces correctly **past a heredoc**.
2. kcov traces correctly **into a `$(...)` command substitution**.
3. kcov traces correctly **past a heredoc-fed stub reading stdin via `$(cat)`**.
4. kcov traces correctly **past a `cd` that relocates the process** (including
`cd /tmp`, where the jail's own working dir lives) — lines after the `cd`
are recorded.
5. kcov traces correctly **past a heredoc piped into a stubbed external** —
`mysql -u root <<EOF` with a `cat >/dev/null` stub, under strict mode.
Lines after it are recorded.
The cause remains unknown. The practical rule:
> **A suspicious percentage means re-measure, not re-write the test.** If a
> suite's assertions pass while its number looks absurd, suspect the
> instrument before suspecting the tests. **Never "fix" a number by weakening
> an assertion.**
Note the failure is _silent and one-directional_: it under-reports, so it can
only ever keep a lib on the allowlist that deserves to come off. It cannot
promote an untested lib. That is why the campaign can continue around it.

View File

@ -0,0 +1,40 @@
{
"intent": "Get the Shell tests workflow green again. The previous commit added `kcov` to its apt install step on the assumption it was packaged for Ubuntu; it is not, and CI went red in 34s on 'E: Unable to locate package kcov'. CI now skips jail-requiring suites with an announced reason, while the pre-push hook keeps running them for real on the developer machine, which has kcov.",
"scope": [
".github/workflows/shell-tests.yml: drop kcov from the install step; SKIP jail_args-marked suites with a printed reason",
"Non-goal: building kcov from source in CI -- minutes added to every run for a check the pre-push hook already performs on every push",
"Non-goal: any change to ci_mirror.sh, which runs these suites for real and is unaffected",
"Non-goal: weakening the empty-marker guard, which stays a hard failure"
],
"changes": [
"The assumption that kcov was apt-installable was inferred, not verified, and CI disproved it in 34 seconds. Upstream (SimonKagstrom/kcov v43) publishes no prebuilt binary asset either, so apt is not merely a packaging gap.",
"CI now declares what it cannot do rather than pretending to check it -- the same trade the arch-tests job already makes for Arch-only tests. The skip prints 'SKIP <runner> (needs the kcov jail; enforced by pre-push)' so it is announced, never silent.",
"The enforcement is not lost: ci_mirror.sh runs the marked suites through the real jail on every push, and that gate is already proven to fail on a broken assertion.",
"The empty-jail_args guard remains a hard failure, so a misconfigured marker still turns CI red instead of being skipped."
],
"verification": [
{
"command": "local simulation of the workflow step's loop over all six discovered runners",
"result": "pass",
"evidence": "5 PASS (scripts/lib, digital_wellbeing, hosts, periodic_background, utils), 1 SKIP (features/lib), failed=0. Every discovered runner is accounted for -- none silently dropped."
},
{
"command": "replace jail_args with comments+blank lines only, re-run the marker check, restore",
"result": "pass",
"evidence": "'FAIL (empty marker) -> correctly detected'. The fail-closed guard survives the rewrite; git diff --stat empty after restore."
},
{
"command": "gh run view 32567260903 --log-failed",
"result": "fail",
"evidence": "'E: Unable to locate package kcov' / 'Process completed with exit code 100' -- the failure this commit fixes, recorded as the measured reason for the redesign."
}
],
"risks": [
"The features/lib suite is now verified only by the pre-push hook, not by CI. A push made with --no-verify would land it unchecked. Acceptable because --no-verify is already forbidden by memories/git-rules.md, and the alternative (building kcov from source) taxes every CI run.",
"If a future contributor works on a machine without kcov, ci_mirror.sh's jailed branch cannot run there either; the jail already errors with 'Error: kcov is not installed' rather than passing silently."
],
"rollback": [
"git revert the commit; CI returns to attempting the kcov install and goes red again on the same apt error.",
"After rollback, confirm the shell-tests workflow fails at the install step, which is the symptom this commit removes."
]
}