From 6ebb3462ab63907c5514c9a4936c064498a9140b Mon Sep 17 00:00:00 2001 From: Krzysztof kuhy Rudnicki Date: Sat, 22 Aug 2026 12:27:33 +0200 Subject: [PATCH] CI: skip the kcov-jail suites instead of failing to install kcov The previous commit added kcov to the shell-tests apt step on the assumption it was packaged for Ubuntu. It is not -- CI went red in 34s on "E: Unable to locate package kcov" -- and upstream ships no prebuilt binary either, so it would have to be built from source on every run. CI now skips jail_args-marked suites with a printed reason, the same trade the arch-tests job already makes for Arch-only tests: declare what this runner cannot do rather than pretend to check it. Enforcement is not lost -- ci_mirror.sh runs those suites through the real jail on every push, and that gate is already proven to fail on a broken assertion. An empty jail_args is still a hard failure. Simulated over all six discovered runners: 5 PASS, 1 SKIP, none dropped. Committed with CI_GREEN_SKIP=1, the ci-baseline-green hook's documented escape hatch for a commit that IS the fix for the red baseline. Also records the kcov instrument defect in docs/shell-split-verification.md. rpi_nc_install.sh measures 11.36% while its 29 assertions pass and its writes land on disk. Five hypotheses are now disproven against minimal reproductions -- the three from the previous session, plus `cd`-relocation and a heredoc piped into a stubbed external. The cause is still unknown, so the rule is documented instead: a suspicious percentage means re-measure, not re-write the test. The failure only ever under-reports, so it can keep a lib on the allowlist but can never promote an untested one. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_016sxxsDHWgdRhmdmzNMA6XM --- .github/workflows/shell-tests.yml | 38 +++++++++--------- docs/shell-split-verification.md | 35 ++++++++++++++++ .../ci-skips-kcov-jail-suites-2026-08-22.json | 40 +++++++++++++++++++ 3 files changed, 95 insertions(+), 18 deletions(-) create mode 100644 docs/superpowers/evidence/ci-skips-kcov-jail-suites-2026-08-22.json diff --git a/.github/workflows/shell-tests.yml b/.github/workflows/shell-tests.yml index 146d7ad9..8bbbabe9 100644 --- a/.github/workflows/shell-tests.yml +++ b/.github/workflows/shell-tests.yml @@ -12,11 +12,8 @@ jobs: steps: - uses: actions/checkout@v4 - # kcov + unshare back shell_coverage_jail.sh, which the jail_args-marked - # lib/tests suites below run under. util-linux (unshare) is preinstalled; - # kcov is not. - - name: Install bats and kcov - run: sudo apt-get update && sudo apt-get install -y bats kcov + - name: Install bats + run: sudo apt-get update && sudo apt-get install -y bats - name: Run bats suites run: bats linux_configuration/tests/*.bats @@ -73,26 +70,31 @@ jobs: for r in "${runners[@]}"; do printf '\n=== %s ===\n' "$r" # A suite that writes to real system paths ships a `jail_args` - # file and REFUSES to run bare; run it through the namespace jail - # with the flags it declares. Empty marker = configuration error. + # file and REFUSES to run bare, because running it bare would + # rewrite the runner's own /etc. It needs shell_coverage_jail.sh, + # which needs kcov -- and kcov is NOT packaged for Ubuntu 24.04 + # ("E: Unable to locate package kcov", measured) and upstream + # ships no prebuilt binary, so it would have to be built from + # source on every run. + # + # These suites are therefore SKIPPED here and run for real by the + # ci-mirror pre-push hook on the developer machine, which has + # kcov. That is the same trade the arch-tests job below makes: + # declare what this runner cannot do rather than pretend to check + # it. A skip is announced, never silent. + # + # Empty marker = configuration error, still a hard failure. args_file="${r%/run_all.sh}/jail_args" if [ -f "$args_file" ]; then - mapfile -t jail_args < <(grep -v '^#\|^$' "$args_file" || true) - if [ "${#jail_args[@]}" -eq 0 ]; then + if ! grep -qv '^#\|^$' "$args_file"; then printf 'FAIL %s (jail_args present but empty)\n' "$r" failed=$((failed + 1)) continue fi - # --min 0: this step gates on the ASSERTIONS, not on coverage. - # The jail's default --min 100 would otherwise measure the - # runner itself and fail on a number this step never asked for; - # the coverage bar belongs to check_shell_coverage.sh. - cmd=(bash meta/scripts/shell_coverage_jail.sh \ - --subject "$r" "${jail_args[@]}" --min 0 --fail-on-case-error -- "") - else - cmd=("$r") + printf 'SKIP %s (needs the kcov jail; enforced by pre-push)\n' "$r" + continue fi - if "${cmd[@]}"; then + if "$r"; then printf 'PASS %s\n' "$r" else printf 'FAIL %s\n' "$r" diff --git a/docs/shell-split-verification.md b/docs/shell-split-verification.md index 47e90564..80c21c07 100644 --- a/docs/shell-split-verification.md +++ b/docs/shell-split-verification.md @@ -164,3 +164,38 @@ and `main`, which between them assign and read the same configuration globals. Getting the entry under the cap needs `parse_args` restructured rather than relocated — a refactor, not a verbatim move, so it is out of scope for this method. Attempt it with a test harness, or accept an entry that stays over. + +### kcov silently under-reports some subjects: never trust a percentage alone + +`rpi_nc_install.sh` measures **10/88 = 11.36%** under +`meta/scripts/shell_coverage_jail.sh`, reporting lines 13–60 as covered and +everything from 61 on as not. The code past line 61 provably runs: the suite's +29 assertions pass, and both `/root/.nextcloud_db_password` (line 61) and +`/etc/apache2/sites-available/nextcloud.conf` (a heredoc ending at line 100) +exist on disk after a run. The two sibling libs in the same directory measure +sensibly, so this is not universal — and nobody has bounded which subjects it +affects. + +**Five hypotheses have each been tested against a minimal reproduction and +DISPROVEN. Do not retry them:** + +1. kcov traces correctly **past a heredoc**. +2. kcov traces correctly **into a `$(...)` command substitution**. +3. kcov traces correctly **past a heredoc-fed stub reading stdin via `$(cat)`**. +4. kcov traces correctly **past a `cd` that relocates the process** (including + `cd /tmp`, where the jail's own working dir lives) — lines after the `cd` + are recorded. +5. kcov traces correctly **past a heredoc piped into a stubbed external** — + `mysql -u root </dev/null` stub, under strict mode. + Lines after it are recorded. + +The cause remains unknown. The practical rule: + +> **A suspicious percentage means re-measure, not re-write the test.** If a +> suite's assertions pass while its number looks absurd, suspect the +> instrument before suspecting the tests. **Never "fix" a number by weakening +> an assertion.** + +Note the failure is _silent and one-directional_: it under-reports, so it can +only ever keep a lib on the allowlist that deserves to come off. It cannot +promote an untested lib. That is why the campaign can continue around it. diff --git a/docs/superpowers/evidence/ci-skips-kcov-jail-suites-2026-08-22.json b/docs/superpowers/evidence/ci-skips-kcov-jail-suites-2026-08-22.json new file mode 100644 index 00000000..66b6f332 --- /dev/null +++ b/docs/superpowers/evidence/ci-skips-kcov-jail-suites-2026-08-22.json @@ -0,0 +1,40 @@ +{ + "intent": "Get the Shell tests workflow green again. The previous commit added `kcov` to its apt install step on the assumption it was packaged for Ubuntu; it is not, and CI went red in 34s on 'E: Unable to locate package kcov'. CI now skips jail-requiring suites with an announced reason, while the pre-push hook keeps running them for real on the developer machine, which has kcov.", + "scope": [ + ".github/workflows/shell-tests.yml: drop kcov from the install step; SKIP jail_args-marked suites with a printed reason", + "Non-goal: building kcov from source in CI -- minutes added to every run for a check the pre-push hook already performs on every push", + "Non-goal: any change to ci_mirror.sh, which runs these suites for real and is unaffected", + "Non-goal: weakening the empty-marker guard, which stays a hard failure" + ], + "changes": [ + "The assumption that kcov was apt-installable was inferred, not verified, and CI disproved it in 34 seconds. Upstream (SimonKagstrom/kcov v43) publishes no prebuilt binary asset either, so apt is not merely a packaging gap.", + "CI now declares what it cannot do rather than pretending to check it -- the same trade the arch-tests job already makes for Arch-only tests. The skip prints 'SKIP (needs the kcov jail; enforced by pre-push)' so it is announced, never silent.", + "The enforcement is not lost: ci_mirror.sh runs the marked suites through the real jail on every push, and that gate is already proven to fail on a broken assertion.", + "The empty-jail_args guard remains a hard failure, so a misconfigured marker still turns CI red instead of being skipped." + ], + "verification": [ + { + "command": "local simulation of the workflow step's loop over all six discovered runners", + "result": "pass", + "evidence": "5 PASS (scripts/lib, digital_wellbeing, hosts, periodic_background, utils), 1 SKIP (features/lib), failed=0. Every discovered runner is accounted for -- none silently dropped." + }, + { + "command": "replace jail_args with comments+blank lines only, re-run the marker check, restore", + "result": "pass", + "evidence": "'FAIL (empty marker) -> correctly detected'. The fail-closed guard survives the rewrite; git diff --stat empty after restore." + }, + { + "command": "gh run view 32567260903 --log-failed", + "result": "fail", + "evidence": "'E: Unable to locate package kcov' / 'Process completed with exit code 100' -- the failure this commit fixes, recorded as the measured reason for the redesign." + } + ], + "risks": [ + "The features/lib suite is now verified only by the pre-push hook, not by CI. A push made with --no-verify would land it unchecked. Acceptable because --no-verify is already forbidden by memories/git-rules.md, and the alternative (building kcov from source) taxes every CI run.", + "If a future contributor works on a machine without kcov, ci_mirror.sh's jailed branch cannot run there either; the jail already errors with 'Error: kcov is not installed' rather than passing silently." + ], + "rollback": [ + "git revert the commit; CI returns to attempting the kcov install and goes red again on the same apt error.", + "After rollback, confirm the shell-tests workflow fails at the install step, which is the symptom this commit removes." + ] +}