From fbec46567db71dd63fb8a581f11aa28fe584a07e Mon Sep 17 00:00:00 2001 From: Krzysztof kuhy Rudnicki Date: Fri, 21 Aug 2026 15:42:51 +0200 Subject: [PATCH] shell-tests: actually run the lib/tests suites in CI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four */lib/tests/run_all.sh runners existed — periodic_background, hosts, single_use/utils and the new digital_wellbeing one — and no workflow ran any of them. Every assertion the cap-split sessions wrote was executing only on the author's machine. Discovered rather than listed: each split adds another runner, and a hardcoded list is what let the first four go unnoticed. The step fails if discovery matches zero runners, so a future path change cannot quietly turn this into a vacuous green step. Verified by running the step's exact logic under bash: 4 runners discovered, all exit 0. Checked for Arch-only dependencies first — the only `command -v` probe in the harnesses is `unshare` (present on ubuntu-latest, and guarded), and no suite touches pacman, chattr, systemctl or adb, so these belong in the Ubuntu job rather than the Arch container. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01F3PzGc75gBcdxP3PkHXGcV --- .github/workflows/shell-tests.yml | 33 +++++++++++++++++++ ...l-tests-libtests-discovery-2026-08-21.json | 32 ++++++++++++++++++ 2 files changed, 65 insertions(+) create mode 100644 docs/superpowers/evidence/shell-tests-libtests-discovery-2026-08-21.json diff --git a/.github/workflows/shell-tests.yml b/.github/workflows/shell-tests.yml index b70eb98e..99613f57 100644 --- a/.github/workflows/shell-tests.yml +++ b/.github/workflows/shell-tests.yml @@ -48,6 +48,39 @@ jobs: exit 1 fi + # The lib/tests/ suites that came out of the 250-line-cap splits. These + # are DISCOVERED rather than listed: every split adds another + # lib/tests/run_all.sh, and a hardcoded list silently leaves each new + # suite running only on the author's machine. Four such runners already + # existed and none of them ran in CI until this step was added. + # + # Each runner is invoked directly, not as `bash `: the same + # invocation the coverage wrapper needs, and it keeps the shebang + # authoritative. + - name: Run lib/tests suites + run: | + set -euo pipefail + mapfile -t runners < <(find . -path ./node_modules -prune -o \ + -path '*/lib/tests/run_all.sh' -print | sort) + if [ "${#runners[@]}" -eq 0 ]; then + printf 'No lib/tests/run_all.sh found — the discovery glob is wrong\n' + exit 1 + fi + failed=0 + for r in "${runners[@]}"; do + printf '\n=== %s ===\n' "$r" + if "$r"; then + printf 'PASS %s\n' "$r" + else + printf 'FAIL %s\n' "$r" + failed=$((failed + 1)) + fi + done + if [ "$failed" -gt 0 ]; then + printf '\n%d lib/tests suite(s) failed\n' "$failed" + exit 1 + fi + # Tests that assume Arch: they exercise the pacman wrapper, makepkg wrapper # and hosts-guard hooks, so they need an Arch userland rather than Ubuntu. # They run as root here, which is what lets them touch /etc safely — the diff --git a/docs/superpowers/evidence/shell-tests-libtests-discovery-2026-08-21.json b/docs/superpowers/evidence/shell-tests-libtests-discovery-2026-08-21.json new file mode 100644 index 00000000..fd459e3c --- /dev/null +++ b/docs/superpowers/evidence/shell-tests-libtests-discovery-2026-08-21.json @@ -0,0 +1,32 @@ +{ + "intent": "Make the lib/tests/ suites produced by the 250-line-cap splits actually run in CI. Four such runners existed and none of them were executed by any workflow, so every assertion in them only ever ran on the author's machine.", + "scope": [ + ".github/workflows/shell-tests.yml: new 'Run lib/tests suites' step in the shell-tests job", + "Non-goal: changing any test, or adding the Arch-only suites to the Ubuntu job" + ], + "changes": [ + "Added a discovery-based step that finds every */lib/tests/run_all.sh and runs it, failing the job on any non-zero runner. Discovery rather than a hardcoded list: each cap split adds another runner, and a list silently leaves new suites unrun — which is exactly the state this fixes.", + "The step fails loudly if discovery matches zero runners, so a future path change cannot degrade into a vacuous green step.", + "Each runner is invoked directly rather than as `bash `, matching how the coverage wrapper must invoke it." + ], + "verification": [ + { + "command": "bash -c 'mapfile -t runners < <(find . -path ./node_modules -prune -o -path \"*/lib/tests/run_all.sh\" -print | sort); ...' (the step's logic, run verbatim under bash)", + "result": "pass", + "evidence": "Discovered 4 runners (digital_wellbeing, hosts, periodic_background, single_use/utils) and all 4 exited 0. failed=0." + }, + { + "command": "grep for Arch-only externals in the suites", + "result": "pass", + "evidence": "The only `command -v` probe across the harnesses is `unshare`, which exists on ubuntu-latest and is guarded by the probe. No suite invokes pacman, chattr, systemctl or adb at top level, so none of them need the Arch container job." + } + ], + "risks": [ + "The suites have only ever been executed on Arch. If one proves environment-sensitive on ubuntu-latest the shell-tests job goes red; the failure would be a genuine portability defect in that suite rather than a problem with this step.", + "Discovery runs whatever it finds, so a future runner that is slow or has side effects would be picked up automatically. The lib/tests convention is side-effect-free by design (that is why these suites exist), and the zero-match guard is the tripwire if the convention moves." + ], + "rollback": [ + "git revert the commit; the step disappears and the lib/tests suites go back to running only locally.", + "After rollback, confirm the shell-tests workflow still parses and the bats + side-effect-free steps still run." + ] +}