shell-tests: actually run the lib/tests suites in CI

Four */lib/tests/run_all.sh runners existed — periodic_background, hosts,
single_use/utils and the new digital_wellbeing one — and no workflow ran any
of them. Every assertion the cap-split sessions wrote was executing only on
the author's machine.

Discovered rather than listed: each split adds another runner, and a
hardcoded list is what let the first four go unnoticed. The step fails if
discovery matches zero runners, so a future path change cannot quietly turn
this into a vacuous green step.

Verified by running the step's exact logic under bash: 4 runners discovered,
all exit 0. Checked for Arch-only dependencies first — the only `command -v`
probe in the harnesses is `unshare` (present on ubuntu-latest, and guarded),
and no suite touches pacman, chattr, systemctl or adb, so these belong in the
Ubuntu job rather than the Arch container.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F3PzGc75gBcdxP3PkHXGcV
This commit is contained in:
Krzysztof Rudnicki 2026-08-21 15:42:51 +02:00
parent 63454fa544
commit fbec46567d
2 changed files with 65 additions and 0 deletions

View File

@ -48,6 +48,39 @@ jobs:
exit 1
fi
# The lib/tests/ suites that came out of the 250-line-cap splits. These
# are DISCOVERED rather than listed: every split adds another
# lib/tests/run_all.sh, and a hardcoded list silently leaves each new
# suite running only on the author's machine. Four such runners already
# existed and none of them ran in CI until this step was added.
#
# Each runner is invoked directly, not as `bash <runner>`: the same
# invocation the coverage wrapper needs, and it keeps the shebang
# authoritative.
- name: Run lib/tests suites
run: |
set -euo pipefail
mapfile -t runners < <(find . -path ./node_modules -prune -o \
-path '*/lib/tests/run_all.sh' -print | sort)
if [ "${#runners[@]}" -eq 0 ]; then
printf 'No lib/tests/run_all.sh found — the discovery glob is wrong\n'
exit 1
fi
failed=0
for r in "${runners[@]}"; do
printf '\n=== %s ===\n' "$r"
if "$r"; then
printf 'PASS %s\n' "$r"
else
printf 'FAIL %s\n' "$r"
failed=$((failed + 1))
fi
done
if [ "$failed" -gt 0 ]; then
printf '\n%d lib/tests suite(s) failed\n' "$failed"
exit 1
fi
# Tests that assume Arch: they exercise the pacman wrapper, makepkg wrapper
# and hosts-guard hooks, so they need an Arch userland rather than Ubuntu.
# They run as root here, which is what lets them touch /etc safely — the

View File

@ -0,0 +1,32 @@
{
"intent": "Make the lib/tests/ suites produced by the 250-line-cap splits actually run in CI. Four such runners existed and none of them were executed by any workflow, so every assertion in them only ever ran on the author's machine.",
"scope": [
".github/workflows/shell-tests.yml: new 'Run lib/tests suites' step in the shell-tests job",
"Non-goal: changing any test, or adding the Arch-only suites to the Ubuntu job"
],
"changes": [
"Added a discovery-based step that finds every */lib/tests/run_all.sh and runs it, failing the job on any non-zero runner. Discovery rather than a hardcoded list: each cap split adds another runner, and a list silently leaves new suites unrun — which is exactly the state this fixes.",
"The step fails loudly if discovery matches zero runners, so a future path change cannot degrade into a vacuous green step.",
"Each runner is invoked directly rather than as `bash <runner>`, matching how the coverage wrapper must invoke it."
],
"verification": [
{
"command": "bash -c 'mapfile -t runners < <(find . -path ./node_modules -prune -o -path \"*/lib/tests/run_all.sh\" -print | sort); ...' (the step's logic, run verbatim under bash)",
"result": "pass",
"evidence": "Discovered 4 runners (digital_wellbeing, hosts, periodic_background, single_use/utils) and all 4 exited 0. failed=0."
},
{
"command": "grep for Arch-only externals in the suites",
"result": "pass",
"evidence": "The only `command -v` probe across the harnesses is `unshare`, which exists on ubuntu-latest and is guarded by the probe. No suite invokes pacman, chattr, systemctl or adb at top level, so none of them need the Arch container job."
}
],
"risks": [
"The suites have only ever been executed on Arch. If one proves environment-sensitive on ubuntu-latest the shell-tests job goes red; the failure would be a genuine portability defect in that suite rather than a problem with this step.",
"Discovery runs whatever it finds, so a future runner that is slow or has side effects would be picked up automatically. The lib/tests convention is side-effect-free by design (that is why these suites exist), and the zero-match guard is the tripwire if the convention moves."
],
"rollback": [
"git revert the commit; the step disappears and the lib/tests suites go back to running only locally.",
"After rollback, confirm the shell-tests workflow still parses and the bats + side-effect-free steps still run."
]
}