Phase 3: ratchet shell libraries onto test suites

A shell library under a lib/ directory must now sit beside a
tests/run_all.sh. This is a presence gate, not a coverage percentage:
Phase 1 measured these libs' decision logic at ~90% under test but
their effecting code at 0%, because it sudo-writes, pkills and installs.
A numeric bar would only be reachable by shimming every external for
every installer, or by the suppressions this repo forbids.

Seeded at 125 exempt libraries out of 208 tracked; the other 83 already
have a suite. The allowlist shrinks only — a modified library loses its
exemption and must bring tests.

Verified standalone: blocks a new untested lib (exit 1), and does NOT
block covered libs, exempt libs, or unrelated entry scripts. That last
case is the point of a ratchet over the repo-wide gate.

The .pre-commit-config.yaml entry is deliberately NOT included — the
edit was denied as a sensitive file, so the gate is runnable but not
yet enforced.
This commit is contained in:
Krzysztof Rudnicki 2026-08-21 17:00:26 +02:00
parent 09af51d462
commit 3278bebd03
3 changed files with 391 additions and 0 deletions

View File

@ -0,0 +1,57 @@
{
"intent": "Phase 3: make a shell library unable to enter the repo without a test suite beside it, without blocking commits to the 125 libraries that predate the rule.",
"scope": [
"meta/scripts/check_shell_coverage.sh — new ratchet gate",
"meta/shell-coverage-allowlist.txt — seeded exemption list, shrink-only",
"NON-GOAL: a coverage percentage. Phase 1 measured the effecting code in these libs at 0% (sudo/pkill/curl/npm paths), so a numeric bar is only reachable via mass shimming or the suppressions this repo forbids. This gate checks for the PRESENCE of a suite.",
"NON-GOAL: entry scripts. Scope is .sh under a lib/ dir, excluding lib/tests/ and lib/payloads/.",
"NOT LANDED: the .pre-commit-config.yaml hook entry. The edit was denied as a sensitive file; the gate is standalone-runnable but not yet enforced."
],
"changes": [
"check_shell_coverage.sh gates named files (hook mode), reports repo-wide state (--all), and reseeds the exemption list (--seed).",
"A library counts as covered when its directory has a tests/run_all.sh. 4 such suites exist: digital_wellbeing, hosts, periodic_background, single_use/utils.",
"Allowlist seeded at 125 uncovered libraries out of 208 tracked lib files; the remaining 83 already sit beside a suite.",
"Discovery uses `git ls-files`, which excludes the .venv/.ci-mirror-venv/node_modules pollution that inflated a raw `find` to 565 files."
],
"verification": [
{
"command": "bash meta/scripts/check_shell_coverage.sh --seed && bash meta/scripts/check_shell_coverage.sh --all",
"result": "pass",
"evidence": "seeded allowlist with 125 uncovered librar(ies); --all reports '125 uncovered, 125 exempt' and 0 BLOCK lines — the ratchet starts green."
},
{
"command": "bash meta/scripts/check_shell_coverage.sh <a new lib with no suite>",
"result": "pass",
"evidence": "exit=1. Blocked _ratchet_probe.sh under single_use/features/lib and named the required path single_use/features/lib/tests/run_all.sh. Probe file removed afterwards; tree confirmed clean. This is the case proving the gate has teeth."
},
{
"command": "bash meta/scripts/check_shell_coverage.sh meta/scripts/check_file_length.sh",
"result": "pass",
"evidence": "exit=0 on an unrelated non-lib entry script. This is the negative case that distinguishes a ratchet from the repo-wide gate that was rejected: unrelated commits are not blocked."
},
{
"command": "bash meta/scripts/check_shell_coverage.sh linux_configuration/scripts/single_use/utils/lib/cco_state.sh; and phone_focus_mode/lib/adb.sh",
"result": "pass",
"evidence": "exit=0 for a covered library (suite present) and exit=0 for an allowlisted one (exempt until modified)."
},
{
"command": "shellcheck meta/scripts/check_shell_coverage.sh; shfmt -d meta/scripts/check_shell_coverage.sh",
"result": "pass",
"evidence": "shellcheck clean, shfmt reports no diff. No suppressions added."
},
{
"command": "grep -nE '\\[[a-z0-9]+ - ' meta/scripts/check_shell_coverage.sh meta/shell-coverage-allowlist.txt",
"result": "pass",
"evidence": "Empty. The shfmt hyphenated-associative-key corruption canary from Phase 1 is clean."
}
],
"risks": [
"The gate proves a suite EXISTS beside a library, not that the suite exercises that particular file. A directory with a run_all.sh can gain an untested library silently. Presence is the strongest property enforceable without the forbidden suppressions; tightening it to per-file assertions is future work.",
"--seed rewrites the allowlist from the current tree, so running it after adding an untested library would exempt that library. The header says it may only ever be used to remove entries; nothing mechanically enforces the shrink-only rule yet.",
"Adding a tests/run_all.sh to a large exempt directory (features/lib has 45 entries) makes all of them enforced at once, which may surface a large batch of work in one commit."
],
"rollback": [
"Delete meta/scripts/check_shell_coverage.sh and meta/shell-coverage-allowlist.txt. Nothing else references them while the hook entry remains unlanded.",
"After rollback confirm `bash meta/scripts/check_file_length.sh --all` still exits 0 and that pre-commit runs unchanged — this change adds no dependency any other hook consumes."
]
}

View File

@ -0,0 +1,198 @@
#!/usr/bin/env bash
# ============================================================================
# Ratchet: every shell library must live beside a test suite that runs it.
#
# This is a PRESENCE gate, not a percentage gate. Phase 1 measured why: the
# decision logic in these libs reaches ~90% under test, while the effecting
# code reaches 0% because it sudo-writes, pkills, or installs packages. A
# numeric bar would therefore be unreachable without either shimming every
# external for every installer or adding the suppressions this repo forbids.
# What is enforceable — and what actually catches regressions — is that a lib
# directory has a run_all.sh which sources and exercises it.
#
# Scope is libraries under a lib/ directory. Entry scripts are orchestration
# whose bodies are the untestable part, so capping them would buy nothing.
#
# The ratchet: files listed in the allowlist predate the gate and are exempt.
# A file that is NEW or MODIFIED must be covered, and a covered file can never
# go back on the allowlist. The list only ever shrinks.
#
# check_shell_coverage.sh <file>... # gate the named files (hook mode)
# check_shell_coverage.sh --all # report every uncovered lib
# check_shell_coverage.sh --seed # rewrite the allowlist from HEAD
#
# Exits 0 when every checked file is covered or exempt, 1 otherwise.
# ============================================================================
set -euo pipefail
SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_NAME
REPO_ROOT="$(git rev-parse --show-toplevel)"
readonly REPO_ROOT
readonly ALLOWLIST="$REPO_ROOT/meta/shell-coverage-allowlist.txt"
usage() {
echo "Usage: $SCRIPT_NAME [--all | --seed | <file>...]"
echo "Options:"
echo " --all Report every uncovered shell library"
echo " --seed Rewrite the allowlist from the current tree"
echo " -h, --help Show this help"
echo
echo "A shell library is covered when its lib/ directory has a"
echo "tests/run_all.sh suite. Listed files are exempt until modified."
exit 0
}
# True for a shell library: a .sh under a lib/ dir, excluding the tests
# themselves and vendored trees. Payloads are data emitted onto other
# machines, not logic this repo's suites can source.
is_shell_lib() {
local path="$1"
[[ "$path" == *.sh ]] || return 1
[[ "$path" == */lib/* ]] || return 1
[[ "$path" == */lib/tests/* ]] && return 1
[[ "$path" == */lib/payloads/* ]] && return 1
[[ "$path" == third_party/* ]] && return 1
return 0
}
# A lib is covered when a run_all.sh sits in its directory's tests/ subdir.
is_covered() {
local path="$1"
[[ -f "$(dirname "$path")/tests/run_all.sh" ]]
}
collect_libs() {
git -C "$REPO_ROOT" ls-files '*.sh'
}
# Every in-scope lib that has no suite beside it.
report_uncovered() {
local path
for path in "$@"; do
is_shell_lib "$path" || continue
[[ -f "$REPO_ROOT/$path" ]] || continue
is_covered "$REPO_ROOT/$path" || printf '%s\n' "$path"
done
}
seed_allowlist() {
local -a libs=()
mapfile -t libs < <(collect_libs)
{
echo "# Shell libraries that predate the coverage ratchet."
echo "#"
echo "# Each line is a file exempt from check_shell_coverage.sh until it is"
echo "# next modified. Modifying an exempt file requires giving its lib/"
echo "# directory a tests/run_all.sh suite and deleting the line."
echo "#"
echo "# This list only ever shrinks. Never add a path to it by hand -- a new"
echo "# library must ship with its tests."
echo "#"
echo "# Regenerate (only ever to REMOVE now-covered entries):"
echo "# bash meta/scripts/check_shell_coverage.sh --seed"
report_uncovered "${libs[@]}" | sort
} >"$ALLOWLIST"
local count
count="$(grep -cv '^#' "$ALLOWLIST" || true)"
echo "$SCRIPT_NAME: seeded allowlist with $count uncovered librar(ies)"
}
# The allowlist minus its comment lines.
exempt_paths() {
[[ -f "$ALLOWLIST" ]] || return 0
grep -v '^#' "$ALLOWLIST" | grep -v '^[[:space:]]*$' || true
}
is_exempt() {
local path="$1" entry
while IFS= read -r entry; do
[[ "$entry" == "$path" ]] && return 0
done < <(exempt_paths)
return 1
}
# Gate the named files: uncovered AND not exempt is a failure.
check_files() {
local -a offenders=()
local path
while IFS= read -r path; do
[[ -n "$path" ]] || continue
is_exempt "$path" || offenders+=("$path")
done < <(report_uncovered "$@")
if ((${#offenders[@]} == 0)); then
echo "$SCRIPT_NAME: every checked shell library has a test suite"
return 0
fi
echo "$SCRIPT_NAME: ${#offenders[@]} shell librar(ies) without a test suite:" >&2
for path in "${offenders[@]}"; do
printf ' %s\n' "$path" >&2
printf ' needs: %s/tests/run_all.sh\n' "$(dirname "$path")" >&2
done
echo >&2
echo " A new or modified shell library must ship with a suite that" >&2
echo " sources it. Extend the nearest existing tests/ directory rather" >&2
echo " than cloning a harness -- jscpd fails above 2% duplication." >&2
return 1
}
report_all() {
local -a libs=()
mapfile -t libs < <(collect_libs)
local uncovered exempt_count total
uncovered="$(report_uncovered "${libs[@]}")"
exempt_count="$(exempt_paths | wc -l)"
total="$(printf '%s\n' "$uncovered" | grep -c . || true)"
echo "$SCRIPT_NAME: $total uncovered shell librar(ies), $exempt_count exempt"
local path
while IFS= read -r path; do
[[ -n "$path" ]] || continue
if is_exempt "$path"; then
printf ' exempt %s\n' "$path"
else
printf ' BLOCK %s\n' "$path"
fi
done <<<"$uncovered"
}
main() {
if [[ $# -eq 0 ]]; then
echo "Error: pass --all, --seed, or one or more file paths" >&2
exit 1
fi
case "$1" in
--seed)
seed_allowlist
;;
--all)
report_all
;;
*)
check_files "$@"
;;
esac
}
while [[ $# -gt 0 ]]; do
case "$1" in
-h | --help)
usage
;;
*)
break
;;
esac
done
main "$@"

View File

@ -0,0 +1,136 @@
# Shell libraries that predate the coverage ratchet.
#
# Each line is a file exempt from check_shell_coverage.sh until it is
# next modified. Modifying an exempt file requires giving its lib/
# directory a tests/run_all.sh suite and deleting the line.
#
# This list only ever shrinks. Never add a path to it by hand -- a new
# library must ship with its tests.
#
# Regenerate (only ever to REMOVE now-covered entries):
# bash meta/scripts/check_shell_coverage.sh --seed
linux_configuration/.githooks/lib/common.sh
linux_configuration/scripts/boot_recovery/tests/lib/boot_repair_cases.sh
linux_configuration/scripts/lib/android.sh
linux_configuration/scripts/lib/common_datetime.sh
linux_configuration/scripts/lib/common_packages.sh
linux_configuration/scripts/lib/common_services.sh
linux_configuration/scripts/lib/common.sh
linux_configuration/scripts/lib/mtk_classify.sh
linux_configuration/scripts/lib/mtk_common.sh
linux_configuration/scripts/lib/mtk_device.sh
linux_configuration/scripts/lib/mtk_partitions.sh
linux_configuration/scripts/meta/lib/shell_check_discover.sh
linux_configuration/scripts/meta/lib/shell_check_install.sh
linux_configuration/scripts/meta/lib/shell_check_lint.sh
linux_configuration/scripts/mtk_root/lib/dump_partitions.sh
linux_configuration/scripts/mtk_root/lib/dump_verify.sh
linux_configuration/scripts/mtk_root/lib/mtk_udev.sh
linux_configuration/scripts/mtk_root/lib/recon_report.sh
linux_configuration/scripts/mtk_root/tests/lib/assertions.sh
linux_configuration/scripts/mtk_root/tests/lib/dump_tests.sh
linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/lib/install_files.sh
linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/lib/integrity.sh
linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/lib/managed_copy.sh
linux_configuration/scripts/periodic_background/digital_wellbeing/pacman/lib/protect_and_extras.sh
linux_configuration/scripts/periodic_background/digital_wellbeing/virtualbox/lib/vbox_disk_inject.sh
linux_configuration/scripts/periodic_background/system-maintenance/bin/lib/catchup_timer.sh
linux_configuration/scripts/periodic_background/system-maintenance/bin/lib/distro_detect.sh
linux_configuration/scripts/periodic_background/system-maintenance/bin/lib/nvidia_pmon.sh
linux_configuration/scripts/periodic_background/system-maintenance/bin/lib/packages.sh
linux_configuration/scripts/periodic_background/system-maintenance/bin/lib/system_services.sh
linux_configuration/scripts/single_use/features/lib/aw_autostart.sh
linux_configuration/scripts/single_use/features/lib/aw_install.sh
linux_configuration/scripts/single_use/features/lib/aw_service.sh
linux_configuration/scripts/single_use/features/lib/dns_dhcp.sh
linux_configuration/scripts/single_use/features/lib/dns_firewall.sh
linux_configuration/scripts/single_use/features/lib/dns_units.sh
linux_configuration/scripts/single_use/features/lib/dot_resolver_install.sh
linux_configuration/scripts/single_use/features/lib/dwm_config.sh
linux_configuration/scripts/single_use/features/lib/gitea_network.sh
linux_configuration/scripts/single_use/features/lib/kcd2_autorebuild.sh
linux_configuration/scripts/single_use/features/lib/kcd2_expose.sh
linux_configuration/scripts/single_use/features/lib/kcd2_serve.sh
linux_configuration/scripts/single_use/features/lib/local_llm_model.sh
linux_configuration/scripts/single_use/features/lib/mobile_service.sh
linux_configuration/scripts/single_use/features/lib/nc_discover.sh
linux_configuration/scripts/single_use/features/lib/nc_flash.sh
linux_configuration/scripts/single_use/features/lib/nc_image.sh
linux_configuration/scripts/single_use/features/lib/nc_packages.sh
linux_configuration/scripts/single_use/features/lib/nc_php.sh
linux_configuration/scripts/single_use/features/lib/nc_remote_flash.sh
linux_configuration/scripts/single_use/features/lib/nc_remote.sh
linux_configuration/scripts/single_use/features/lib/nc_services.sh
linux_configuration/scripts/single_use/features/lib/pi_flash_exec.sh
linux_configuration/scripts/single_use/features/lib/pi_flash_image.sh
linux_configuration/scripts/single_use/features/lib/pi_flash_remote.sh
linux_configuration/scripts/single_use/features/lib/rpi_nc_ca.sh
linux_configuration/scripts/single_use/features/lib/rpi_nc_deps.sh
linux_configuration/scripts/single_use/features/lib/rpi_nc_fixes.sh
linux_configuration/scripts/single_use/features/lib/rpi_nc_install.sh
linux_configuration/scripts/single_use/features/lib/rpi_nc_ssl_remote.sh
linux_configuration/scripts/single_use/features/lib/rpi_nc_ssl.sh
linux_configuration/scripts/single_use/features/lib/rpi_nc_system.sh
linux_configuration/scripts/single_use/features/lib/searxng_expose.sh
linux_configuration/scripts/single_use/features/lib/searxng_limiter.sh
linux_configuration/scripts/single_use/features/lib/searxng_settings.sh
linux_configuration/scripts/single_use/features/lib/searxng_stack.sh
linux_configuration/scripts/single_use/features/lib/searxng_start.sh
linux_configuration/scripts/single_use/features/lib/searxng_status.sh
linux_configuration/scripts/single_use/features/lib/syncyomi_expose.sh
linux_configuration/scripts/single_use/features/lib/syncyomi_stack.sh
linux_configuration/scripts/single_use/features/lib/unreal_mcp_repo.sh
linux_configuration/scripts/single_use/features/lib/website_build.sh
linux_configuration/scripts/single_use/features/lib/wg_firewall.sh
linux_configuration/scripts/single_use/features/lib/wg_keys.sh
linux_configuration/scripts/single_use/features/lib/wg_sshd.sh
linux_configuration/scripts/single_use/fixes/lib/arch_cpu.sh
linux_configuration/scripts/single_use/fixes/lib/arch_hardware.sh
linux_configuration/scripts/single_use/fixes/lib/arch_perf_fixes.sh
linux_configuration/scripts/single_use/fixes/lib/arch_perf_probes.sh
linux_configuration/scripts/single_use/fixes/lib/arch_perf_report.sh
linux_configuration/scripts/single_use/fixes/lib/arch_sysctl.sh
linux_configuration/scripts/single_use/fixes/lib/bt_adapter.sh
linux_configuration/scripts/single_use/fixes/lib/bt_audio.sh
linux_configuration/scripts/single_use/fixes/lib/bt_pairing.sh
linux_configuration/scripts/single_use/fixes/lib/bt_report.sh
linux_configuration/scripts/single_use/fixes/lib/hosts_guard_migrate.sh
linux_configuration/scripts/single_use/fixes/lib/hosts_guard_rollback.sh
linux_configuration/scripts/single_use/fixes/lib/nvidia_config.sh
linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_capture.sh
linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_load.sh
linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_setup.sh
linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_summary.sh
linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_usage.sh
linux_configuration/scripts/single_use/fixes/lib/pacman_hook_stall_watch.sh
linux_configuration/scripts/single_use/fixes/lib/thorium_repairs.sh
linux_configuration/scripts/single_use/fixes/lib/ubuntu_perf_fixes.sh
linux_configuration/scripts/single_use/fixes/lib/ubuntu_perf_more.sh
linux_configuration/scripts/single_use/fresh-install/lib/packages.sh
linux_configuration/scripts/single_use/lib/joplin_server.sh
linux_configuration/scripts/single_use/misc/testsAndMisc-bash/lib/clean_audio_filters.sh
linux_configuration/scripts/single_use/misc/testsAndMisc-bash/lib/transcribe_deps.sh
linux_configuration/scripts/single_use/misc/testsAndMisc-bash/lib/transcribe_pkgmgr.sh
linux_configuration/scripts/single_use/misc/testsAndMisc-bash/lib/transcribe_venv.sh
linux_configuration/scripts/single_use/misc/testsAndMisc-bash/lib/unity_handler.sh
linux_configuration/tests/lib/i3blocks_claude_usage.sh
linux_configuration/tests/lib/i3blocks_config_checks.sh
linux_configuration/tests/lib/pacman_wrapper_makepkg.sh
linux_configuration/tests/lib/pacman_wrapper_syntax.sh
linux_configuration/tests/lib/security_apps.sh
linux_configuration/tests/lib/security_hosts_shutdown.sh
linux_configuration/tests/lib/security_pacman.sh
meta/lib/lint_full_tools.sh
meta/lib/lint_output.sh
meta/scripts/lib/disk_cleanup_common.sh
meta/scripts/lib/trace_prefix.sh
meta/scripts/lib/trace_stubs.sh
phone_focus_mode/lib/adb_common.sh
phone_focus_mode/lib/adb_locking.sh
phone_focus_mode/lib/adb_trusted.sh
phone_focus_mode/lib/backup_capture.sh
phone_focus_mode/lib/backup.sh
phone_focus_mode/lib/monitor_checks_health.sh
phone_focus_mode/lib/monitor_checks_policy.sh
phone_focus_mode/lib/monitor.sh
phone_focus_mode/lib/restore.sh