testsAndMisc/docs/DOCS-shell-split-harness.md

Shell split verification: the harness

Split out of docs/shell-split-verification.md when that file hit the 250-line cap. That file covers what makes a green split wrong; this one covers the tooling that catches it — the trace harness, its --prefix mode, the regression fixtures, and two traps that have each cost a revert.

Read docs/shell-split-verification.md first: the hazards there decide whether the harness can prove anything about your target at all.

trace_shell_split.sh — the run, made routine

meta/scripts/trace_shell_split.sh (added 2026-08-16) does the stubbed run for you. It shadows ~30 mutating binaries (sudo, pacman, systemctl, adb, nft, mount, mkinitcpio, reboot, …) with PATH stubs that record each call and exit 0, runs the script, and prints a diffable trace: exit status, stubbed calls in order with arguments, stdout, stderr.

# baseline, in a detached worktree at the pre-split commit
meta/scripts/trace_shell_split.sh <script> --out /tmp/before.txt
# after the split
meta/scripts/trace_shell_split.sh <script> --out /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt

It was verified against both bugs above: the set -e function tail shows up as exit 1 with later calls missing, and the nameref as circular name reference in stderr despite exit 0. A fixture calling systemctl left the unit untouched.

Four things to watch:

• A silent stub produces an identical, meaningless trace. This is the trap. du stubbed to print nothing makes size=$(du -sk x) empty, so every ((size > 0)) is false, every guarded branch is skipped, and two such traces match perfectly while exercising none of the code you changed. Give value-producing commands a plausible output: --stub 'du=4096\t/some/dir'. Measured on a fixture: the empty stub never reached the guarded sudo rm -rf; the valued stub printed BRANCH TAKEN and captured it. Ask of every trace: did this actually go down the branch I touched?
• Stubs always exit 0. A script that branches on a real failure status takes a path it would not take live. Hand-write a stub for those and say so in the split's evidence file.
• Anything not stubbed runs for real. git, curl, wget and makepkg are deliberately excluded from the default set, because many scripts read git state harmlessly and a blanket stub would change what they see. Grep the target for network and build verbs, then pass them: --stub git,curl,makepkg.
• pkill is NOT stubbed and --prefix cannot contain it — signals are not filesystem writes. install_leechblock.sh:217 pkills every Chromium-family browser before seeding, so tracing it closes the user's browsers (measured: two probe runs fired it, harmless only because none was open). Grep a target for pkill/killall/systemctl stop and pass --stub pkill first. The mirror image also escapes the prefix: setup_thorium_startup.sh launches a browser mid-run ("Browser launched with PID: …"). Process spawning is not a file write either — check ps after tracing anything that starts a program.

--prefix — scripts whose job is placing files

Shell redirections (cat > /etc/modprobe.d/x.conf) are builtins, so no PATH stub can intercept them. Stub cp/chmod/ln and the trace is nearly empty; don't, and the run rewrites the live system. --prefix (added 2026-08-16) resolves that in two layers — see meta/scripts/lib/trace_prefix.sh:

# $HOME-rooted destinations: no bwrap, no root needed
meta/scripts/trace_shell_split.sh install_usage_monitoring.sh --prefix /tmp/pfx-before

# hardcoded absolutes: bind each LEAF directory
meta/scripts/trace_shell_split.sh nvidia_troubleshoot.sh --prefix /tmp/pfx-nv \
  --bind-abs /etc/modprobe.d --bind-abs /etc/X11 --stub git,lsmod,nvidia-smi

Every run needs its own empty prefix, hence the -before/-after naming. Reusing one leaves the first run's files in place, so a dropped write stops showing up entirely — measured: the fixture pair below diffs clean on a reused prefix. The harness refuses a non-empty prefix rather than deleting it.

HOME and every XDG_* base are redirected. Exporting the XDG vars explicitly is not belt-and-braces: install_leechblock.sh reads XDG_DATA_HOME directly and then runs rsync -a --delete at it, so inheriting the caller's real value would delete live data.

The trace gains a === files written (prefix) section listing every file with its size and a content hash. That section is the point: without it a dropped cat > shows up as nothing at all — same exit status, same stubbed calls, same stdout — and the traces match while one lost a file. Measured on the dropped_write_{before,after}.sh fixture pair: with the manifest the lost .timer is a one-line diff; stripped, the two traces are byte-identical.

Six traps, each of which cost a debugging round:

• /usr/bin cannot be bound at all. --bind-abs /usr/bin overlays the directory the sandbox needs to exec anything (bwrap: execvp bash). A script whose last act replaces a binary there (install_pacman_wrapper.sh:304, ln -sf … /usr/bin/pacman) therefore stops at that line with exit 1 on EPERM. Stable across runs, so still a usable baseline — but it proves nothing at or after the failing line, so keep those statements in the entry script. The scanner misses /usr/bin here: it follows variables into cat >/cp, so a literal ln -sf (or cp "$f" "$f.backup.$(date …)") escapes it.
• Bind leaf directories, never /etc. Binding /etc wholesale shadows /etc/passwd and /etc/os-release. Measured: $SUDO_USER lookup collapsed to /home//pyroveil, /etc/profile became "No such file or directory", and a nested /etc/X11 bind was masked so its write vanished from the manifest. All three look like a broken split. The scanner therefore never emits a bare /etc. A --bind-abs naming a file (/etc/profile) is bound file-over-file, seeded with a copy of the original — bwrap cannot mount a directory over a file. The copy matters because a script that greps a config before appending would otherwise read an empty file and branch wrong. Note a file bind cannot cover new siblings: cp "$f" "$f.backup.<stamp>" still EPERMs, which is what truncates nvidia_troubleshoot.sh's trace at step 3.
• require_root truncates a stubbed run to nothing. lib/common.sh does exec sudo "$0" "$@"; under the sudo stub that records one line and exits 0 — a three-line trace that diffs clean against any split. --bind-abs runs under bwrap --unshare-user --uid 0, so $EUID is 0 and the script proceeds for real. Without a bind there is no uid 0 and the truncation is silent, so check the trace actually entered the code you moved.
• Unquoted heredocs bake the prefix into the artifact. install_usage_monitoring.sh:222 uses << SCRIPT, so $HOME interpolates at write time. Hashes are taken over content with the prefix normalized to @PREFIX@; without that, a mktemp -d prefix makes every run differ.
• A timestamp the target embeds still varies. nvidia_troubleshoot.sh writes $(date) into its config, so its hash changes every run. Don't freeze the clock and don't fall back to comparing sizes — diff the written files' contents with the date pattern normalized away. That still catches a dropped or altered line, which a size comparison does not.

Anything absolute that was not bound fails read-only/EPERM rather than succeeding, so Decision 6 holds by construction. --prefix refuses to run when the target writes to absolute paths and no --bind-abs was given; the scan follows variables (MODPROBE_DIR="/etc/modprobe.d" → cat >"$CONFIG_FILE"), because a literal-only scan produced a silent empty manifest.

Verified untouched after the runs above: ~/.config/systemd/user, ~/.local/bin, ~/.local/share, /etc/modprobe.d (identical listings), and systemctl --user list-unit-files | grep -c usage-report unchanged at 2.

block_compulsive_opening.sh remains out of scope: install_all copies the running script into /usr/local/bin, and the handoff records that a bad split there breaks three daily-use apps plus the pacman rewrap hook.

Regression fixtures

meta/scripts/fixtures/ holds runnable reproductions of every trap above, so a harness change can be checked instead of trusted:

Fixture	Reproduces
set_e_function_tail.sh	exit 1, later calls missing from the trace
circular_nameref.sh	exit 0, correct stdout, warning only in stderr
valued_stub_branch.sh	the silent-stub trap (--stub du vs du=4096)
dropped_write_{before,after}.sh	a split that silently loses one write
xdg_destinations.sh	writes addressed via XDG_*, not $HOME

A lint fix once broke one of these invisibly: adding a trailing echo to satisfy SC2034 made that echo the function's last statement, so the fixture exited 0 and reproduced nothing while still "passing". Re-run the fixtures after touching them, and check the exit status is still the one documented.

Run the baseline from the script's own directory when it reads relative paths. fresh-install/main.sh reads aur_packages.txt and pacman_packages.txt from the cwd; tracing it from elsewhere dies at the first read and looks exactly like a broken split.

Note it does not need a main(), which matters: none of the remaining near-miss shell targets (install_usage_monitoring.sh, install_pacman_wrapper.sh, disk_cleanup_check.sh, fresh-install/main.sh) have one, so verify_shell_split.sh cannot check any of them.

Note the knock-on: renaming the nameref local hides the array from shellcheck (SC2034), because it is then only ever named as a bare word. Pass the name through a variable (CODE_FILES_MAP="LANG_CODE_FILES") so the reference is real, rather than suppressing.

Verify through every entry point, especially symlinks

Adding a source "$SCRIPT_DIR/lib/..." line to meta/lint_python.sh broke ./lint_python.sh from the repo root, because that root path is a symlink into meta/ (so are pyproject.toml, requirements.txt, run.sh, .fvmrc). SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" does not resolve symlinks — dirname is string manipulation — so it became the repo root, where lib/ does not exist, and set -euo pipefail turned that into an instant exit.

It was invisible because every check used bash meta/lint_python.sh, the one path that cannot fail. Before splitting a script, ask how else it is reachable:

find . -maxdepth 2 -type l -lname '*<script>*'   # symlinks pointing at it
grep -rn '<script>' .github/workflows/ .pre-commit-config.yaml meta/run.sh

The fix is the pattern already used elsewhere in this repo:

SCRIPT_DIR="$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")"

Note the pre-split script had the same unresolved SCRIPT_DIR; it only fed PROJECT_ROOT, so the bug was latent. Adding a source line is what turns a quiet wrong value into a hard failure — which is exactly what a split does.

Do not make a clean file shfmt-dirty

157 of 298 linux_configuration shell files already fail shfmt and there is no .editorconfig or shfmt pre-commit hook, so shell_check.sh's run_linters exits 1 on the full tree regardless. That is not licence to add more: check shfmt -d <file> before and after a split. analyze_repo.sh was clean and a split made it dirty over one double blank line.

shfmt -w also deletes comment banners it considers stray — it removed a #==== STEP header. Re-count the banners after running it.