Duty cycles and stale tests

testsAndMisc · 9 commits, 836f4ef..5faed23 · 2026-07-19 · 24 files, +1122 / −43

1. Background

testsAndMisc is a monorepo of personal automation. The part this change touches, linux_configuration/, holds the shell scripts that run this Arch machine: a pacman wrapper that blocks certain packages, guards that keep /etc/hosts immutable so a blocklist can't be edited away, a daemon that stops music playing alongside focus apps, and a scheduler that locks the desktop at night.

Alongside them sits linux_configuration/tests/ — fourteen plain-bash test scripts with hand-rolled fail and assert_equals helpers. They look like a test suite. They had one problem: nothing ran them. The repo's CI had two workflows, pre-commit.yml and python-tests.yml, and neither so much as globbed for a .sh file. The tests existed, were committed, were never executed.

That matters because of what happened next. At some point scripts/ was reorganised and most things moved under a new periodic_background/ directory. Nothing told the tests. Four of them kept pointing at paths that no longer existed, and because no runner ever invoked them, nobody found out.

The thing being measured

Throughput on this machine is read straight from the kernel's own byte counter, /sys/class/net/enp6s0/statistics/rx_bytes. Sample it twice, subtract, divide by the elapsed seconds, and you have a rate — no external tools, no forks in the loop. The only real decision is how often you sample, and that decision turned out to be the whole story.

2. Intuition

Start with the number everyone quotes: the download client said ~170 Mbps. Sampling the NIC every five seconds agreed — a flat, believable plateau, the kind of figure you take to an ISP.

Sampling the same link every one second showed something else entirely. There was no plateau. The link alternated between multi-hundred-Mbps bursts and seconds of complete silence:

The average was arithmetically correct and descriptively useless. It reported a rate that no individual second ever ran at. Once you can see the shape, the quantity that matters factors cleanly:

This reframes the diagnosis. A slow burst rate means a network problem: congestion, peering, distance. A low duty cycle means something is choosing not to transfer, and no amount of changing servers will help. Averaging collapses both into one number and throws away the distinction — so every remedy aimed at the wrong factor looks equally ineffective, which is exactly how the investigation had been going.

Play with it

Drag the two factors. The top strip is what 1-second sampling sees; the bars below are the same traffic averaged into 5-second buckets — the view that hid the problem.

The second theme: tests that describe a world that moved

Running the fourteen shell tests for the first time surfaced four distinct failures, and all four were the same bug wearing different clothes — a test still naming something by its old name:

The last row is worth pausing on. That test demanded a literal sleep 60 polling loop. The code had since been improved to sleep once, until the next day boundary, then block on wait — strictly fewer wakeups. The test failed the code for getting better. A test pinned to implementation text rather than the property it cares about will eventually do that.

The failure mode worth remembering

One test failed in a way that looked nothing like a path bug. test_music_parallelism.sh copies the daemon into a temporary worktree and drops a stub common.sh beside it, so it can mock the desktop. The daemon resolves its library two levels up:

if [[ -f "$SCRIPT_DIR/../../lib/common.sh" ]]; then
  source "$SCRIPT_DIR/../../lib/common.sh"
elif [[ -f "/usr/local/lib/common.sh" ]]; then   # ← the trap
  source "/usr/local/lib/common.sh"

The reorganisation left the test building its worktree one directory too shallow. From there, ../../lib/common.sh resolved outside the worktree and missed the stub — so the elif fired and the daemon quietly loaded the real, installed library instead. The test then ran against the live machine, ignored its own mock, and reported expected 30, actual 15: a confident, specific, entirely fictitious daemon bug.

The fix does two things, and the second matters more. It derives the worktree path from the script's real position, so the depth can never drift again — and it makes the stub write a marker that every run asserts, so if the wrong library is ever loaded the test says so instead of inventing a number. Fixing the instance would have lasted until the next reorganisation; fixing the class ends it.

3. Code

a. Making bash unit-testable

The two new diagnostics live in linux_configuration/diagnostics/. Their arithmetic — rate conversion, duty percentage, the mean of a sample window — is pulled out into pure functions, with the measurement loop kept separate:

bytes_to_mbps() {
    local delta="$1" secs="$2"
    (( secs > 0 )) || { printf '0'; return; }        # no divide-by-zero
    printf '%d' $(( delta * 8 / secs / 1000000 ))
}

# ...

if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    main "$@"
fi

That last guard is what makes the whole thing testable. Executed directly, the script runs a measurement; sourced, it defines its helpers and stops. The bats suite sources both scripts in setup() and asserts on the functions — no sleeping, no live NIC, no mocks.

Thirty assertions cover the boundaries, and one of them earned its keep immediately: the sustained_of test was written expecting 30 for a three-sample input and got 25. The code was right — odd counts drop only the first sample, keeping two of three. The expectation was wrong, and it was fixed rather than the behaviour, because 25 is what produced every figure in the investigation.

b. Two CI jobs, split by what they need

.github/workflows/shell-tests.yml runs the suite in two places, because the tests divide cleanly:

Two details were found empirically rather than assumed. archlinux:base-devel ships no git, so actions/checkout can't clone until it's installed; and the i3blocks test counts execve calls with strace, which needs --cap-add=SYS_PTRACE. The whole set was validated locally in Docker — including a --network=none run, which proves the job's dependency list is complete rather than quietly pulling something at runtime.

c. The stale-name repairs

Mechanical, once identified: four tests had their source paths corrected, three renamed symbols were updated, and three systemd unit checks were widened to accept either generation —

if systemctl is-active --quiet guard-file@hosts.path 2>/dev/null ||
    systemctl is-active --quiet hosts-guard.path 2>/dev/null; then

— with the bind-mount check also accepting the read-only mount on /etc/hosts itself, since the mount is the actual invariant and which unit established it is an implementation detail.

d. A security test that works in both worlds

test_security_hardening.sh is really two suites in one file: seven checks read repository files, the rest inspect a configured machine (immutable flags, systemd units, /usr/local payloads). In a container the second group can only fail. A LIVE_SECTION flag now marks them, and off-host they report skip instead.

The safeguard against over-leniency is a negative test: faking the host markers inside a container with no protection installed still produces ten failures. Green means protected, not permissive — and that check is recorded in the evidence artifact as mandatory for any future change here.

e. Outcomes over mechanisms

This section exists because of a wrong conclusion. Four guard units read as dead, and the obvious reading — protection is down — was reported as such. It wasn't true. A refactor had replaced the standalone units with templated ones, and the replacements were running the whole time. What settled it was checking the outcome rather than the mechanism:

lsattr /etc/hosts          →  ----i---------e-------   # immutable
findmnt -no OPTIONS /etc/hosts  →  ro,noatime          # read-only
systemctl is-active guard-file@hosts.path  →  active

Unit names are an implementation detail and they churn. The invariant — that the file cannot be written — does not. Tests that assert the invariant survive refactors; tests that assert the mechanism raise false alarms, and a false alarm on a security control is expensive in a way a false alarm elsewhere is not.

4. Quiz

Five questions on the reasoning, not the trivia. Click an option to reveal.