signal-bot/scripts/test_watchdog_harness.sh
Krzysztof kuhy Rudnicki 3a348a550c
Some checks are pending
CI / check (push) Waiting to run
fix: the watchdog stops burying the desktop in popups nobody asked for
Three defects, and the loudest one was self-inflicted.

test_watchdog.sh stubbed only curl and docker, so notify-send resolved to
the real binary and the suite fired a dozen genuine critical popups every
run -- from a pre-commit hook. On 2026-08-31 the timer emitted zero
messages and the desktop still collected 297, all of them from committing.
The stubs now live in the harness, and the suite asserts the counts.

give_up() had no once-only flag, so a six-hour window announced itself
every two minutes -- up to 180 times -- while its own text said "no
further restarts". A stamp in the state dir now gates the popup, and only
the popup: journald still gets every line, which is why the existing
stdout assertions are untouched.

alarm() hardcoded --urgency=critical, which xfce4-notifyd never expires,
and passed no replace id, so they stacked instead of superseding.
notify-send --print-id now feeds --replace-id from one slot. WATCHDOG_NOTIFY
picks how much reaches the desktop at all; it defaults to escalation, so
the routine restart/recovery pair is journald-only from here.

test_watchdog.sh crossed the 250-line cap, so the sandbox and the
assertions moved to test_watchdog_harness.sh, and the hook that runs the
suite now watches all three files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01416TegJ3sa8xL5qGGgrygP
2026-08-31 14:27:03 +02:00

107 lines
2.4 KiB
Bash

# shellcheck shell=bash
# ============================================================================
# The sandbox and the assertions for test_watchdog.sh.
#
# Split out when the two together crossed the shared 250-line cap. Sourced,
# never executed: it defines the stub PATH the suite runs against and the
# expect_* helpers, and it owns WORK/FAILURES set by its caller.
# ============================================================================
# Rebuild a pristine sandbox: stub binaries, stub answers, empty state.
setup() {
rm -rf "$WORK"
mkdir -p "$WORK/bin" "$WORK/state"
printf '200' >"$WORK/bot_code"
printf '200' >"$WORK/api_code"
printf '0' >"$WORK/fix_on_restart"
cat >"$WORK/bin/curl" <<EOF
#!/bin/bash
url="\${*: -1}"
case "\$url" in
*/healthz) cat "$WORK/bot_code" ;;
*) cat "$WORK/api_code" ;;
esac
EOF
cat >"$WORK/bin/docker" <<EOF
#!/bin/bash
echo "\$@" >> "$WORK/docker.log"
if [[ "\$(cat "$WORK/fix_on_restart")" == "1" ]]; then
printf '200' > "$WORK/bot_code"
printf '200' > "$WORK/api_code"
fi
EOF
# Stubbed because they are NOT stubbed by default: notify-send and logger
# resolve to the real binaries, so this suite used to fire a dozen genuine
# critical popups per run -- and it runs from pre-commit, which made every
# commit spam the desktop. notify.log is now also what proves the dedup.
cat >"$WORK/bin/notify-send" <<EOF
#!/bin/bash
echo "\$@" >> "$WORK/notify.log"
echo 4242
EOF
cat >"$WORK/bin/logger" <<EOF
#!/bin/bash
echo "\$@" >> "$WORK/logger.log"
EOF
chmod +x "$WORK/bin/curl" "$WORK/bin/docker" "$WORK/bin/notify-send" "$WORK/bin/logger"
}
notify_count() {
[[ -f "$WORK/notify.log" ]] || {
printf '0'
return 0
}
grep -c '' "$WORK/notify.log"
}
expect_notifications() {
local actual
actual="$(notify_count)"
if [[ "$actual" == "$1" ]]; then
ok "$2"
else
fail "$2 -- got $actual popup(s), expected $1"
fi
}
run_watchdog() {
PATH="$WORK/bin:$PATH" \
PHONE_NUMBER="+15550000000" \
STATE_DIRECTORY="$WORK/state" \
SETTLE_SECONDS=0 \
FAILURES_BEFORE_RESTART=2 \
MAX_RESTARTS=2 \
RESTART_WINDOW=3600 \
"$WATCHDOG" 2>&1
}
expect_contains() {
if [[ "$1" == *"$2"* ]]; then
ok "$3"
else
fail "$3 -- got: $1"
fi
}
expect_missing() {
if [[ "$1" != *"$2"* ]]; then
ok "$3"
else
fail "$3 -- unexpectedly found $2"
fi
}
expect_file() {
local actual
actual="$(cat "$1" 2>/dev/null || echo MISSING)"
if [[ "$actual" == "$2" ]]; then
ok "$3"
else
fail "$3 -- $1 is '$actual', expected '$2'"
fi
}