signal-bot/scripts/install_service.sh
Krzysztof kuhy Rudnicki d5c63b423b feat: refuse to look healthy while deaf, and heal the container by itself
Today the bot was up, healthy and answering nothing for three hours. A host
firewall change (fixed in testsAndMisc 334c0cd9) cut every Docker container
off the internet; signal-cli lost Signal and reconnected in a loop from
15:32:50 onward. Nothing noticed:

  - the container's own healthcheck answered 204
  - /v1/groups answered 200 out of local state, so discovery saw no change
  - the bot's WebSocket to the container stayed OPEN, so the transport's
    reconnect logic never fired
  - /healthz answered {"status": "ok"} the entire time

The only symptom was silence, which is also what a quiet evening looks like.

signal_bot/liveness.py asks the one question that cannot be answered from a
cache -- GET /v1/devices, which needs a live round trip -- but only after
SILENCE_SECONDS (120) with no frame of any kind, so a busy group costs
nothing and an idle one costs one request per window. Three consecutive
failures and /healthz answers 503 with the reason. Verified end to end in
scripts/e2e.sh against the real process: 503 after 84s with the fake's
connection cut, back to 200 on its own 30s after restoring it.

scripts/signal_api_watchdog.sh + a two-minute user timer do the recovery the
bot cannot do for itself, since a wedged bot cannot restart the container it
depends on. Two tiers, so the steady state costs no Signal round trip at
all: the bot's own /healthz first, the authenticated probe only when that is
not 200. It stops after 3 restarts in 6 hours and escalates instead -- a
deregistered account or a blocked network is not fixable by restarting, and
a restart loop would hide it. Its own bash suite stubs curl and docker and
drives every branch including the give-up cap.

signal_bot/trust.py answers the other half: TRUSTED_INVITERS decides who may
put this bot in a group. The gate sits in GroupDirectory._permitted, which
both _join_pending and _adopt already route through -- gating only invites
would gate the door nobody uses, since prime_profile.sh exists precisely so
that an add becomes a direct join. Evidence is either the group-update
envelope, which is the only frame naming an inviter, or a trusted identity
being a full member in the listing. A refusal is logged once, with that
group's members, so the operator can see which string to add.

verify_allowlist moved to membership.py to keep discovery.py under the
250-line gate; it was already the module that owns MembershipError.

Verified live, by stopping the container by hand and touching nothing else:

  18:54:35  docker stop signal-api
  18:56:36  liveness: failure 1 of 3 after 123s of silence
  18:57:36  failure 3 of 3 -> /healthz answers 503 from 18:57:45
  18:58:18  watchdog: probe failed (1/2)
  19:00:28  watchdog: probe failed (2/2) -> docker restart signal-api
  19:00:36  liveness: receive path recovered after 8 failed probe(s)
  19:01:13  watchdog: signal-api is back

Six minutes, unattended, against three hours of silence this afternoon. All
four real groups are still adopted with TRUSTED_INVITERS=+48123456789 set,
and no group was refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbjfWdAyHbCubac7wJKJZW
2026-08-29 19:01:55 +02:00

151 lines
4.7 KiB
Bash
Executable File

#!/bin/bash
# ============================================================================
# Install, start and verify the signal-bot user service.
#
# A *user* unit, not a system one: the bot runs as whoever registered the
# Signal account and reads that user's .env. Lingering is what makes it start
# at boot with nobody logged in -- without it, "survives reboots" is a lie the
# unit file cannot tell you about, so this script checks and fails loudly.
# ============================================================================
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
readonly REPO_ROOT
SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_NAME
readonly UNIT_NAME="signal-bot.service"
readonly UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
readonly TEMPLATE="${REPO_ROOT}/scripts/${UNIT_NAME}"
readonly WATCHDOG_UNITS=("signal-api-watchdog.service" "signal-api-watchdog.timer")
readonly WATCHDOG_TIMER="signal-api-watchdog.timer"
readonly DEFAULT_PORT=8137
VERIFY_ONLY="false"
# shellcheck source=scripts/lib_signal.sh
source "${REPO_ROOT}/scripts/lib_signal.sh"
usage() {
echo "Usage: $SCRIPT_NAME [--verify-only]"
echo " --verify-only Check the running service without reinstalling it."
exit 0
}
die() {
echo "Error: $1" >&2
exit 1
}
# The port the bot will actually bind: .env wins, code default otherwise.
bot_port() {
local port=""
if [[ -f "${REPO_ROOT}/.env" ]]; then
port="$(sed -n 's/^BIND_PORT=[[:space:]]*//p' "${REPO_ROOT}/.env" | tail -1)"
fi
printf '%s' "${port:-$DEFAULT_PORT}"
}
validate_requirements() {
command -v systemctl >/dev/null || die "systemctl not found; this needs systemd"
[[ -f "${REPO_ROOT}/.env" ]] ||
die "no ${REPO_ROOT}/.env -- run scripts/register_bot.sh first"
[[ -x "${REPO_ROOT}/run.sh" ]] || die "no executable ${REPO_ROOT}/run.sh"
[[ -f "$TEMPLATE" ]] || die "no unit template at $TEMPLATE"
[[ -x "${REPO_ROOT}/scripts/signal_api_watchdog.sh" ]] ||
die "no executable ${REPO_ROOT}/scripts/signal_api_watchdog.sh"
local unit
for unit in "${WATCHDOG_UNITS[@]}"; do
[[ -f "${REPO_ROOT}/scripts/${unit}" ]] || die "no unit template for ${unit}"
done
}
# Without linger, a user unit dies at logout and never starts at boot.
require_linger() {
if [[ "$(loginctl show-user "$USER" --property=Linger --value)" == "yes" ]]; then
echo "== linger enabled: the bot starts at boot with nobody logged in"
return 0
fi
echo "== enabling linger for $USER"
if ! sudo -n loginctl enable-linger "$USER" 2>/dev/null; then
echo "Error: linger is off and sudo needs a password here." >&2
echo " Without it the bot will NOT come back after a reboot." >&2
echo " Run this, then re-run with --verify-only:" >&2
echo >&2
echo " sudo loginctl enable-linger $USER" >&2
return 1
fi
}
install_unit() {
mkdir -p "$UNIT_DIR"
sed "s|@REPO_ROOT@|${REPO_ROOT}|g" "$TEMPLATE" > "${UNIT_DIR}/${UNIT_NAME}"
systemctl --user daemon-reload
systemctl --user enable "$UNIT_NAME" >/dev/null
systemctl --user restart "$UNIT_NAME"
echo "== installed and started ${UNIT_DIR}/${UNIT_NAME}"
}
# The bot cannot restart the container it depends on from inside a wedged
# event loop, so recovery from a dead Signal connection lives in a timer.
install_watchdog() {
local unit
for unit in "${WATCHDOG_UNITS[@]}"; do
sed "s|@REPO_ROOT@|${REPO_ROOT}|g" "${REPO_ROOT}/scripts/${unit}" >"${UNIT_DIR}/${unit}"
done
systemctl --user daemon-reload
systemctl --user enable --now "$WATCHDOG_TIMER" >/dev/null
echo "== installed and armed ${UNIT_DIR}/${WATCHDOG_TIMER}"
}
# Report what actually happened, not that the commands returned 0.
verify() {
local port="$1"
if ! systemctl --user is-active --quiet "$UNIT_NAME"; then
echo "Error: ${UNIT_NAME} is not active." >&2
systemctl --user --no-pager --lines=20 status "$UNIT_NAME" >&2 || true
return 1
fi
signal_wait_for_url "http://127.0.0.1:${port}/healthz" "signal-bot" 60
echo "== active, and answering on http://127.0.0.1:${port}/healthz"
echo "== enabled at boot: $(systemctl --user is-enabled "$UNIT_NAME")"
if ! systemctl --user is-active --quiet "$WATCHDOG_TIMER"; then
echo "Error: ${WATCHDOG_TIMER} is not active; nothing will recover a dead" >&2
echo " connection to Signal automatically." >&2
return 1
fi
systemctl --user list-timers --no-pager "$WATCHDOG_TIMER" | sed -n '2p'
}
main() {
validate_requirements
local port
port="$(bot_port)"
if [[ "$VERIFY_ONLY" != "true" ]]; then
require_linger
install_unit
install_watchdog
fi
verify "$port"
echo "== logs: journalctl --user -u ${UNIT_NAME} -f"
}
while [[ $# -gt 0 ]]; do
case $1 in
--verify-only)
VERIFY_ONLY="true"
shift
;;
-h | --help) usage ;;
*)
echo "Unknown option: $1" >&2
exit 1
;;
esac
done
main "$@"