Krzysztof kuhy Rudnicki
d5c63b423b
Today the bot was up, healthy and answering nothing for three hours. A host
firewall change (fixed in testsAndMisc 334c0cd9) cut every Docker container
off the internet; signal-cli lost Signal and reconnected in a loop from
15:32:50 onward. Nothing noticed:
- the container's own healthcheck answered 204
- /v1/groups answered 200 out of local state, so discovery saw no change
- the bot's WebSocket to the container stayed OPEN, so the transport's
reconnect logic never fired
- /healthz answered {"status": "ok"} the entire time
The only symptom was silence, which is also what a quiet evening looks like.
signal_bot/liveness.py asks the one question that cannot be answered from a
cache -- GET /v1/devices, which needs a live round trip -- but only after
SILENCE_SECONDS (120) with no frame of any kind, so a busy group costs
nothing and an idle one costs one request per window. Three consecutive
failures and /healthz answers 503 with the reason. Verified end to end in
scripts/e2e.sh against the real process: 503 after 84s with the fake's
connection cut, back to 200 on its own 30s after restoring it.
scripts/signal_api_watchdog.sh + a two-minute user timer do the recovery the
bot cannot do for itself, since a wedged bot cannot restart the container it
depends on. Two tiers, so the steady state costs no Signal round trip at
all: the bot's own /healthz first, the authenticated probe only when that is
not 200. It stops after 3 restarts in 6 hours and escalates instead -- a
deregistered account or a blocked network is not fixable by restarting, and
a restart loop would hide it. Its own bash suite stubs curl and docker and
drives every branch including the give-up cap.
signal_bot/trust.py answers the other half: TRUSTED_INVITERS decides who may
put this bot in a group. The gate sits in GroupDirectory._permitted, which
both _join_pending and _adopt already route through -- gating only invites
would gate the door nobody uses, since prime_profile.sh exists precisely so
that an add becomes a direct join. Evidence is either the group-update
envelope, which is the only frame naming an inviter, or a trusted identity
being a full member in the listing. A refusal is logged once, with that
group's members, so the operator can see which string to add.
verify_allowlist moved to membership.py to keep discovery.py under the
250-line gate; it was already the module that owns MembershipError.
Verified live, by stopping the container by hand and touching nothing else:
18:54:35 docker stop signal-api
18:56:36 liveness: failure 1 of 3 after 123s of silence
18:57:36 failure 3 of 3 -> /healthz answers 503 from 18:57:45
18:58:18 watchdog: probe failed (1/2)
19:00:28 watchdog: probe failed (2/2) -> docker restart signal-api
19:00:36 liveness: receive path recovered after 8 failed probe(s)
19:01:13 watchdog: signal-api is back
Six minutes, unattended, against three hours of silence this afternoon. All
four real groups are still adopted with TRUSTED_INVITERS=+48123456789 set,
and no group was refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbjfWdAyHbCubac7wJKJZW
2.4 KiB
Going deaf, and who may add the bot
Two gates that exist because of one afternoon. Both are referenced from the README; this is the detail.
When it cannot hear
On 2026-08-29 the bot was up, healthy and deaf for three hours. A host
firewall change cut every Docker container off the internet, signal-cli
lost its connection to Signal and reconnected in a loop, and every health
signal stayed green: the container's own healthcheck answered 204,
/v1/groups answered 200 from local state, the bot's WebSocket to the
container stayed open, and /healthz answered {"status": "ok"}. The only
symptom was silence, which is also what a quiet evening looks like.
Two things now make that impossible to miss:
signal_bot/liveness.py— afterSILENCE_SECONDS(120 by default) with no frame of any kind, the bot probes/v1/devices/<number>, the one endpoint that needs a live round trip to Signal. Three consecutive failures and/healthzanswers 503 with the reason, and every failure is aWARNINGin the journal. Worst case from the last envelope to a red/healthzis 120 + 3×30 ≈ 3.5 minutes.scripts/signal_api_watchdog.sh, on a two-minute user timer — probes the same endpoint from outside the process, restarts the container after two consecutive failures, and stops after 3 restarts in 6 hours, escalating withnotify-sendand a journaldERRORinstead. A deregistered account or a blocked network is not fixable by restarting, and a forever-restart loop would hide it.scripts/install_service.shinstalls and arms it beside the bot's own unit.
systemctl --user list-timers signal-api-watchdog.timer
journalctl --user -u signal-api-watchdog.service -n 20
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8137/healthz
Who may add the bot to a group
TRUSTED_INVITERS in .env is a comma-separated list of +E.164 numbers,
or the member UUIDs /v1/groups uses for anybody the account has never
messaged. Empty means anyone who knows the number can pull the bot into a
group and be answered there, and the bot says so in a WARNING at startup.
With it set, adding the bot is still the whole setup for you: a group is adopted when a group-update envelope says a trusted identity added the bot, or when a trusted identity is a full member of it in the listing. Anything else is refused once, loudly, with that group's member list in the log so you can see which string to add.