mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-21 09:32:33 +01:00
Today the bot was up, healthy and answering nothing for three hours. A host
firewall change (fixed in testsAndMisc 334c0cd9) cut every Docker container
off the internet; signal-cli lost Signal and reconnected in a loop from
15:32:50 onward. Nothing noticed:
- the container's own healthcheck answered 204
- /v1/groups answered 200 out of local state, so discovery saw no change
- the bot's WebSocket to the container stayed OPEN, so the transport's
reconnect logic never fired
- /healthz answered {"status": "ok"} the entire time
The only symptom was silence, which is also what a quiet evening looks like.
signal_bot/liveness.py asks the one question that cannot be answered from a
cache -- GET /v1/devices, which needs a live round trip -- but only after
SILENCE_SECONDS (120) with no frame of any kind, so a busy group costs
nothing and an idle one costs one request per window. Three consecutive
failures and /healthz answers 503 with the reason. Verified end to end in
scripts/e2e.sh against the real process: 503 after 84s with the fake's
connection cut, back to 200 on its own 30s after restoring it.
scripts/signal_api_watchdog.sh + a two-minute user timer do the recovery the
bot cannot do for itself, since a wedged bot cannot restart the container it
depends on. Two tiers, so the steady state costs no Signal round trip at
all: the bot's own /healthz first, the authenticated probe only when that is
not 200. It stops after 3 restarts in 6 hours and escalates instead -- a
deregistered account or a blocked network is not fixable by restarting, and
a restart loop would hide it. Its own bash suite stubs curl and docker and
drives every branch including the give-up cap.
signal_bot/trust.py answers the other half: TRUSTED_INVITERS decides who may
put this bot in a group. The gate sits in GroupDirectory._permitted, which
both _join_pending and _adopt already route through -- gating only invites
would gate the door nobody uses, since prime_profile.sh exists precisely so
that an add becomes a direct join. Evidence is either the group-update
envelope, which is the only frame naming an inviter, or a trusted identity
being a full member in the listing. A refusal is logged once, with that
group's members, so the operator can see which string to add.
verify_allowlist moved to membership.py to keep discovery.py under the
250-line gate; it was already the module that owns MembershipError.
Verified live, by stopping the container by hand and touching nothing else:
18:54:35 docker stop signal-api
18:56:36 liveness: failure 1 of 3 after 123s of silence
18:57:36 failure 3 of 3 -> /healthz answers 503 from 18:57:45
18:58:18 watchdog: probe failed (1/2)
19:00:28 watchdog: probe failed (2/2) -> docker restart signal-api
19:00:36 liveness: receive path recovered after 8 failed probe(s)
19:01:13 watchdog: signal-api is back
Six minutes, unattended, against three hours of silence this afternoon. All
four real groups are still adopted with TRUSTED_INVITERS=+48516501330 set,
and no group was refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbjfWdAyHbCubac7wJKJZW
50 lines
2.4 KiB
Markdown
50 lines
2.4 KiB
Markdown
# Going deaf, and who may add the bot
|
||
|
||
Two gates that exist because of one afternoon. Both are referenced from
|
||
the README; this is the detail.
|
||
|
||
## When it cannot hear
|
||
|
||
On 2026-08-29 the bot was up, healthy and deaf for three hours. A host
|
||
firewall change cut every Docker container off the internet, `signal-cli`
|
||
lost its connection to Signal and reconnected in a loop, and **every** health
|
||
signal stayed green: the container's own healthcheck answered 204,
|
||
`/v1/groups` answered 200 from local state, the bot's WebSocket to the
|
||
container stayed open, and `/healthz` answered `{"status": "ok"}`. The only
|
||
symptom was silence, which is also what a quiet evening looks like.
|
||
|
||
Two things now make that impossible to miss:
|
||
|
||
- **`signal_bot/liveness.py`** — after `SILENCE_SECONDS` (120 by default)
|
||
with no frame of any kind, the bot probes `/v1/devices/<number>`, the one
|
||
endpoint that needs a live round trip to Signal. Three consecutive
|
||
failures and `/healthz` answers **503** with the reason, and every failure
|
||
is a `WARNING` in the journal. Worst case from the last envelope to a red
|
||
`/healthz` is 120 + 3×30 ≈ 3.5 minutes.
|
||
- **`scripts/signal_api_watchdog.sh`**, on a two-minute user timer — probes
|
||
the same endpoint from outside the process, restarts the container after
|
||
two consecutive failures, and **stops after 3 restarts in 6 hours**,
|
||
escalating with `notify-send` and a journald `ERROR` instead. A
|
||
deregistered account or a blocked network is not fixable by restarting,
|
||
and a forever-restart loop would hide it. `scripts/install_service.sh`
|
||
installs and arms it beside the bot's own unit.
|
||
|
||
```bash
|
||
systemctl --user list-timers signal-api-watchdog.timer
|
||
journalctl --user -u signal-api-watchdog.service -n 20
|
||
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8137/healthz
|
||
```
|
||
|
||
## Who may add the bot to a group
|
||
|
||
`TRUSTED_INVITERS` in `.env` is a comma-separated list of `+E.164` numbers,
|
||
or the member UUIDs `/v1/groups` uses for anybody the account has never
|
||
messaged. Empty means anyone who knows the number can pull the bot into a
|
||
group and be answered there, and the bot says so in a `WARNING` at startup.
|
||
|
||
With it set, adding the bot is still the whole setup for you: a group is
|
||
adopted when a group-update envelope says a trusted identity added the bot,
|
||
or when a trusted identity is a full member of it in the listing. Anything
|
||
else is refused once, loudly, with that group's member list in the log so
|
||
you can see which string to add.
|