signal-bot/DOCS-liveness-and-trust.md
Krzysztof kuhy Rudnicki d659ff5e9c feat: refuse to look healthy while deaf, and heal the container by itself
Today the bot was up, healthy and answering nothing for three hours. A host
firewall change (fixed in testsAndMisc 334c0cd9) cut every Docker container
off the internet; signal-cli lost Signal and reconnected in a loop from
15:32:50 onward. Nothing noticed:

  - the container's own healthcheck answered 204
  - /v1/groups answered 200 out of local state, so discovery saw no change
  - the bot's WebSocket to the container stayed OPEN, so the transport's
    reconnect logic never fired
  - /healthz answered {"status": "ok"} the entire time

The only symptom was silence, which is also what a quiet evening looks like.

signal_bot/liveness.py asks the one question that cannot be answered from a
cache -- GET /v1/devices, which needs a live round trip -- but only after
SILENCE_SECONDS (120) with no frame of any kind, so a busy group costs
nothing and an idle one costs one request per window. Three consecutive
failures and /healthz answers 503 with the reason. Verified end to end in
scripts/e2e.sh against the real process: 503 after 84s with the fake's
connection cut, back to 200 on its own 30s after restoring it.

scripts/signal_api_watchdog.sh + a two-minute user timer do the recovery the
bot cannot do for itself, since a wedged bot cannot restart the container it
depends on. Two tiers, so the steady state costs no Signal round trip at
all: the bot's own /healthz first, the authenticated probe only when that is
not 200. It stops after 3 restarts in 6 hours and escalates instead -- a
deregistered account or a blocked network is not fixable by restarting, and
a restart loop would hide it. Its own bash suite stubs curl and docker and
drives every branch including the give-up cap.

signal_bot/trust.py answers the other half: TRUSTED_INVITERS decides who may
put this bot in a group. The gate sits in GroupDirectory._permitted, which
both _join_pending and _adopt already route through -- gating only invites
would gate the door nobody uses, since prime_profile.sh exists precisely so
that an add becomes a direct join. Evidence is either the group-update
envelope, which is the only frame naming an inviter, or a trusted identity
being a full member in the listing. A refusal is logged once, with that
group's members, so the operator can see which string to add.

verify_allowlist moved to membership.py to keep discovery.py under the
250-line gate; it was already the module that owns MembershipError.

Verified live, by stopping the container by hand and touching nothing else:

  18:54:35  docker stop signal-api
  18:56:36  liveness: failure 1 of 3 after 123s of silence
  18:57:36  failure 3 of 3 -> /healthz answers 503 from 18:57:45
  18:58:18  watchdog: probe failed (1/2)
  19:00:28  watchdog: probe failed (2/2) -> docker restart signal-api
  19:00:36  liveness: receive path recovered after 8 failed probe(s)
  19:01:13  watchdog: signal-api is back

Six minutes, unattended, against three hours of silence this afternoon. All
four real groups are still adopted with TRUSTED_INVITERS=+48516501330 set,
and no group was refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbjfWdAyHbCubac7wJKJZW
2026-08-29 19:01:55 +02:00

50 lines
2.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Going deaf, and who may add the bot
Two gates that exist because of one afternoon. Both are referenced from
the README; this is the detail.
## When it cannot hear
On 2026-08-29 the bot was up, healthy and deaf for three hours. A host
firewall change cut every Docker container off the internet, `signal-cli`
lost its connection to Signal and reconnected in a loop, and **every** health
signal stayed green: the container's own healthcheck answered 204,
`/v1/groups` answered 200 from local state, the bot's WebSocket to the
container stayed open, and `/healthz` answered `{"status": "ok"}`. The only
symptom was silence, which is also what a quiet evening looks like.
Two things now make that impossible to miss:
- **`signal_bot/liveness.py`** — after `SILENCE_SECONDS` (120 by default)
with no frame of any kind, the bot probes `/v1/devices/<number>`, the one
endpoint that needs a live round trip to Signal. Three consecutive
failures and `/healthz` answers **503** with the reason, and every failure
is a `WARNING` in the journal. Worst case from the last envelope to a red
`/healthz` is 120 + 3×30 ≈ 3.5 minutes.
- **`scripts/signal_api_watchdog.sh`**, on a two-minute user timer — probes
the same endpoint from outside the process, restarts the container after
two consecutive failures, and **stops after 3 restarts in 6 hours**,
escalating with `notify-send` and a journald `ERROR` instead. A
deregistered account or a blocked network is not fixable by restarting,
and a forever-restart loop would hide it. `scripts/install_service.sh`
installs and arms it beside the bot's own unit.
```bash
systemctl --user list-timers signal-api-watchdog.timer
journalctl --user -u signal-api-watchdog.service -n 20
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8137/healthz
```
## Who may add the bot to a group
`TRUSTED_INVITERS` in `.env` is a comma-separated list of `+E.164` numbers,
or the member UUIDs `/v1/groups` uses for anybody the account has never
messaged. Empty means anyone who knows the number can pull the bot into a
group and be answered there, and the bot says so in a `WARNING` at startup.
With it set, adding the bot is still the whole setup for you: a group is
adopted when a group-update envelope says a trusted identity added the bot,
or when a trusted identity is a full member of it in the listing. Anything
else is refused once, loudly, with that group's member list in the log so
you can see which string to add.