signal-bot/DOCS-liveness-and-trust.md
Krzysztof kuhy Rudnicki 72482629d9 fix: a signal-api that is not up yet no longer kills the bot at boot
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-30 15:23:42 +02:00

84 lines
4.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Going deaf, not being up yet, and who may add the bot
Three gates, each from a failure that looked like nothing at the time. All
are referenced from the README; this is the detail.
## When it cannot hear
On 2026-08-29 the bot was up, healthy and deaf for three hours. A host
firewall change cut every Docker container off the internet, `signal-cli`
lost its connection to Signal and reconnected in a loop, and **every** health
signal stayed green: the container's own healthcheck answered 204,
`/v1/groups` answered 200 from local state, the bot's WebSocket to the
container stayed open, and `/healthz` answered `{"status": "ok"}`. The only
symptom was silence, which is also what a quiet evening looks like.
Two things now make that impossible to miss:
- **`signal_bot/liveness.py`** — after `SILENCE_SECONDS` (120 by default)
with no frame of any kind, the bot probes `/v1/devices/<number>`, the one
endpoint that needs a live round trip to Signal. Three consecutive
failures and `/healthz` answers **503** with the reason, and every failure
is a `WARNING` in the journal. Worst case from the last envelope to a red
`/healthz` is 120 + 3×30 ≈ 3.5 minutes.
- **`scripts/signal_api_watchdog.sh`**, on a two-minute user timer — probes
the same endpoint from outside the process, restarts the container after
two consecutive failures, and **stops after 3 restarts in 6 hours**,
escalating with `notify-send` and a journald `ERROR` instead. A
deregistered account or a blocked network is not fixable by restarting,
and a forever-restart loop would hide it. `scripts/install_service.sh`
installs and arms it beside the bot's own unit.
```bash
systemctl --user list-timers signal-api-watchdog.timer
journalctl --user -u signal-api-watchdog.service -n 20
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8137/healthz
```
## When it is not up yet
On 2026-08-30 the machine rebooted. The bot started before the `signal-api`
container was ready, its first HTTP call raised
`RemoteProtocolError: Server disconnected without sending a response`,
uvicorn reported `Application startup failed. Exiting.`, and `RestartSec=5`
with `StartLimitBurst=5` gave the container **25 seconds** to appear. It took
minutes. The unit landed in `failed` and stayed there for eight hours, and
the group had no bot for all of them.
`After=docker.service` cannot fix this and never could: dockerd being up says
nothing about the container inside it, and signal-cli then spends further
minutes loading the account. So the connection is no longer part of starting
up. `signal_bot/startup.py` serves first and connects afterwards:
- uvicorn begins serving immediately, and `/healthz` answers **503** with
`{"status": "connecting"}` and the actual error until the bot is connected.
A booting bot is visibly unhealthy rather than absent.
- The retry backs off 5s → 60s and never gives up. Recovery needs nobody.
- Errors no retry can fix are still fatal, and still land the unit in
`failed` where `systemctl --user --failed` shows them: a container in the
wrong `MODE`, a `GROUP_IDS` entry naming a group the bot is genuinely not
in, a missing `PHONE_NUMBER`, a taken port.
The distinction that makes the last point safe: a `GROUP_IDS` entry is only
judged against a group listing that was **actually read**. A failed listing
leaves the watched set empty, and convicting the allowlist on that would turn
"the container is not up yet" into a verdict nothing can retry.
`scripts/cold_start.sh` is the proof, and runs in CI: it boots the bot with
nothing at the other end, asserts the process stays up and says why, then
starts the fake Signal server underneath it and asserts the bot recovers
unaided with all four groups rediscovered.
## Who may add the bot to a group
`TRUSTED_INVITERS` in `.env` is a comma-separated list of `+E.164` numbers,
or the member UUIDs `/v1/groups` uses for anybody the account has never
messaged. Empty means anyone who knows the number can pull the bot into a
group and be answered there, and the bot says so in a `WARNING` at startup.
With it set, adding the bot is still the whole setup for you: a group is
adopted when a group-update envelope says a trusted identity added the bot,
or when a trusted identity is a full member of it in the listing. Anything
else is refused once, loudly, with that group's member list in the log so
you can see which string to add.