signal-bot/DOCS-liveness-and-trust.md

# Going deaf, not being up yet, and who may add the bot

Three gates, each from a failure that looked like nothing at the time. All
are referenced from the README; this is the detail.

## When it cannot hear

On 2026-08-29 the bot was up, healthy and deaf for three hours. A host
firewall change cut every Docker container off the internet, `signal-cli`
lost its connection to Signal and reconnected in a loop, and **every** health
signal stayed green: the container's own healthcheck answered 204,
`/v1/groups` answered 200 from local state, the bot's WebSocket to the
container stayed open, and `/healthz` answered `{"status": "ok"}`. The only
symptom was silence, which is also what a quiet evening looks like.

Two things now make that impossible to miss:

- **`signal_bot/liveness.py`** — after `SILENCE_SECONDS` (120 by default)
  with no frame of any kind, the bot probes `/v1/devices/<number>`, the one
  endpoint that needs a live round trip to Signal. Three consecutive
  failures and `/healthz` answers **503** with the reason, and every failure
  is a `WARNING` in the journal. Worst case from the last envelope to a red
  `/healthz` is 120 + 3×30 ≈ 3.5 minutes.
- **`scripts/signal_api_watchdog.sh`**, on a two-minute user timer — probes
  the same endpoint from outside the process, restarts the container after
  two consecutive failures, and **stops after 3 restarts in 6 hours**,
  escalating with `notify-send` and a journald `ERROR` instead. A
  deregistered account or a blocked network is not fixable by restarting,
  and a forever-restart loop would hide it. `scripts/install_service.sh`
  installs and arms it beside the bot's own unit.

## What reaches the desktop

Every watchdog message goes to journald twice — once via
`logger --tag signal-api-watchdog`, once as the unit's own stdout — and the
desktop popup is a *summary* of that, never the record. `WATCHDOG_NOTIFY`
decides how much of it pops: `escalation` (the default, give-up only),
`always`, or `never`.

It is a summary because it was once a flood. Three things made it one:

- The popups fired per **check**, not per state change, and `give_up()` had no
  once-only flag — so a six-hour give-up window announced itself every two
  minutes, up to 180 times, while its own text said "no further restarts".
- `alarm()` hardcoded `--urgency=critical`, which `xfce4-notifyd` never
  expires, and passed no replace id, so they stacked instead of superseding.
  `notify-send --print-id` now feeds `--replace-id` from
  `${STATE_DIR}/notify_id`: one slot, reused.
- Worst of all, none of it needed an outage. `scripts/test_watchdog.sh` stubbed
  only `curl` and `docker`, so `notify-send` was the real binary and the suite
  fired a dozen genuine critical popups per run — from a pre-commit hook. On
  2026-08-31 the timer emitted **zero** messages and the desktop still
  collected 297, every one of them self-inflicted by committing.

The stubs now live in the harness, and the suite asserts the counts rather than
the absence: a healthy check pops nothing, a routine restart-and-recovery pops
nothing, and a sustained give-up window pops exactly once.

```bash
systemctl --user list-timers signal-api-watchdog.timer
journalctl --user -u signal-api-watchdog.service -n 20
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8137/healthz
```

## When it is not up yet

On 2026-08-30 the machine rebooted. The bot started before the `signal-api`
container was ready, its first HTTP call raised
`RemoteProtocolError: Server disconnected without sending a response`,
uvicorn reported `Application startup failed. Exiting.`, and `RestartSec=5`
with `StartLimitBurst=5` gave the container **25 seconds** to appear. It took
minutes. The unit landed in `failed` and stayed there for eight hours, and
the group had no bot for all of them.

`After=docker.service` cannot fix this and never could: dockerd being up says
nothing about the container inside it, and signal-cli then spends further
minutes loading the account. So the connection is no longer part of starting
up. `signal_bot/startup.py` serves first and connects afterwards:

- uvicorn begins serving immediately, and `/healthz` answers **503** with
  `{"status": "connecting"}` and the actual error until the bot is connected.
  A booting bot is visibly unhealthy rather than absent.
- The retry backs off 5s → 60s and never gives up. Recovery needs nobody.
- Errors no retry can fix are still fatal, and still land the unit in
  `failed` where `systemctl --user --failed` shows them: a container in the
  wrong `MODE`, a `GROUP_IDS` entry naming a group the bot is genuinely not
  in, a missing `PHONE_NUMBER`, a taken port.

The distinction that makes the last point safe: a `GROUP_IDS` entry is only
judged against a group listing that was **actually read**. A failed listing
leaves the watched set empty, and convicting the allowlist on that would turn
"the container is not up yet" into a verdict nothing can retry.

`scripts/cold_start.sh` is the proof, and runs in CI: it boots the bot with
nothing at the other end, asserts the process stays up and says why, then
starts the fake Signal server underneath it and asserts the bot recovers
unaided with all four groups rediscovered.

## Who may add the bot to a group

`TRUSTED_INVITERS` in `.env` is a comma-separated list of `+E.164` numbers,
or the member UUIDs `/v1/groups` uses for anybody the account has never
messaged. Empty means anyone who knows the number can pull the bot into a
group and be answered there, and the bot says so in a `WARNING` at startup.

With it set, adding the bot is still the whole setup for you: a group is
adopted when a group-update envelope says a trusted identity added the bot,
or when a trusted identity is a full member of it in the listing. Anything
else is refused once, loudly, with that group's member list in the log so
you can see which string to add.