signal-bot/DOCS-liveness-and-trust.md

Going deaf, not being up yet, and who may add the bot

Three gates, each from a failure that looked like nothing at the time. All are referenced from the README; this is the detail.

When it cannot hear

On 2026-08-29 the bot was up, healthy and deaf for three hours. A host firewall change cut every Docker container off the internet, signal-cli lost its connection to Signal and reconnected in a loop, and every health signal stayed green: the container's own healthcheck answered 204, /v1/groups answered 200 from local state, the bot's WebSocket to the container stayed open, and /healthz answered {"status": "ok"}. The only symptom was silence, which is also what a quiet evening looks like.

Two things now make that impossible to miss:

• signal_bot/liveness.py — after SILENCE_SECONDS (120 by default) with no frame of any kind, the bot probes /v1/devices/<number>, the one endpoint that needs a live round trip to Signal. Three consecutive failures and /healthz answers 503 with the reason, and every failure is a WARNING in the journal. Worst case from the last envelope to a red /healthz is 120 + 3×30 ≈ 3.5 minutes.
• scripts/signal_api_watchdog.sh, on a two-minute user timer — probes the same endpoint from outside the process, restarts the container after two consecutive failures, and stops after 3 restarts in 6 hours, escalating with notify-send and a journald ERROR instead. A deregistered account or a blocked network is not fixable by restarting, and a forever-restart loop would hide it. scripts/install_service.sh installs and arms it beside the bot's own unit.

What reaches the desktop

Every watchdog message goes to journald twice — once via logger --tag signal-api-watchdog, once as the unit's own stdout — and the desktop popup is a summary of that, never the record. WATCHDOG_NOTIFY decides how much of it pops: escalation (the default, give-up only), always, or never.

It is a summary because it was once a flood. Three things made it one:

• The popups fired per check, not per state change, and give_up() had no once-only flag — so a six-hour give-up window announced itself every two minutes, up to 180 times, while its own text said "no further restarts".
• alarm() hardcoded --urgency=critical, which xfce4-notifyd never expires, and passed no replace id, so they stacked instead of superseding. notify-send --print-id now feeds --replace-id from ${STATE_DIR}/notify_id: one slot, reused.
• Worst of all, none of it needed an outage. scripts/test_watchdog.sh stubbed only curl and docker, so notify-send was the real binary and the suite fired a dozen genuine critical popups per run — from a pre-commit hook. On 2026-08-31 the timer emitted zero messages and the desktop still collected 297, every one of them self-inflicted by committing.

The stubs now live in the harness, and the suite asserts the counts rather than the absence: a healthy check pops nothing, a routine restart-and-recovery pops nothing, and a sustained give-up window pops exactly once.

systemctl --user list-timers signal-api-watchdog.timer
journalctl --user -u signal-api-watchdog.service -n 20
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8137/healthz

When it is not up yet

On 2026-08-30 the machine rebooted. The bot started before the signal-api container was ready, its first HTTP call raised RemoteProtocolError: Server disconnected without sending a response, uvicorn reported Application startup failed. Exiting., and RestartSec=5 with StartLimitBurst=5 gave the container 25 seconds to appear. It took minutes. The unit landed in failed and stayed there for eight hours, and the group had no bot for all of them.

After=docker.service cannot fix this and never could: dockerd being up says nothing about the container inside it, and signal-cli then spends further minutes loading the account. So the connection is no longer part of starting up. signal_bot/startup.py serves first and connects afterwards:

• uvicorn begins serving immediately, and /healthz answers 503 with {"status": "connecting"} and the actual error until the bot is connected. A booting bot is visibly unhealthy rather than absent.
• The retry backs off 5s → 60s and never gives up. Recovery needs nobody.
• Errors no retry can fix are still fatal, and still land the unit in failed where systemctl --user --failed shows them: a container in the wrong MODE, a GROUP_IDS entry naming a group the bot is genuinely not in, a missing PHONE_NUMBER, a taken port.

The distinction that makes the last point safe: a GROUP_IDS entry is only judged against a group listing that was actually read. A failed listing leaves the watched set empty, and convicting the allowlist on that would turn "the container is not up yet" into a verdict nothing can retry.

scripts/cold_start.sh is the proof, and runs in CI: it boots the bot with nothing at the other end, asserts the process stays up and says why, then starts the fake Signal server underneath it and asserts the bot recovers unaided with all four groups rediscovered.

Who may add the bot to a group

TRUSTED_INVITERS in .env is a comma-separated list of +E.164 numbers, or the member UUIDs /v1/groups uses for anybody the account has never messaged. Empty means anyone who knows the number can pull the bot into a group and be answered there, and the bot says so in a WARNING at startup.

With it set, adding the bot is still the whole setup for you: a group is adopted when a group-update envelope says a trusted identity added the bot, or when a trusted identity is a full member of it in the listing. Anything else is refused once, loudly, with that group's member list in the log so you can see which string to add.