mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-12 08:51:04 +01:00
Krzysztof kuhy Rudnicki
d659ff5e9c
Today the bot was up, healthy and answering nothing for three hours. A host
firewall change (fixed in testsAndMisc 334c0cd9) cut every Docker container
off the internet; signal-cli lost Signal and reconnected in a loop from
15:32:50 onward. Nothing noticed:
- the container's own healthcheck answered 204
- /v1/groups answered 200 out of local state, so discovery saw no change
- the bot's WebSocket to the container stayed OPEN, so the transport's
reconnect logic never fired
- /healthz answered {"status": "ok"} the entire time
The only symptom was silence, which is also what a quiet evening looks like.
signal_bot/liveness.py asks the one question that cannot be answered from a
cache -- GET /v1/devices, which needs a live round trip -- but only after
SILENCE_SECONDS (120) with no frame of any kind, so a busy group costs
nothing and an idle one costs one request per window. Three consecutive
failures and /healthz answers 503 with the reason. Verified end to end in
scripts/e2e.sh against the real process: 503 after 84s with the fake's
connection cut, back to 200 on its own 30s after restoring it.
scripts/signal_api_watchdog.sh + a two-minute user timer do the recovery the
bot cannot do for itself, since a wedged bot cannot restart the container it
depends on. Two tiers, so the steady state costs no Signal round trip at
all: the bot's own /healthz first, the authenticated probe only when that is
not 200. It stops after 3 restarts in 6 hours and escalates instead -- a
deregistered account or a blocked network is not fixable by restarting, and
a restart loop would hide it. Its own bash suite stubs curl and docker and
drives every branch including the give-up cap.
signal_bot/trust.py answers the other half: TRUSTED_INVITERS decides who may
put this bot in a group. The gate sits in GroupDirectory._permitted, which
both _join_pending and _adopt already route through -- gating only invites
would gate the door nobody uses, since prime_profile.sh exists precisely so
that an add becomes a direct join. Evidence is either the group-update
envelope, which is the only frame naming an inviter, or a trusted identity
being a full member in the listing. A refusal is logged once, with that
group's members, so the operator can see which string to add.
verify_allowlist moved to membership.py to keep discovery.py under the
250-line gate; it was already the module that owns MembershipError.
Verified live, by stopping the container by hand and touching nothing else:
18:54:35 docker stop signal-api
18:56:36 liveness: failure 1 of 3 after 123s of silence
18:57:36 failure 3 of 3 -> /healthz answers 503 from 18:57:45
18:58:18 watchdog: probe failed (1/2)
19:00:28 watchdog: probe failed (2/2) -> docker restart signal-api
19:00:36 liveness: receive path recovered after 8 failed probe(s)
19:01:13 watchdog: signal-api is back
Six minutes, unattended, against three hours of silence this afternoon. All
four real groups are still adopted with TRUSTED_INVITERS=+48516501330 set,
and no group was refused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbjfWdAyHbCubac7wJKJZW
53 lines
1.8 KiB
Python
53 lines
1.8 KiB
Python
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
|
"""/healthz must go red when the bot cannot hear, not just when it is down."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import replace
|
|
from types import SimpleNamespace
|
|
from typing import TYPE_CHECKING
|
|
|
|
import httpx
|
|
from fastapi import Response
|
|
|
|
from signal_bot import app as app_module
|
|
from signal_bot.app import healthz
|
|
from signal_bot.liveness import Liveness
|
|
|
|
if TYPE_CHECKING:
|
|
from signal_bot.config import Config
|
|
|
|
DEAD_BODY = {"error": "Failed to get linked devices: Closed unexpectedly"}
|
|
|
|
|
|
def _gate(config: Config, response: httpx.Response) -> Liveness:
|
|
"""A liveness gate whose probe always gets this answer."""
|
|
client = httpx.AsyncClient(transport=httpx.MockTransport(lambda request: response))
|
|
return Liveness(config, client)
|
|
|
|
|
|
def _install(liveness: Liveness) -> None:
|
|
"""Point the app at a runtime carrying this gate."""
|
|
app_module.app.state.runtime = SimpleNamespace(bot=SimpleNamespace(liveness=liveness))
|
|
|
|
|
|
async def test_a_healthy_bot_answers_ok(config: Config) -> None:
|
|
_install(_gate(config, httpx.Response(200, json=[])))
|
|
response = Response()
|
|
assert await healthz(response) == {"status": "ok"}
|
|
assert response.status_code == httpx.codes.OK
|
|
|
|
|
|
async def test_a_deaf_bot_answers_503_and_says_why(config: Config) -> None:
|
|
"""The failure this exists for: up, listening, and hearing nothing."""
|
|
live = _gate(replace(config, silence_seconds=0), httpx.Response(400, json=DEAD_BODY))
|
|
for _ in range(3):
|
|
await live.check()
|
|
_install(live)
|
|
|
|
response = Response()
|
|
body = await healthz(response)
|
|
assert response.status_code == httpx.codes.SERVICE_UNAVAILABLE
|
|
assert body["status"] == "deaf"
|
|
assert "Closed unexpectedly" in body["reason"]
|