signal-bot/signal_bot/app.py
Krzysztof kuhy Rudnicki d5c63b423b feat: refuse to look healthy while deaf, and heal the container by itself
Today the bot was up, healthy and answering nothing for three hours. A host
firewall change (fixed in testsAndMisc 334c0cd9) cut every Docker container
off the internet; signal-cli lost Signal and reconnected in a loop from
15:32:50 onward. Nothing noticed:

  - the container's own healthcheck answered 204
  - /v1/groups answered 200 out of local state, so discovery saw no change
  - the bot's WebSocket to the container stayed OPEN, so the transport's
    reconnect logic never fired
  - /healthz answered {"status": "ok"} the entire time

The only symptom was silence, which is also what a quiet evening looks like.

signal_bot/liveness.py asks the one question that cannot be answered from a
cache -- GET /v1/devices, which needs a live round trip -- but only after
SILENCE_SECONDS (120) with no frame of any kind, so a busy group costs
nothing and an idle one costs one request per window. Three consecutive
failures and /healthz answers 503 with the reason. Verified end to end in
scripts/e2e.sh against the real process: 503 after 84s with the fake's
connection cut, back to 200 on its own 30s after restoring it.

scripts/signal_api_watchdog.sh + a two-minute user timer do the recovery the
bot cannot do for itself, since a wedged bot cannot restart the container it
depends on. Two tiers, so the steady state costs no Signal round trip at
all: the bot's own /healthz first, the authenticated probe only when that is
not 200. It stops after 3 restarts in 6 hours and escalates instead -- a
deregistered account or a blocked network is not fixable by restarting, and
a restart loop would hide it. Its own bash suite stubs curl and docker and
drives every branch including the give-up cap.

signal_bot/trust.py answers the other half: TRUSTED_INVITERS decides who may
put this bot in a group. The gate sits in GroupDirectory._permitted, which
both _join_pending and _adopt already route through -- gating only invites
would gate the door nobody uses, since prime_profile.sh exists precisely so
that an add becomes a direct join. Evidence is either the group-update
envelope, which is the only frame naming an inviter, or a trusted identity
being a full member in the listing. A refusal is logged once, with that
group's members, so the operator can see which string to add.

verify_allowlist moved to membership.py to keep discovery.py under the
250-line gate; it was already the module that owns MembershipError.

Verified live, by stopping the container by hand and touching nothing else:

  18:54:35  docker stop signal-api
  18:56:36  liveness: failure 1 of 3 after 123s of silence
  18:57:36  failure 3 of 3 -> /healthz answers 503 from 18:57:45
  18:58:18  watchdog: probe failed (1/2)
  19:00:28  watchdog: probe failed (2/2) -> docker restart signal-api
  19:00:36  liveness: receive path recovered after 8 failed probe(s)
  19:01:13  watchdog: signal-api is back

Six minutes, unattended, against three hours of silence this afternoon. All
four real groups are still adopted with TRUSTED_INVITERS=+48123456789 set,
and no group was refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KbjfWdAyHbCubac7wJKJZW
2026-08-29 19:01:55 +02:00

225 lines
7.6 KiB
Python

# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Application wiring and the ASGI entrypoint."""
from __future__ import annotations
import asyncio
import contextlib
import datetime as dt
import logging
from dataclasses import dataclass, field
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import AsyncIterator
from signal_bot.transport_base import Transport
import httpx
from fastapi import FastAPI, Response, status
from signal_bot.animals import register_animal_commands
from signal_bot.config import Config, load_config
from signal_bot.counting import format_tally
from signal_bot.decorators import cooldown
from signal_bot.discovery import GroupDirectory
from signal_bot.dispatcher import Dispatcher
from signal_bot.http_client import ImageFetcher
from signal_bot.liveness import Liveness
from signal_bot.registry import CommandRegistry
from signal_bot.scheduler import DailyScheduler
from signal_bot.store import Store
from signal_bot.transport_rest import RestTransport
from signal_bot.trust import InviteTrust
logger = logging.getLogger(__name__)
COOLDOWN_MESSAGE = "BEEP BOOP POCZEKAJ {seconds} SEKUND."
HTTP_TIMEOUT_SECONDS = 30.0
def now_local() -> dt.datetime:
"""Current local time, timezone-aware."""
return dt.datetime.now().astimezone()
@dataclass
class Bot:
"""The wired-up bot and its two long-running loops."""
config: Config
store: Store
transport: Transport
directory: GroupDirectory
dispatcher: Dispatcher
liveness: Liveness
async def publish_report(self) -> None:
"""Post each group's own tally into that group, and reset it.
Each group is isolated on purpose. One group refusing the report --
the bot removed from it, a transient send failure -- must not cost
every other group its report, and a tally is cleared only after its
report has actually gone out, so a failed send loses nothing. A
group where nobody spoke is skipped entirely.
"""
for group_id in self.directory.groups:
rows = [(t.display_name or t.uuid, t.count) for t in self.store.tallies(group_id)]
if not rows:
# Logged, not posted. The bot is in every group it was ever
# added to, so posting "nobody spoke" would mean a nightly
# ping into every quiet group, forever.
logger.info("no messages in %s today; not posting a report", group_id)
continue
try:
await self.transport.send_text(format_tally(rows), group_id)
except Exception:
logger.exception("daily report failed for group %s", group_id)
continue
self.store.reset_tallies(group_id)
async def listen(self) -> None:
"""Consume frames forever, handing each to the dispatcher."""
async for frame in self.transport.receive():
# Every frame counts, including receipts and typing indicators:
# the question this answers is "can anything still reach us?",
# not "did anybody address the bot?".
self.liveness.note_envelope()
await self.dispatcher.handle(frame)
def build_bot(
config: Config,
client: httpx.AsyncClient,
store: Store,
transport: Transport,
directory: GroupDirectory,
liveness: Liveness,
) -> Bot:
"""Assemble the object graph from its injected dependencies."""
registry = CommandRegistry()
register_animal_commands(
registry,
ImageFetcher(client),
decorate=cooldown(config.cooldown_seconds, COOLDOWN_MESSAGE),
endpoints={"cat": config.cat_api_url, "dog": config.dog_api_url},
)
dispatcher = Dispatcher(
config=config,
directory=directory,
registry=registry,
store=store,
transport=transport,
clock=now_local,
)
return Bot(
config=config,
store=store,
transport=transport,
directory=directory,
dispatcher=dispatcher,
liveness=liveness,
)
@dataclass
class Runtime:
"""Owns everything that must be torn down when the app stops."""
bot: Bot
client: httpx.AsyncClient
store: Store
tasks: list[asyncio.Task[None]] = field(default_factory=list)
def start(self) -> None:
"""Spawn the listener and the daily scheduler as background tasks."""
scheduler = DailyScheduler(
self.bot.config.report_hour,
self.bot.config.report_minute,
self.bot.publish_report,
now_local,
asyncio.sleep,
)
self.tasks = [
asyncio.create_task(self.bot.listen()),
asyncio.create_task(scheduler.run()),
# Third loop: without it, a group added later is only noticed
# when somebody happens to speak in it.
asyncio.create_task(self.bot.directory.run()),
# Fourth loop: without it, a dead connection to Signal is
# indistinguishable from a quiet evening, for hours.
asyncio.create_task(self.bot.liveness.run()),
]
async def stop(self) -> None:
"""Cancel the background tasks and close every resource."""
for task in self.tasks:
task.cancel()
for task in self.tasks:
with contextlib.suppress(asyncio.CancelledError):
await task
await self.client.aclose()
self.store.close()
async def create_runtime(config: Config, client: httpx.AsyncClient) -> Runtime:
"""Build a Runtime, verifying the container exposes a WebSocket first."""
store = Store(config.database_path)
transport = RestTransport(config, client)
await transport.verify_mode()
directory = GroupDirectory(config, client, store, InviteTrust(config.trusted_inviters))
await directory.refresh()
await directory.verify_allowlist()
# No probe here: /v1/devices has been seen to hang past 20s, and a
# startup that can hang is a startup that can burn the unit's restart
# limit. The gate starts healthy and earns its verdict at runtime.
liveness = Liveness(config, client)
return Runtime(
bot=build_bot(config, client, store, transport, directory, liveness),
client=client,
store=store,
)
@contextlib.asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
"""Start the background tasks without blocking ASGI startup.
The previous version awaited asyncio.gather() on two infinite tasks inside
an on_event("startup") handler, so uvicorn never finished starting up and
the HTTP app never began serving.
"""
config = load_config()
client = httpx.AsyncClient(timeout=HTTP_TIMEOUT_SECONDS)
runtime = await create_runtime(config, client)
runtime.start()
logger.info(
"listening as %s, watching %d group(s): %r",
config.phone_number,
len(runtime.bot.directory.groups),
list(runtime.bot.directory.groups),
)
app.state.runtime = runtime
try:
yield
finally:
await runtime.stop()
app = FastAPI(lifespan=lifespan)
@app.get("/healthz")
async def healthz(response: Response) -> dict[str, str]:
"""Liveness probe, and the only place the bot admits it has gone deaf.
Answering 200 while nothing can reach Signal is what let a three-hour
outage look like a quiet evening, so this must go red before anybody
thinks to look at the logs.
"""
liveness: Liveness = app.state.runtime.bot.liveness
if liveness.healthy:
return {"status": "ok"}
response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
return {"status": "deaf", "reason": liveness.reason}