signal-bot/signal_bot/inflight.py
Krzysztof kuhy Rudnicki 4af32f8bd5 Handle frames side by side, and budget the bot's own share of a reply
Bot.listen awaited every frame to completion, so one Commons fetch or one
attachment upload held up every text reply behind it (DOCS-latency.md).
Each frame now runs in its own task, never more than LIMIT at once; the
reader waits for a slot before spawning, so a burst applies back-pressure
at the websocket instead of fanning out upstream. One person's commands
stay serialised (user_lock.py): two quick !kot would otherwise both pass
the cooldown gate before either recorded a use.

Every served command reports its timings split into the bot's own time
and everybody else's (served.py); /_selftest carries that block and
tool/latency.py holds own_ms to 100 ms at p95, with the command's
cooldown switched off for the run and put back afterwards.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-09-04 21:35:34 +02:00

95 lines
3.7 KiB
Python

# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Frames in flight: each one its own task, never more than a few at once.
``Bot.listen`` used to await every frame to completion before reading the
next, so one slow upstream fetch -- a picture from Commons, a 2.6 s
attachment upload -- delayed every message behind it (``DOCS-latency.md``).
Now a frame is handed to a task of its own and the reader carries on.
The count is bounded on purpose. Unbounded tasks would let a burst of frames
open a hundred upstream connections at once, and the reader would sail on
while memory filled with work nothing had got to. Waiting for a slot
*before* spawning applies the pressure where it belongs: the websocket
reader pauses, the socket buffers, and signal-cli holds the rest.
"""
from __future__ import annotations
import asyncio
import contextlib
import logging
from typing import TYPE_CHECKING, Final
if TYPE_CHECKING:
from collections.abc import AsyncIterator, Coroutine
logger = logging.getLogger(__name__)
# Handlers running at once. Enough that a slow picture never queues a text
# reply behind it; few enough that a burst cannot fan out into a flood of
# upstream requests. Not configuration: nothing about a group changes it.
LIMIT: Final = 8
class Inflight:
"""A bounded set of running frame tasks."""
def __init__(self, limit: int = LIMIT) -> None:
"""Allow at most ``limit`` frames to be handled at the same time."""
self._slots = asyncio.Semaphore(limit)
self._tasks: set[asyncio.Task[object]] = set()
# Set whenever the last task has been *accounted for*, which is a
# moment after it has finished: a done callback runs on a later loop
# turn, and gather() on an already-finished task may not yield at
# all, so "no tasks left" cannot be read off the task set alone.
self._idle = asyncio.Event()
self._idle.set()
@property
def count(self) -> int:
"""How many spawned frames have not finished yet."""
return len(self._tasks)
@contextlib.asynccontextmanager
async def slot(self) -> AsyncIterator[None]:
"""Hold one of the slots for the duration of the block."""
async with self._slots:
yield
async def start(self, work: Coroutine[object, object, object]) -> None:
"""Run ``work`` in its own task, once a slot is free.
Returns as soon as the task exists, not when it finishes -- that is
the whole point. A task that raises is logged rather than lost:
nothing awaits it, so an unhandled exception would otherwise be
reported only when the task is garbage-collected, if at all.
"""
await self._slots.acquire()
task = asyncio.create_task(work)
self._tasks.add(task)
self._idle.clear()
task.add_done_callback(self._finished)
def _finished(self, task: asyncio.Task[object]) -> None:
# The slot goes back here, not in a wrapper around the work: a task
# cancelled before its first step never runs a wrapper's `finally`,
# and a slot lost that way would be lost until restart.
self._slots.release()
self._tasks.discard(task)
if not self._tasks:
self._idle.set()
if task.cancelled():
return
error = task.exception()
if error is not None:
logger.error("a frame's task failed", exc_info=error)
def cancel(self) -> None:
"""Stop everything in flight; for shutdown, where finishing is not wanted."""
for task in self._tasks:
task.cancel()
async def drain(self) -> None:
"""Wait for everything in flight to finish, whichever way it ends."""
await self._idle.wait()