signal-bot/signal_bot/stats/backfill.py
Krzysztof kuhy Rudnicki 16e066f4ba feat: survive an outage, and keep the counts that predate the event log
Two gaps kuhy pointed at, both about not losing messages the bot was not
awake for.

Missed *counts* already worked and now have a test saying so: an event
is keyed by Signal's own send timestamp, signal-cli keeps receiving in
json-rpc mode while the bot is down, and the replay on reconnect is
absorbed by the primary key. Messages sent during an outage come back
filed under the day they were actually sent.

Missed *reports* did not. A bot that was down at 21:37 never posted that
day and never went back for it. The last day reported is now written
down per group, so returning from an outage posts every day it owes,
oldest first, bounded at a week. A day nobody spoke on is marked settled
rather than reconsidered forever; a day whose send *failed* is
deliberately left unmarked so the next run retries it.

The pre-0.1.0 counter is carried over rather than discarded: 342
messages across four groups, 319 of them in the real group. What the
counter cannot say is when, so before assuming, four places were checked
for a real inbound timestamp -- journald (commands only), signal-cli's
account.db (19 tables, none inbound; message_send_log is the bot's own
outgoing resend buffer), Signal Desktop (unlinked, empty, and it does
not backfill on link), and signal-api/attachments (the one dated
artifact). The attachment mtimes put 12 files on 29 Aug against 79 on
30 Aug inside the counter's window, which is why the whole counter is
filed under 30 Aug rather than split.

That day is marked in group_meta, and a marked day claims no average
message length -- those messages have a count but no lengths, so
"średnio 0 znaków" would be a measurement nobody took. The mark is also
what will keep the day out of personal records: a merged 22-hour day
would otherwise hand somebody a record they never set.

render_day's options move into a DayView value object; the keyword list
had reached eight and this renderer is about to be shared by several
commands that each want a different subset.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015d48t4qsQLBJnDwDeo2Nom
2026-08-30 19:23:53 +02:00

89 lines
3.4 KiB
Python

# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Carrying the pre-0.1.0 counter into the event log, once.
Before 0.1.0 the bot kept one integer per person and deleted it after each
nightly post. When the event log arrived there were 319 messages sitting in
that counter, accumulated since the last successful post. Throwing them away
would have been tidier, but they are real messages and they are the group's
only pre-history.
What the counter cannot say is *when*. It has no timestamp column, and its
window straddles two calendar days -- from the previous night's post to
whenever the migration runs. So a backfilled day is filed under the day that
holds most of that window, and marked, because a merged 22-hour day pretending
to be a normal one would hand somebody a personal record they never set and
would drag every "average day" figure with it.
Marked days keep their totals and stay out of records and per-hour statistics.
``group_meta`` carries the mark, so this needs no schema change.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Final
from signal_bot.stats.events import MessageEvent
if TYPE_CHECKING:
from signal_bot.store import Store
BACKFILL_KEY: Final = "backfill_day"
# Noon: a backfilled event has no real hour, and midnight would look like a
# deliberate reading rather than an absent one. The day is marked anyway, so
# nothing that cares about hours will read these.
BACKFILL_HOUR: Final = 12
def has_legacy_counter(store: Store) -> bool:
"""True when this database predates 0.1.0 and still holds the old table."""
row = store.events.connection.execute(
"SELECT name FROM sqlite_master WHERE type = 'table' AND name = 'message_counts'"
).fetchone()
return row is not None
def backfilled_day(store: Store, group_id: str) -> str | None:
"""The one day whose totals came from the old counter, if any."""
return store.meta(group_id, BACKFILL_KEY)
def seed_from_counter(store: Store, day: str) -> dict[str, int]:
"""Copy every surviving tally into the event log as one marked day.
Args:
store: The open database, which must still hold ``message_counts``.
day: The Warsaw day to file the counts under. Choose the day covering
most of the counter's window.
Returns:
Messages seeded per group id, for reporting.
"""
rows = store.events.connection.execute(
"SELECT group_id, uuid, display_name, count FROM message_counts WHERE count > 0"
).fetchall()
seeded: dict[str, int] = {}
for row in rows:
group_id = row["group_id"]
for index in range(row["count"]):
# Synthetic, strictly increasing, and unique per person per group,
# because (group, uuid, sent_at_ms) is the primary key. They are
# not timestamps anybody should read as one, which is what the
# day marking is for.
store.events.record(
MessageEvent(
group_id=group_id,
uuid=row["uuid"],
display_name=row["display_name"],
sent_at_ms=index + 1,
day=day,
local_hour=BACKFILL_HOUR,
)
)
store.events.note_person(group_id, row["uuid"], row["display_name"], "", day)
seeded[group_id] = seeded.get(group_id, 0) + row["count"]
for group_id in seeded:
store.set_meta(group_id, BACKFILL_KEY, day)
return seeded