signal-bot/signal_bot/stats/events.py
Krzysztof kuhy Rudnicki dc1995b558 feat: the message event log, and the nightly report built on it
The tally was one integer per person, deleted after each post. Nothing
dated survived it, so weeks, months, records, streaks and weekday
averages were all unanswerable. This replaces it with an append-only
per-message log holding metadata only -- never a message body -- and
rebuilds the nightly report on top.

What changes for the group: a photo posted without a caption now counts
(it was worth zero), stickers count, the bot's own messages no longer
do, people who said nothing appear with a 0, and the report covers the
previous full calendar day in Europe/Warsaw instead of "since the last
post" -- which used to merge two days whenever a group went quiet.

Notable decisions, each with the trap it avoids:

- Events and people are PERMANENT_TABLES: neither forget_group nor a
  future SCHEMA_VERSION bump may drop them. Both paths used to cost a
  day's counter and would now cost the entire history, and forget_group
  fires on a listing that merely omits members.
- is_own comes from which envelope branch carried the payload, not from
  comparing identities. The account is spelled as a uuid on some frames
  and a +E.164 number on others, so any comparison misses one of them.
- The roster is stashed in refresh(), not _adopt(), which returns early
  once the watched set stops changing -- i.e. always, in the steady
  state.
- The roster matches on uuid OR number because /v1/groups spells kuhy as
  +48123456789 while his messages carry an ACI uuid. Joining on uuid
  alone would have listed him twice every night, once with 0.
- Members whose display name we have never seen are counted, not named:
  their only spellings are a uuid or a phone number.
- (group, uuid, sent_at_ms) is the primary key, so a websocket
  redelivery after a reconnect cannot inflate the counts.
- Days come from a stored Warsaw date string, never a timestamp range:
  2026-03-29 is 23 hours long and 2026-10-25 is 25.
- Polish number agreement is a real rule with a 12-14 exception, so
  "44 znaki" and "45 znaków" are both spelled correctly.

config.py and startup.py were two lines under the 250-line cap and both
had to grow, so config_env and startup_retry come out of them first, as
pure moves.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015d48t4qsQLBJnDwDeo2Nom
2026-08-30 19:14:43 +02:00

159 lines
5.9 KiB
Python

# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""The append-only message log, and the people who wrote it.
One row per counted message, holding metadata only. There is deliberately no
column for the message body: the group's statistics need to know that someone
wrote 140 characters at 23:41, never what they wrote. ``test_events`` pins the
column set exactly so a later edit cannot quietly add one.
Two tables, because they answer two different questions. ``message_events``
answers "who said how much, when". ``group_people`` answers "who is in this
group", which the event log cannot: somebody who has never spoken has no
events, and the group listing spells its members inconsistently -- an ACI uuid
for most people but a +E.164 number for anyone the account has as a contact.
Carrying both spellings is what lets a roster entry find its events.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import sqlite3
from collections.abc import Iterable
@dataclass(frozen=True, slots=True)
class MessageEvent:
"""One counted message, reduced to the metadata the statistics need."""
group_id: str
uuid: str
display_name: str
sent_at_ms: int
day: str
local_hour: int
media_kind: str = ""
media_count: int = 0
char_len: int = 0
is_command: bool = False
is_reply: bool = False
@dataclass(frozen=True, slots=True)
class Person:
"""Somebody who has written at least one message in a group."""
uuid: str
display_name: str
number: str
class EventStore:
"""Reads and writes the message history. Shares the Store's connection."""
def __init__(self, conn: sqlite3.Connection) -> None:
"""Wrap an open connection whose schema has already been applied."""
self._conn = conn
@property
def connection(self) -> sqlite3.Connection:
"""The shared connection, for the query modules in this package."""
return self._conn
def record(self, event: MessageEvent) -> None:
"""Append one message, ignoring a frame that has already been seen.
``ON CONFLICT DO NOTHING`` against the (group, sender, timestamp)
primary key is what makes a websocket redelivery harmless. signal-cli
replays undelivered frames when the bot reconnects, so without this
every restart would silently inflate the counts it just published.
"""
self._conn.execute(
"""
INSERT INTO message_events (
group_id, uuid, display_name, sent_at_ms, day, local_hour,
media_kind, media_count, char_len, is_command, is_reply
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(group_id, uuid, sent_at_ms) DO NOTHING
""",
(
event.group_id,
event.uuid,
event.display_name,
event.sent_at_ms,
event.day,
event.local_hour,
event.media_kind,
event.media_count,
event.char_len,
int(event.is_command),
int(event.is_reply),
),
)
self._conn.commit()
def note_person(
self, group_id: str, uuid: str, display_name: str, number: str, day: str
) -> None:
"""Record that somebody wrote in a group today, under the name they used.
``display_name`` is last-write-wins here on purpose: this table
answers "what do we call them now". The name they had at the time is
preserved per row in ``message_events``, so a rename cannot rewrite
past attribution.
``number`` only updates when a non-empty one arrives. Envelopes carry
it inconsistently, and an empty string overwriting a known number
would break the roster join for exactly the people it exists for.
"""
self._conn.execute(
"""
INSERT INTO group_people (
group_id, uuid, display_name, number, first_seen, last_seen
) VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(group_id, uuid) DO UPDATE SET
display_name = excluded.display_name,
number = CASE
WHEN excluded.number <> '' THEN excluded.number
ELSE group_people.number
END,
last_seen = excluded.last_seen
""",
(group_id, uuid, display_name, number, day, day),
)
self._conn.commit()
def people(self, group_id: str) -> list[Person]:
"""Everyone who has ever written in a group, by display name."""
rows = self._conn.execute(
"SELECT uuid, display_name, number FROM group_people "
"WHERE group_id = ? ORDER BY display_name ASC",
(group_id,),
).fetchall()
return [Person(r["uuid"], r["display_name"], r["number"]) for r in rows]
def roster(self, group_id: str, identities: Iterable[str]) -> list[Person]:
"""The named people among a set of listing identities.
The group listing spells a member as either an ACI uuid or a +E.164
number, so both columns are candidates. Anyone it cannot match has
never written here and has no name we could print -- the caller
counts them rather than naming them, because printing a raw uuid or
somebody's phone number into a group chat is a leak.
"""
wanted = {i for i in identities if i}
if not wanted:
return []
return [p for p in self.people(group_id) if p.uuid in wanted or p.number in wanted]
def first_day(self, group_id: str) -> str | None:
"""The earliest day with an event, or None when there is no history."""
row = self._conn.execute(
"SELECT MIN(day) AS first FROM message_events WHERE group_id = ?",
(group_id,),
).fetchone()
first: str | None = row["first"]
return first