signal-bot/DOCS-statistics.md

Statistics

This is the message history: who said how much, when. The command history -- what people asked the bot for, whether it worked and how long it took -- is a different table and a different package; see DOCS-analytics.md.

What is recorded

One row per message in message_events: sender uuid and the display name they had at the time, the Signal send timestamp, the Warsaw day and hour, the media kind and count, the character length, and whether it was a command or a reply.

Never the message body. tests/test_stats_events.py asserts the column set exactly, so a later edit cannot quietly add one. Reactions are not messages and stay out. The bot's own posts never count — decided by which envelope branch carried the payload, not by comparing identities, because the account is spelled as a uuid on some frames and a +E.164 number on others.

A photo posted without a caption counts. Before 0.1.0 it was worth zero, which is why everyone's numbers jumped on release.

Days

A day is a calendar day in Europe/Warsaw, computed when the event is recorded and stored as a string. Every query groups on that string rather than on a range of timestamps, because Warsaw days are not all 24 hours long — 2026-03-29 is 23 and 2026-10-25 is 25 — and a timestamp range moves the boundary twice a year.

Two consequences worth knowing before editing anything here:

• "Yesterday" is date - timedelta(days=1) on the date. Subtracting 86400 seconds lands at 23:00 the previous evening on a short day.
• Elapsed-time arithmetic must convert to UTC first. Subtracting two aware datetimes that share a tzinfo gives the wall-clock difference, so the naive spelling reports every day as exactly 24 hours and hides the bug.

The nightly report

Posted at REPORT_HOUR:REPORT_MINUTE for the previous full day: the table, members who said nothing at 0, a summary line, and up to three highlights chosen from around twenty candidates — one per family, so a record, a share and a crown streak about the same person do not become three lines.

Capped at twelve lines. Names are not padded: Signal renders a proportional font, so ljust lines up in a terminal and on nobody's phone.

Only people whose display name has been seen are named. Somebody who has never written is counted, not named — the group listing knows them only as a raw uuid or a phone number, and printing either into a chat is a leak.

Seven numbers are funny to this group and get a marker wherever they appear as a count — 69 (nice), 2137 🙏, 1410 ⚔️, 1916 🇵🇱, 420 🌿, 314 🥧, 67 🤷 — in the nightly report, every !stats_* and !ja, and the highlights. Percentages, shares and the (N 📷) aside stay bare: a 69% share is not a 69, and marking the photo count next to an already-marked message count would be two jokes on one line. The table is FUNNY in stats/text.py.

Outages

Nothing is ever deleted, and the last day reported is recorded per group.

• Missed counts come back on their own. An event carries Signal's own send timestamp, signal-cli keeps receiving in json-rpc mode while the bot is down, and the replay on reconnect is absorbed by the (group, uuid, sent_at_ms) primary key. Messages sent during an outage are filed under the day they were actually sent.
• Missed reports are posted when the bot returns, oldest first, bounded at a week. A day nobody spoke on is marked settled rather than reconsidered forever; a day whose send failed is left unmarked so the next run retries it.

The backfilled day

2026-08-30 in the real group is carried over from the pre-0.1.0 counter, which had no timestamps. Its totals are real; its hours and lengths are not, and its window was 22 hours rather than a day. It is marked in group_meta, claims no average message length, and is excluded from every record and average — a merged day would otherwise hand somebody a personal best they never set.

The counter could not be split across the two days it spanned. Four places were checked for a real inbound timestamp: journald (commands only), signal-cli's account.db (no inbound table), Signal Desktop (unlinked, and it does not backfill on link), and signal-api/attachments (dated, but media only and with no sender). The attachment mtimes put 12 files on 29 August against 79 on the 30th inside that window, which is why the whole counter sits on the 30th.

Debugging the wire format

LOG_LEVEL=DEBUG makes the dispatcher log each frame's shape — keys and value types, never values, with a small allowlist for MIME and groupInfo types that are format rather than content. Libraries stay at WARNING whatever the bot is set to.

journalctl --user -u signal-bot -f | grep 'envelope shape'

Captured shapes live in DOCS-signal-envelopes.md. Read that before changing anything that parses an envelope: quote carries its own nested attachments array belonging to the quoted message, and counting it would credit a replier with somebody else's photo.