# Statistics

This is the **message** history: who said how much, when. The **command**
history -- what people asked the bot for, whether it worked and how long it
took -- is a different table and a different package; see
`DOCS-analytics.md`.

## What is recorded

One row per message in `message_events`: sender uuid and the display name they
had at the time, the Signal send timestamp, the Warsaw day and hour, the media
kind and count, the character length, and whether it was a command or a reply.

**Never the message body.** `tests/test_stats_events.py` asserts the column set
exactly, so a later edit cannot quietly add one. Reactions are not messages and
stay out. The bot's own posts never count — decided by which envelope branch
carried the payload, not by comparing identities, because the account is
spelled as a uuid on some frames and a +E.164 number on others.

A photo posted without a caption counts. Before 0.1.0 it was worth zero, which
is why everyone's numbers jumped on release.

## Days

A day is a calendar day in **Europe/Warsaw**, computed when the event is
recorded and stored as a string. Every query groups on that string rather than
on a range of timestamps, because Warsaw days are not all 24 hours long —
2026-03-29 is 23 and 2026-10-25 is 25 — and a timestamp range moves the
boundary twice a year.

Two consequences worth knowing before editing anything here:

- "Yesterday" is `date - timedelta(days=1)` on the *date*. Subtracting 86400
  seconds lands at 23:00 the previous evening on a short day.
- Elapsed-time arithmetic must convert to UTC first. Subtracting two aware
  datetimes that share a tzinfo gives the *wall-clock* difference, so the naive
  spelling reports every day as exactly 24 hours and hides the bug.

## The nightly report

Posted at `REPORT_HOUR:REPORT_MINUTE` for the previous full day: the table,
members who said nothing at 0, a summary line, and up to three highlights
chosen from around twenty candidates — one per family, so a record, a share
and a crown streak about the same person do not become three lines.

Capped at twelve lines. Names are not padded: Signal renders a proportional
font, so `ljust` lines up in a terminal and on nobody's phone.

Only people whose display name has been seen are named. Somebody who has never
written is counted, not named — the group listing knows them only as a raw
uuid or a phone number, and printing either into a chat is a leak.

Seven numbers are funny to this group and get a marker wherever they appear
as a *count* — `69 (nice)`, `2137 🙏`, `1410 ⚔️`, `1916 🇵🇱`, `420 🌿`,
`314 🥧`, `67 🤷` — in the nightly report, every `!stats_*` and `!ja`, and the
highlights. Percentages, shares and the `(N 📷)` aside stay bare: a 69% share
is not a 69, and marking the photo count next to an already-marked message
count would be two jokes on one line. The table is `FUNNY` in `stats/text.py`.

## Outages

Nothing is ever deleted, and the last day reported is recorded per group.

- **Missed counts** come back on their own. An event carries Signal's own send
  timestamp, signal-cli keeps receiving in json-rpc mode while the bot is
  down, and the replay on reconnect is absorbed by the `(group, uuid,
  sent_at_ms)` primary key. Messages sent during an outage are filed under the
  day they were actually sent.
- **Missed reports** are posted when the bot returns, oldest first, bounded at
  a week. A day nobody spoke on is marked settled rather than reconsidered
  forever; a day whose *send* failed is left unmarked so the next run retries
  it.

## The backfilled day

2026-08-30 in the real group is carried over from the pre-0.1.0 counter, which
had no timestamps. Its totals are real; its hours and lengths are not, and its
window was 22 hours rather than a day. It is marked in `group_meta`, claims no
average message length, and is excluded from every record and average — a
merged day would otherwise hand somebody a personal best they never set.

The counter could not be split across the two days it spanned. Four places
were checked for a real inbound timestamp: journald (commands only),
signal-cli's `account.db` (no inbound table), Signal Desktop (unlinked, and it
does not backfill on link), and `signal-api/attachments` (dated, but media
only and with no sender). The attachment mtimes put 12 files on 29 August
against 79 on the 30th inside that window, which is why the whole counter sits
on the 30th.

## Debugging the wire format

`LOG_LEVEL=DEBUG` makes the dispatcher log each frame's *shape* — keys and
value types, never values, with a small allowlist for MIME and groupInfo types
that are format rather than content. Libraries stay at WARNING whatever the
bot is set to.

    journalctl --user -u signal-bot -f | grep 'envelope shape'

Captured shapes live in `DOCS-signal-envelopes.md`. Read that before changing
anything that parses an envelope: `quote` carries its own nested `attachments`
array belonging to the *quoted* message, and counting it would credit a
replier with somebody else's photo.