Krzysztof kuhy Rudnicki
405541be9b
One row per command attempt, not two aggregates written only on success. command_uses holds last-use only and daily_uses counts successes; both are rate-limiter state and both are written by store.record_use, which the dispatcher calls only when a handler returns cleanly. A gate refusal and a crash returned before it, so a command that failed every time looked exactly like one nobody ran, and nothing knew which command a counted message was. command_events now takes every attempt with its outcome, its duration and the item it served; upstream_calls takes every outbound request made while serving one. The two writes stay separate on purpose -- recording a refusal as a use is what slid the cooldown window forward every time somebody was told to wait -- and test_dispatcher_outcomes asserts last_used does not move. Backfilled from journald joined against message_events, with daily_uses filling the residual so the two sources cannot double-count. Two things were wrong on the first attempt and both were found by measuring rather than by reasoning: the window has to be symmetric, because sent_at_ms comes from the sender's phone and the journal stamp from this host (194 of 299 lines landed *before* their own message, by up to 3.2 s); and matching has to be a global closest-pair assignment, because per-row greedy cannot separate two commands seconds apart. Together those took the dropped rows from 137 to zero, and recovered 418 invocations back to 29 August. Also: !analityka in the group, twelve Polish lines, naming at most three people and only ones whose display name has been seen; an Analytics tab with Nivo charts; and continuous latency measurement, which answers the open question in DOCS-latency.md -- a text send is 235 ms and an image send about 2.2 s, so the upload really is where the time goes. EPHEMERAL_TABLES was doing two unrelated jobs and is now two tuples: what a schema bump rebuilds, and what leaving a group clears. daily_uses left the first (it is a backfill source now) and stayed in the second, and _drop_reshaped took over the repair a bump used to do for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T4QTp3h5Q3MvfrwGV8xqPf
4.5 KiB
Statistics
This is the message history: who said how much, when. The command
history -- what people asked the bot for, whether it worked and how long it
took -- is a different table and a different package; see
DOCS-analytics.md.
What is recorded
One row per message in message_events: sender uuid and the display name they
had at the time, the Signal send timestamp, the Warsaw day and hour, the media
kind and count, the character length, and whether it was a command or a reply.
Never the message body. tests/test_stats_events.py asserts the column set
exactly, so a later edit cannot quietly add one. Reactions are not messages and
stay out. The bot's own posts never count — decided by which envelope branch
carried the payload, not by comparing identities, because the account is
spelled as a uuid on some frames and a +E.164 number on others.
A photo posted without a caption counts. Before 0.1.0 it was worth zero, which is why everyone's numbers jumped on release.
Days
A day is a calendar day in Europe/Warsaw, computed when the event is recorded and stored as a string. Every query groups on that string rather than on a range of timestamps, because Warsaw days are not all 24 hours long — 2026-03-29 is 23 and 2026-10-25 is 25 — and a timestamp range moves the boundary twice a year.
Two consequences worth knowing before editing anything here:
- "Yesterday" is
date - timedelta(days=1)on the date. Subtracting 86400 seconds lands at 23:00 the previous evening on a short day. - Elapsed-time arithmetic must convert to UTC first. Subtracting two aware datetimes that share a tzinfo gives the wall-clock difference, so the naive spelling reports every day as exactly 24 hours and hides the bug.
The nightly report
Posted at REPORT_HOUR:REPORT_MINUTE for the previous full day: the table,
members who said nothing at 0, a summary line, and up to three highlights
chosen from around twenty candidates — one per family, so a record, a share
and a crown streak about the same person do not become three lines.
Capped at twelve lines. Names are not padded: Signal renders a proportional
font, so ljust lines up in a terminal and on nobody's phone.
Only people whose display name has been seen are named. Somebody who has never written is counted, not named — the group listing knows them only as a raw uuid or a phone number, and printing either into a chat is a leak.
Outages
Nothing is ever deleted, and the last day reported is recorded per group.
- Missed counts come back on their own. An event carries Signal's own send
timestamp, signal-cli keeps receiving in json-rpc mode while the bot is
down, and the replay on reconnect is absorbed by the
(group, uuid, sent_at_ms)primary key. Messages sent during an outage are filed under the day they were actually sent. - Missed reports are posted when the bot returns, oldest first, bounded at a week. A day nobody spoke on is marked settled rather than reconsidered forever; a day whose send failed is left unmarked so the next run retries it.
The backfilled day
2026-08-30 in the real group is carried over from the pre-0.1.0 counter, which
had no timestamps. Its totals are real; its hours and lengths are not, and its
window was 22 hours rather than a day. It is marked in group_meta, claims no
average message length, and is excluded from every record and average — a
merged day would otherwise hand somebody a personal best they never set.
The counter could not be split across the two days it spanned. Four places
were checked for a real inbound timestamp: journald (commands only),
signal-cli's account.db (no inbound table), Signal Desktop (unlinked, and it
does not backfill on link), and signal-api/attachments (dated, but media
only and with no sender). The attachment mtimes put 12 files on 29 August
against 79 on the 30th inside that window, which is why the whole counter sits
on the 30th.
Debugging the wire format
LOG_LEVEL=DEBUG makes the dispatcher log each frame's shape — keys and
value types, never values, with a small allowlist for MIME and groupInfo types
that are format rather than content. Libraries stay at WARNING whatever the
bot is set to.
journalctl --user -u signal-bot -f | grep 'envelope shape'
Captured shapes live in DOCS-signal-envelopes.md. Read that before changing
anything that parses an envelope: quote carries its own nested attachments
array belonging to the quoted message, and counting it would credit a
replier with somebody else's photo.