Krzysztof kuhy Rudnicki
405541be9b
One row per command attempt, not two aggregates written only on success. command_uses holds last-use only and daily_uses counts successes; both are rate-limiter state and both are written by store.record_use, which the dispatcher calls only when a handler returns cleanly. A gate refusal and a crash returned before it, so a command that failed every time looked exactly like one nobody ran, and nothing knew which command a counted message was. command_events now takes every attempt with its outcome, its duration and the item it served; upstream_calls takes every outbound request made while serving one. The two writes stay separate on purpose -- recording a refusal as a use is what slid the cooldown window forward every time somebody was told to wait -- and test_dispatcher_outcomes asserts last_used does not move. Backfilled from journald joined against message_events, with daily_uses filling the residual so the two sources cannot double-count. Two things were wrong on the first attempt and both were found by measuring rather than by reasoning: the window has to be symmetric, because sent_at_ms comes from the sender's phone and the journal stamp from this host (194 of 299 lines landed *before* their own message, by up to 3.2 s); and matching has to be a global closest-pair assignment, because per-row greedy cannot separate two commands seconds apart. Together those took the dropped rows from 137 to zero, and recovered 418 invocations back to 29 August. Also: !analityka in the group, twelve Polish lines, naming at most three people and only ones whose display name has been seen; an Analytics tab with Nivo charts; and continuous latency measurement, which answers the open question in DOCS-latency.md -- a text send is 235 ms and an image send about 2.2 s, so the upload really is where the time goes. EPHEMERAL_TABLES was doing two unrelated jobs and is now two tuples: what a schema bump rebuilds, and what leaving a group clears. daily_uses left the first (it is a backfill source now) and stayed in the second, and _drop_reshaped took over the repair a bump used to do for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T4QTp3h5Q3MvfrwGV8xqPf
6.3 KiB
What is private, and what stops it leaking
This repository is public. The bot it describes reads a private group chat, posts from a private photo album, and keeps a database of who said how much. None of that may be in here, and on 2026-08-30 some of it was.
What leaked, and how
Real first names, one full name, two phone numbers and a group member's Signal uuid were in the tree and in the public history. They arrived one way every time: somebody reached for a real name because it was the example in front of them — a test fixture built from a live database dump, an envelope pasted verbatim out of a running container, a help text written while looking at the actual group.
Fixing it needed the tree scrubbed, all 56 commits rewritten with
git filter-repo, and then the repository deleted and recreated, because
GitHub keeps orphaned commits fetchable by their SHA after a force-push and
only the repository going away takes them with it. The closed pull requests
and the star went too.
What is private, and where it lives
| Where | Never in git because | |
|---|---|---|
| the photos and who is in them | ~/.local/share/signal-bot/pack/ |
it is outside the repo, and .gitignore also names people.json, tags.json, media/, pack/ and every media extension |
| the message database | signal_bot.sqlite3 |
*.sqlite3 and its -wal/-shm siblings are ignored |
| the account's number, groups, inviters | .env |
.env is ignored; .env.sample carries the shape and no values |
| everyone's names | the database and the pack manifest | nothing reads them into the repo; the bot logs pack:p1, never a name |
| every command anybody has run | command_events and upstream_calls in the database |
the database is ignored, and the columns are chosen so there is nothing to leak: a resolved pack key or +args instead of the words typed, an exception's class instead of its message, a hostname instead of a URL, and a public identifier instead of a title. See DOCS-analytics.md. |
what people typed after a ! the bot did not know |
unknown_commands in the database |
the database is ignored; the log line names neither the word nor the message, and the export the management page writes goes to the state directory, never into this tree |
All of it is 700/600 on disk. The repository never needs any of it: the
tests, the documents and the offline end-to-end run use invented people —
alfa, beta, gamma, ala, basia, celina — and invented ids that base64-decode
twice into a sentence saying what they are.
The gate
scripts/check_personal_data.sh refuses a commit that carries personal data.
It runs on staged files, on the commit message, and over the whole tree in
CI. Two halves, because they fail differently:
- Names cannot be listed here — a file of real names kept in the repo to
check against is the leak. So the checker learns them at run time from
outside: the photo pack's manifest, the message database, and
.env. It never prints the word it matched, because this output reaches CI logs. - Shapes can be listed — a phone number, a uuid and a Signal group id
look like themselves anywhere, so those are matched by pattern and compared
against
.personal-data-allowlist, which holds only invented values.
On a machine with none of that private data (CI, a fresh clone) the name half has nothing to compare against and says so rather than passing quietly.
It found a real leak the first time it was run: the live test group's id, in
two spellings, in tests/test_config.py — which four rounds of grepping by
hand had missed, because nobody thinks to grep for base64.
When it fires
tests/test_thing.py:12: a name from your private data
Use an invented name. ala, basia, celina, darek are already in use;
so are +15550000000 and ids that decode to e2e-watched-group-id-32-bytes!!!.
DOCS-thing.md:40: phone number '+48123456789' is not in .personal-data-allowlist
Either it belongs to somebody — replace it — or it is invented, in which case
add it to .personal-data-allowlist after checking that it is. A line
there may be scoped to one file (signal_bot/popes.json::hadrian), which is how
a given name that is also a pope stays refused in every test fixture that has
no business carrying it.
Why no command commits the words it posts
Every content command posts what it just fetched. commons.caption_for says
why that matters: the text is fetched at run time and posted to the group;
none of it is ever written into this repository, which is why it may be free
text where traps.json may not.
!wiersz was written the other way first, and it is worth recording because
the other way looked entirely reasonable. A pack of several hundred short
public-domain Polish poems was harvested, filtered for names, and committed —
and scripts/check_personal_data.sh refused it. Three lines carried a real
group member's given name. After a further filter that discarded 31% of the
corpus, two still did.
The reason is structural rather than a matter of filtering harder. Polish
verse capitalises every line, so a capital at the start of a line carries
no information — and that is exactly where a name in a poem tends to sit, in a
dedication (Basi), an epigraph credit (Shakspeare.) or simply mid-sentence
across a line break. A filter strict enough to catch those rejects all verse.
So the pack holds Wolne Lektury slugs, and the poem, its title and its
author arrive over the wire when somebody types the command. Maria Konopnicka is posted into the group every week and appears nowhere in git.
The same shape as traps.json's page ids and art.json's, for the same
reason: an identifier cannot carry a name, and free text eventually will.
The failure mode this avoids is the nasty one — it appears only locally. CI has no private data to compare against, so CI stays green while every commit on the maintainer's machine is refused, over data nobody can edit out by hand.
What the gate cannot do
- It does not scan history. Rewriting that is a deliberate, separate act.
- It knows only what is on this machine. A name nobody has ever typed into the group is a name it cannot recognise.
- It does not stop the bot's own logs holding display names —
journalctl --user -u signal-botdoes, and--vacuum-time=is how that is emptied.