Krzysztof kuhy Rudnicki
08599e6056
!sztuka posts a work of art, skewed 60% Polish; !wiersz posts one short public-domain Polish poem, whole. Both follow the rule the rest of the bot already keeps: the repository commits identifiers, and the words arrive over the wire when somebody asks. !wiersz was written the other way first, and the gate caught it. A pack of short public-domain poems was harvested, name-filtered and committed, and check_personal_data.sh refused it: three lines carried a real group member's given name, and two still did after a filter that had discarded 31% of the corpus. That is structural, not a tuning problem -- Polish verse capitalises every line, so a line-initial capital carries no signal, and that is exactly where a name sits. The pack now holds Wolne Lektury slugs; the poem, its title and its author are fetched at post time and appear nowhere in git. It is also the better version: with no name filter there are no false positives, so the pool is larger and the attribution is a full name again. The gate then refused the art category list too, over a painter whose given name is somebody's. That is the popes.json case exactly, so it takes the same answer: a file-scoped allowlist entry, still refused everywhere else. The Commons harvest is now three modules by what they are, not by where the line count fell: commons_api is the conversation with Commons, commons_harvest turns categories into a pool of licensed page ids, commons_cache makes an interrupted run resumable. Along the way it gained licences in the same request that lists a category (the second pass was 1,500 requests at 2.3s each), maxlag instead of a guessed sleep, 5xx retries because one 504 used to destroy a whole run, and Wikidata for works by creators of Polish citizenship: the Polish pool grew 15,221 -> 20,092. Batching categories through CirrusSearch looked like a further 6x and was measured out again: 2,404 files where the authoritative enumeration returns 2,429, a different ~1% missing on every run. The offline run is 19s where it was four minutes, almost all of it the deafness gate waiting out three thirty-second liveness probes to prove something provable in three; the tick is configuration now, and e2e enforces a 40s budget on itself so that never silently comes back.
5.9 KiB
What is private, and what stops it leaking
This repository is public. The bot it describes reads a private group chat, posts from a private photo album, and keeps a database of who said how much. None of that may be in here, and on 2026-08-30 some of it was.
What leaked, and how
Real first names, one full name, two phone numbers and a group member's Signal uuid were in the tree and in the public history. They arrived one way every time: somebody reached for a real name because it was the example in front of them — a test fixture built from a live database dump, an envelope pasted verbatim out of a running container, a help text written while looking at the actual group.
Fixing it needed the tree scrubbed, all 56 commits rewritten with
git filter-repo, and then the repository deleted and recreated, because
GitHub keeps orphaned commits fetchable by their SHA after a force-push and
only the repository going away takes them with it. The closed pull requests
and the star went too.
What is private, and where it lives
| Where | Never in git because | |
|---|---|---|
| the photos and who is in them | ~/.local/share/signal-bot/pack/ |
it is outside the repo, and .gitignore also names people.json, tags.json, media/, pack/ and every media extension |
| the message database | signal_bot.sqlite3 |
*.sqlite3 and its -wal/-shm siblings are ignored |
| the account's number, groups, inviters | .env |
.env is ignored; .env.sample carries the shape and no values |
| everyone's names | the database and the pack manifest | nothing reads them into the repo; the bot logs pack:p1, never a name |
what people typed after a ! the bot did not know |
unknown_commands in the database |
the database is ignored; the log line names neither the word nor the message, and the export the management page writes goes to the state directory, never into this tree |
All of it is 700/600 on disk. The repository never needs any of it: the
tests, the documents and the offline end-to-end run use invented people —
alfa, beta, gamma, ala, basia, celina — and invented ids that base64-decode
twice into a sentence saying what they are.
The gate
scripts/check_personal_data.sh refuses a commit that carries personal data.
It runs on staged files, on the commit message, and over the whole tree in
CI. Two halves, because they fail differently:
- Names cannot be listed here — a file of real names kept in the repo to
check against is the leak. So the checker learns them at run time from
outside: the photo pack's manifest, the message database, and
.env. It never prints the word it matched, because this output reaches CI logs. - Shapes can be listed — a phone number, a uuid and a Signal group id
look like themselves anywhere, so those are matched by pattern and compared
against
.personal-data-allowlist, which holds only invented values.
On a machine with none of that private data (CI, a fresh clone) the name half has nothing to compare against and says so rather than passing quietly.
It found a real leak the first time it was run: the live test group's id, in
two spellings, in tests/test_config.py — which four rounds of grepping by
hand had missed, because nobody thinks to grep for base64.
When it fires
tests/test_thing.py:12: a name from your private data
Use an invented name. ala, basia, celina, darek are already in use;
so are +15550000000 and ids that decode to e2e-watched-group-id-32-bytes!!!.
DOCS-thing.md:40: phone number '+48123456789' is not in .personal-data-allowlist
Either it belongs to somebody — replace it — or it is invented, in which case
add it to .personal-data-allowlist after checking that it is. A line
there may be scoped to one file (signal_bot/popes.json::hadrian), which is how
a given name that is also a pope stays refused in every test fixture that has
no business carrying it.
Why no command commits the words it posts
Every content command posts what it just fetched. commons.caption_for says
why that matters: the text is fetched at run time and posted to the group;
none of it is ever written into this repository, which is why it may be free
text where traps.json may not.
!wiersz was written the other way first, and it is worth recording because
the other way looked entirely reasonable. A pack of several hundred short
public-domain Polish poems was harvested, filtered for names, and committed —
and scripts/check_personal_data.sh refused it. Three lines carried a real
group member's given name. After a further filter that discarded 31% of the
corpus, two still did.
The reason is structural rather than a matter of filtering harder. Polish
verse capitalises every line, so a capital at the start of a line carries
no information — and that is exactly where a name in a poem tends to sit, in a
dedication (Basi), an epigraph credit (Shakspeare.) or simply mid-sentence
across a line break. A filter strict enough to catch those rejects all verse.
So the pack holds Wolne Lektury slugs, and the poem, its title and its
author arrive over the wire when somebody types the command. Maria Konopnicka is posted into the group every week and appears nowhere in git.
The same shape as traps.json's page ids and art.json's, for the same
reason: an identifier cannot carry a name, and free text eventually will.
The failure mode this avoids is the nasty one — it appears only locally. CI has no private data to compare against, so CI stays green while every commit on the maintainer's machine is refused, over data nobody can edit out by hand.
What the gate cannot do
- It does not scan history. Rewriting that is a deliberate, separate act.
- It knows only what is on this machine. A name nobody has ever typed into the group is a name it cannot recognise.
- It does not stop the bot's own logs holding display names —
journalctl --user -u signal-botdoes, and--vacuum-time=is how that is emptied.