signal-bot/DOCS-privacy.md
Krzysztof kuhy Rudnicki 08599e6056 feat: two commands that answer with somebody else's work
!sztuka posts a work of art, skewed 60% Polish; !wiersz posts one short
public-domain Polish poem, whole. Both follow the rule the rest of the bot
already keeps: the repository commits identifiers, and the words arrive over
the wire when somebody asks.

!wiersz was written the other way first, and the gate caught it. A pack of
short public-domain poems was harvested, name-filtered and committed, and
check_personal_data.sh refused it: three lines carried a real group member's
given name, and two still did after a filter that had discarded 31% of the
corpus. That is structural, not a tuning problem -- Polish verse capitalises
every line, so a line-initial capital carries no signal, and that is exactly
where a name sits. The pack now holds Wolne Lektury slugs; the poem, its
title and its author are fetched at post time and appear nowhere in git.
It is also the better version: with no name filter there are no false
positives, so the pool is larger and the attribution is a full name again.

The gate then refused the art category list too, over a painter whose given
name is somebody's. That is the popes.json case exactly, so it takes the
same answer: a file-scoped allowlist entry, still refused everywhere else.

The Commons harvest is now three modules by what they are, not by where the
line count fell: commons_api is the conversation with Commons, commons_harvest
turns categories into a pool of licensed page ids, commons_cache makes an
interrupted run resumable. Along the way it gained licences in the same
request that lists a category (the second pass was 1,500 requests at 2.3s
each), maxlag instead of a guessed sleep, 5xx retries because one 504 used to
destroy a whole run, and Wikidata for works by creators of Polish
citizenship: the Polish pool grew 15,221 -> 20,092.

Batching categories through CirrusSearch looked like a further 6x and was
measured out again: 2,404 files where the authoritative enumeration returns
2,429, a different ~1% missing on every run.

The offline run is 19s where it was four minutes, almost all of it the
deafness gate waiting out three thirty-second liveness probes to prove
something provable in three; the tick is configuration now, and e2e enforces
a 40s budget on itself so that never silently comes back.
2026-08-31 21:32:07 +02:00

116 lines
5.9 KiB
Markdown

# What is private, and what stops it leaking
This repository is public. The bot it describes reads a private group chat,
posts from a private photo album, and keeps a database of who said how much.
None of that may be in here, and on 2026-08-30 some of it was.
## What leaked, and how
Real first names, one full name, two phone numbers and a group member's
Signal uuid were in the tree *and* in the public history. They arrived one
way every time: **somebody reached for a real name because it was the example
in front of them** — a test fixture built from a live database dump, an
envelope pasted verbatim out of a running container, a help text written
while looking at the actual group.
Fixing it needed the tree scrubbed, all 56 commits rewritten with
`git filter-repo`, and then the repository **deleted and recreated**, because
GitHub keeps orphaned commits fetchable by their SHA after a force-push and
only the repository going away takes them with it. The closed pull requests
and the star went too.
## What is private, and where it lives
| | Where | Never in git because |
|---|---|---|
| the photos and who is in them | `~/.local/share/signal-bot/pack/` | it is outside the repo, and `.gitignore` also names `people.json`, `tags.json`, `media/`, `pack/` and every media extension |
| the message database | `signal_bot.sqlite3` | `*.sqlite3` and its `-wal`/`-shm` siblings are ignored |
| the account's number, groups, inviters | `.env` | `.env` is ignored; `.env.sample` carries the shape and no values |
| everyone's names | the database and the pack manifest | nothing reads them into the repo; the bot logs `pack:p1`, never a name |
| what people typed after a `!` the bot did not know | `unknown_commands` in the database | the database is ignored; the log line names neither the word nor the message, and the export the management page writes goes to the state directory, never into this tree |
All of it is `700`/`600` on disk. The repository never needs any of it: the
tests, the documents and the offline end-to-end run use invented people —
alfa, beta, gamma, ala, basia, celina — and invented ids that base64-decode
twice into a sentence saying what they are.
## The gate
`scripts/check_personal_data.sh` refuses a commit that carries personal data.
It runs on staged files, on the commit message, and over the whole tree in
CI. Two halves, because they fail differently:
- **Names cannot be listed here** — a file of real names kept in the repo to
check against *is* the leak. So the checker learns them at run time from
outside: the photo pack's manifest, the message database, and `.env`. It
never prints the word it matched, because this output reaches CI logs.
- **Shapes can be listed** — a phone number, a uuid and a Signal group id
look like themselves anywhere, so those are matched by pattern and compared
against `.personal-data-allowlist`, which holds only invented values.
On a machine with none of that private data (CI, a fresh clone) the name half
has nothing to compare against and **says so** rather than passing quietly.
It found a real leak the first time it was run: the live test group's id, in
two spellings, in `tests/test_config.py` — which four rounds of grepping by
hand had missed, because nobody thinks to grep for base64.
## When it fires
```
tests/test_thing.py:12: a name from your private data
```
Use an invented name. `ala`, `basia`, `celina`, `darek` are already in use;
so are `+15550000000` and ids that decode to `e2e-watched-group-id-32-bytes!!!`.
```
DOCS-thing.md:40: phone number '+48123456789' is not in .personal-data-allowlist
```
Either it belongs to somebody — replace it — or it is invented, in which case
add it to `.personal-data-allowlist` **after checking that it is**. A line
there may be scoped to one file (`signal_bot/popes.json::hadrian`), which is how
a given name that is also a pope stays refused in every test fixture that has
no business carrying it.
## Why no command commits the words it posts
Every content command posts what it just fetched. `commons.caption_for` says
why that matters: *the text is fetched at run time and posted to the group;
none of it is ever written into this repository, which is why it may be free
text where `traps.json` may not.*
`!wiersz` was written the other way first, and it is worth recording because
the other way looked entirely reasonable. A pack of several hundred short
public-domain Polish poems was harvested, filtered for names, and committed —
and `scripts/check_personal_data.sh` refused it. Three lines carried a real
group member's given name. After a further filter that discarded 31% of the
corpus, two still did.
The reason is structural rather than a matter of filtering harder. Polish
verse capitalises **every** line, so a capital at the start of a line carries
no information — and that is exactly where a name in a poem tends to sit, in a
dedication (`Basi`), an epigraph credit (`Shakspeare.`) or simply mid-sentence
across a line break. A filter strict enough to catch those rejects all verse.
So the pack holds Wolne Lektury **slugs**, and the poem, its title and its
author arrive over the wire when somebody types the command. `Maria
Konopnicka` is posted into the group every week and appears nowhere in git.
The same shape as `traps.json`'s page ids and `art.json`'s, for the same
reason: an identifier cannot carry a name, and free text eventually will.
The failure mode this avoids is the nasty one — it appears **only locally**.
CI has no private data to compare against, so CI stays green while every
commit on the maintainer's machine is refused, over data nobody can edit out
by hand.
## What the gate cannot do
- It does not scan history. Rewriting that is a deliberate, separate act.
- It knows only what is on *this* machine. A name nobody has ever typed into
the group is a name it cannot recognise.
- It does not stop the bot's own logs holding display names —
`journalctl --user -u signal-bot` does, and `--vacuum-time=` is how that is
emptied.