mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-22 15:22:23 +01:00
!sztuka posts a work of art, skewed 60% Polish; !wiersz posts one short public-domain Polish poem, whole. Both follow the rule the rest of the bot already keeps: the repository commits identifiers, and the words arrive over the wire when somebody asks. !wiersz was written the other way first, and the gate caught it. A pack of short public-domain poems was harvested, name-filtered and committed, and check_personal_data.sh refused it: three lines carried a real group member's given name, and two still did after a filter that had discarded 31% of the corpus. That is structural, not a tuning problem -- Polish verse capitalises every line, so a line-initial capital carries no signal, and that is exactly where a name sits. The pack now holds Wolne Lektury slugs; the poem, its title and its author are fetched at post time and appear nowhere in git. It is also the better version: with no name filter there are no false positives, so the pool is larger and the attribution is a full name again. The gate then refused the art category list too, over a painter whose given name is somebody's. That is the popes.json case exactly, so it takes the same answer: a file-scoped allowlist entry, still refused everywhere else. The Commons harvest is now three modules by what they are, not by where the line count fell: commons_api is the conversation with Commons, commons_harvest turns categories into a pool of licensed page ids, commons_cache makes an interrupted run resumable. Along the way it gained licences in the same request that lists a category (the second pass was 1,500 requests at 2.3s each), maxlag instead of a guessed sleep, 5xx retries because one 504 used to destroy a whole run, and Wikidata for works by creators of Polish citizenship: the Polish pool grew 15,221 -> 20,092. Batching categories through CirrusSearch looked like a further 6x and was measured out again: 2,404 files where the authoritative enumeration returns 2,429, a different ~1% missing on every run. The offline run is 19s where it was four minutes, almost all of it the deafness gate waiting out three thirty-second liveness probes to prove something provable in three; the tick is configuration now, and e2e enforces a 40s budget on itself so that never silently comes back.
116 lines
5.9 KiB
Markdown
116 lines
5.9 KiB
Markdown
# What is private, and what stops it leaking
|
|
|
|
This repository is public. The bot it describes reads a private group chat,
|
|
posts from a private photo album, and keeps a database of who said how much.
|
|
None of that may be in here, and on 2026-08-30 some of it was.
|
|
|
|
## What leaked, and how
|
|
|
|
Real first names, one full name, two phone numbers and a group member's
|
|
Signal uuid were in the tree *and* in the public history. They arrived one
|
|
way every time: **somebody reached for a real name because it was the example
|
|
in front of them** — a test fixture built from a live database dump, an
|
|
envelope pasted verbatim out of a running container, a help text written
|
|
while looking at the actual group.
|
|
|
|
Fixing it needed the tree scrubbed, all 56 commits rewritten with
|
|
`git filter-repo`, and then the repository **deleted and recreated**, because
|
|
GitHub keeps orphaned commits fetchable by their SHA after a force-push and
|
|
only the repository going away takes them with it. The closed pull requests
|
|
and the star went too.
|
|
|
|
## What is private, and where it lives
|
|
|
|
| | Where | Never in git because |
|
|
|---|---|---|
|
|
| the photos and who is in them | `~/.local/share/signal-bot/pack/` | it is outside the repo, and `.gitignore` also names `people.json`, `tags.json`, `media/`, `pack/` and every media extension |
|
|
| the message database | `signal_bot.sqlite3` | `*.sqlite3` and its `-wal`/`-shm` siblings are ignored |
|
|
| the account's number, groups, inviters | `.env` | `.env` is ignored; `.env.sample` carries the shape and no values |
|
|
| everyone's names | the database and the pack manifest | nothing reads them into the repo; the bot logs `pack:p1`, never a name |
|
|
| what people typed after a `!` the bot did not know | `unknown_commands` in the database | the database is ignored; the log line names neither the word nor the message, and the export the management page writes goes to the state directory, never into this tree |
|
|
|
|
All of it is `700`/`600` on disk. The repository never needs any of it: the
|
|
tests, the documents and the offline end-to-end run use invented people —
|
|
alfa, beta, gamma, ala, basia, celina — and invented ids that base64-decode
|
|
twice into a sentence saying what they are.
|
|
|
|
## The gate
|
|
|
|
`scripts/check_personal_data.sh` refuses a commit that carries personal data.
|
|
It runs on staged files, on the commit message, and over the whole tree in
|
|
CI. Two halves, because they fail differently:
|
|
|
|
- **Names cannot be listed here** — a file of real names kept in the repo to
|
|
check against *is* the leak. So the checker learns them at run time from
|
|
outside: the photo pack's manifest, the message database, and `.env`. It
|
|
never prints the word it matched, because this output reaches CI logs.
|
|
- **Shapes can be listed** — a phone number, a uuid and a Signal group id
|
|
look like themselves anywhere, so those are matched by pattern and compared
|
|
against `.personal-data-allowlist`, which holds only invented values.
|
|
|
|
On a machine with none of that private data (CI, a fresh clone) the name half
|
|
has nothing to compare against and **says so** rather than passing quietly.
|
|
|
|
It found a real leak the first time it was run: the live test group's id, in
|
|
two spellings, in `tests/test_config.py` — which four rounds of grepping by
|
|
hand had missed, because nobody thinks to grep for base64.
|
|
|
|
## When it fires
|
|
|
|
```
|
|
tests/test_thing.py:12: a name from your private data
|
|
```
|
|
|
|
Use an invented name. `ala`, `basia`, `celina`, `darek` are already in use;
|
|
so are `+15550000000` and ids that decode to `e2e-watched-group-id-32-bytes!!!`.
|
|
|
|
```
|
|
DOCS-thing.md:40: phone number '+48123456789' is not in .personal-data-allowlist
|
|
```
|
|
|
|
Either it belongs to somebody — replace it — or it is invented, in which case
|
|
add it to `.personal-data-allowlist` **after checking that it is**. A line
|
|
there may be scoped to one file (`signal_bot/popes.json::hadrian`), which is how
|
|
a given name that is also a pope stays refused in every test fixture that has
|
|
no business carrying it.
|
|
|
|
## Why no command commits the words it posts
|
|
|
|
Every content command posts what it just fetched. `commons.caption_for` says
|
|
why that matters: *the text is fetched at run time and posted to the group;
|
|
none of it is ever written into this repository, which is why it may be free
|
|
text where `traps.json` may not.*
|
|
|
|
`!wiersz` was written the other way first, and it is worth recording because
|
|
the other way looked entirely reasonable. A pack of several hundred short
|
|
public-domain Polish poems was harvested, filtered for names, and committed —
|
|
and `scripts/check_personal_data.sh` refused it. Three lines carried a real
|
|
group member's given name. After a further filter that discarded 31% of the
|
|
corpus, two still did.
|
|
|
|
The reason is structural rather than a matter of filtering harder. Polish
|
|
verse capitalises **every** line, so a capital at the start of a line carries
|
|
no information — and that is exactly where a name in a poem tends to sit, in a
|
|
dedication (`Basi`), an epigraph credit (`Shakspeare.`) or simply mid-sentence
|
|
across a line break. A filter strict enough to catch those rejects all verse.
|
|
|
|
So the pack holds Wolne Lektury **slugs**, and the poem, its title and its
|
|
author arrive over the wire when somebody types the command. `Maria
|
|
Konopnicka` is posted into the group every week and appears nowhere in git.
|
|
The same shape as `traps.json`'s page ids and `art.json`'s, for the same
|
|
reason: an identifier cannot carry a name, and free text eventually will.
|
|
|
|
The failure mode this avoids is the nasty one — it appears **only locally**.
|
|
CI has no private data to compare against, so CI stays green while every
|
|
commit on the maintainer's machine is refused, over data nobody can edit out
|
|
by hand.
|
|
|
|
## What the gate cannot do
|
|
|
|
- It does not scan history. Rewriting that is a deliberate, separate act.
|
|
- It knows only what is on *this* machine. A name nobody has ever typed into
|
|
the group is a name it cannot recognise.
|
|
- It does not stop the bot's own logs holding display names —
|
|
`journalctl --user -u signal-bot` does, and `--vacuum-time=` is how that is
|
|
emptied.
|