mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-10 19:27:20 +01:00
Krzysztof kuhy Rudnicki
405541be9b
One row per command attempt, not two aggregates written only on success. command_uses holds last-use only and daily_uses counts successes; both are rate-limiter state and both are written by store.record_use, which the dispatcher calls only when a handler returns cleanly. A gate refusal and a crash returned before it, so a command that failed every time looked exactly like one nobody ran, and nothing knew which command a counted message was. command_events now takes every attempt with its outcome, its duration and the item it served; upstream_calls takes every outbound request made while serving one. The two writes stay separate on purpose -- recording a refusal as a use is what slid the cooldown window forward every time somebody was told to wait -- and test_dispatcher_outcomes asserts last_used does not move. Backfilled from journald joined against message_events, with daily_uses filling the residual so the two sources cannot double-count. Two things were wrong on the first attempt and both were found by measuring rather than by reasoning: the window has to be symmetric, because sent_at_ms comes from the sender's phone and the journal stamp from this host (194 of 299 lines landed *before* their own message, by up to 3.2 s); and matching has to be a global closest-pair assignment, because per-row greedy cannot separate two commands seconds apart. Together those took the dropped rows from 137 to zero, and recovered 418 invocations back to 29 August. Also: !analityka in the group, twelve Polish lines, naming at most three people and only ones whose display name has been seen; an Analytics tab with Nivo charts; and continuous latency measurement, which answers the open question in DOCS-latency.md -- a text send is 235 ms and an image send about 2.2 s, so the upload really is where the time goes. EPHEMERAL_TABLES was doing two unrelated jobs and is now two tuples: what a schema bump rebuilds, and what leaving a group clears. daily_uses left the first (it is a backfill source now) and stayed in the second, and _drop_reshaped took over the repair a bump used to do for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T4QTp3h5Q3MvfrwGV8xqPf
117 lines
6.3 KiB
Markdown
117 lines
6.3 KiB
Markdown
# What is private, and what stops it leaking
|
|
|
|
This repository is public. The bot it describes reads a private group chat,
|
|
posts from a private photo album, and keeps a database of who said how much.
|
|
None of that may be in here, and on 2026-08-30 some of it was.
|
|
|
|
## What leaked, and how
|
|
|
|
Real first names, one full name, two phone numbers and a group member's
|
|
Signal uuid were in the tree *and* in the public history. They arrived one
|
|
way every time: **somebody reached for a real name because it was the example
|
|
in front of them** — a test fixture built from a live database dump, an
|
|
envelope pasted verbatim out of a running container, a help text written
|
|
while looking at the actual group.
|
|
|
|
Fixing it needed the tree scrubbed, all 56 commits rewritten with
|
|
`git filter-repo`, and then the repository **deleted and recreated**, because
|
|
GitHub keeps orphaned commits fetchable by their SHA after a force-push and
|
|
only the repository going away takes them with it. The closed pull requests
|
|
and the star went too.
|
|
|
|
## What is private, and where it lives
|
|
|
|
| | Where | Never in git because |
|
|
|---|---|---|
|
|
| the photos and who is in them | `~/.local/share/signal-bot/pack/` | it is outside the repo, and `.gitignore` also names `people.json`, `tags.json`, `media/`, `pack/` and every media extension |
|
|
| the message database | `signal_bot.sqlite3` | `*.sqlite3` and its `-wal`/`-shm` siblings are ignored |
|
|
| the account's number, groups, inviters | `.env` | `.env` is ignored; `.env.sample` carries the shape and no values |
|
|
| everyone's names | the database and the pack manifest | nothing reads them into the repo; the bot logs `pack:p1`, never a name |
|
|
| every command anybody has run | `command_events` and `upstream_calls` in the database | the database is ignored, and the columns are chosen so there is nothing to leak: a resolved pack key or `+args` instead of the words typed, an exception's *class* instead of its message, a hostname instead of a URL, and a public identifier instead of a title. See `DOCS-analytics.md`. |
|
|
| what people typed after a `!` the bot did not know | `unknown_commands` in the database | the database is ignored; the log line names neither the word nor the message, and the export the management page writes goes to the state directory, never into this tree |
|
|
|
|
All of it is `700`/`600` on disk. The repository never needs any of it: the
|
|
tests, the documents and the offline end-to-end run use invented people —
|
|
alfa, beta, gamma, ala, basia, celina — and invented ids that base64-decode
|
|
twice into a sentence saying what they are.
|
|
|
|
## The gate
|
|
|
|
`scripts/check_personal_data.sh` refuses a commit that carries personal data.
|
|
It runs on staged files, on the commit message, and over the whole tree in
|
|
CI. Two halves, because they fail differently:
|
|
|
|
- **Names cannot be listed here** — a file of real names kept in the repo to
|
|
check against *is* the leak. So the checker learns them at run time from
|
|
outside: the photo pack's manifest, the message database, and `.env`. It
|
|
never prints the word it matched, because this output reaches CI logs.
|
|
- **Shapes can be listed** — a phone number, a uuid and a Signal group id
|
|
look like themselves anywhere, so those are matched by pattern and compared
|
|
against `.personal-data-allowlist`, which holds only invented values.
|
|
|
|
On a machine with none of that private data (CI, a fresh clone) the name half
|
|
has nothing to compare against and **says so** rather than passing quietly.
|
|
|
|
It found a real leak the first time it was run: the live test group's id, in
|
|
two spellings, in `tests/test_config.py` — which four rounds of grepping by
|
|
hand had missed, because nobody thinks to grep for base64.
|
|
|
|
## When it fires
|
|
|
|
```
|
|
tests/test_thing.py:12: a name from your private data
|
|
```
|
|
|
|
Use an invented name. `ala`, `basia`, `celina`, `darek` are already in use;
|
|
so are `+15550000000` and ids that decode to `e2e-watched-group-id-32-bytes!!!`.
|
|
|
|
```
|
|
DOCS-thing.md:40: phone number '+48123456789' is not in .personal-data-allowlist
|
|
```
|
|
|
|
Either it belongs to somebody — replace it — or it is invented, in which case
|
|
add it to `.personal-data-allowlist` **after checking that it is**. A line
|
|
there may be scoped to one file (`signal_bot/popes.json::hadrian`), which is how
|
|
a given name that is also a pope stays refused in every test fixture that has
|
|
no business carrying it.
|
|
|
|
## Why no command commits the words it posts
|
|
|
|
Every content command posts what it just fetched. `commons.caption_for` says
|
|
why that matters: *the text is fetched at run time and posted to the group;
|
|
none of it is ever written into this repository, which is why it may be free
|
|
text where `traps.json` may not.*
|
|
|
|
`!wiersz` was written the other way first, and it is worth recording because
|
|
the other way looked entirely reasonable. A pack of several hundred short
|
|
public-domain Polish poems was harvested, filtered for names, and committed —
|
|
and `scripts/check_personal_data.sh` refused it. Three lines carried a real
|
|
group member's given name. After a further filter that discarded 31% of the
|
|
corpus, two still did.
|
|
|
|
The reason is structural rather than a matter of filtering harder. Polish
|
|
verse capitalises **every** line, so a capital at the start of a line carries
|
|
no information — and that is exactly where a name in a poem tends to sit, in a
|
|
dedication (`Basi`), an epigraph credit (`Shakspeare.`) or simply mid-sentence
|
|
across a line break. A filter strict enough to catch those rejects all verse.
|
|
|
|
So the pack holds Wolne Lektury **slugs**, and the poem, its title and its
|
|
author arrive over the wire when somebody types the command. `Maria
|
|
Konopnicka` is posted into the group every week and appears nowhere in git.
|
|
The same shape as `traps.json`'s page ids and `art.json`'s, for the same
|
|
reason: an identifier cannot carry a name, and free text eventually will.
|
|
|
|
The failure mode this avoids is the nasty one — it appears **only locally**.
|
|
CI has no private data to compare against, so CI stays green while every
|
|
commit on the maintainer's machine is refused, over data nobody can edit out
|
|
by hand.
|
|
|
|
## What the gate cannot do
|
|
|
|
- It does not scan history. Rewriting that is a deliberate, separate act.
|
|
- It knows only what is on *this* machine. A name nobody has ever typed into
|
|
the group is a name it cannot recognise.
|
|
- It does not stop the bot's own logs holding display names —
|
|
`journalctl --user -u signal-bot` does, and `--vacuum-time=` is how that is
|
|
emptied.
|