mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-10 18:17:02 +01:00
feat: refuse to commit personal data, learned from outside the repository
The leak this repository had was not carelessness with a secret. It was somebody reaching for a real name because it was the example in front of them -- a fixture built from a live database, an envelope pasted out of a running container, a help text written while looking at the actual group. Three different people-shaped mistakes, none of which a rule about "be careful" would have stopped. So it is a gate. The hard part is that the thing being checked must not contain the list of what to refuse, or the check *is* the leak. It reads the names at run time from outside: the photo pack's manifest, the message database, the local .env. It never prints the word it matched, because this output reaches CI logs. On a machine with none of that private data it says "shapes only" out loud rather than passing quietly, and CI enforces the half that needs nothing -- phone numbers, uuids and Signal group ids, compared against an allowlist of values invented for the fixtures. It runs on staged files, on the commit message -- where two of the leaks actually travelled -- and over the whole tree in CI. It paid for itself twice before it was wired up. The live test group's id was in tests/test_config.py in two spellings, and truncated in three more files, after four rounds of grepping by hand had pronounced the tree clean -- nobody thinks to grep for base64. Then it refused the commit that introduced it, because DOCS-privacy.md had used a real name as the example of a name. An allowlist line may be scoped to one file, which is how the given names that are also popes stay legal in popes.json and refused in every fixture that has no business carrying them. The threshold is three characters, because one of this group's given names is three characters long. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017w1HZ3cZ7neL3rFdQ6VMBA
This commit is contained in:
parent
a9535f52e4
commit
37a27c6dff
6
.github/workflows/ci.yml
vendored
6
.github/workflows/ci.yml
vendored
@ -36,6 +36,12 @@ jobs:
|
|||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# The name half of this check needs the operator's private data and so
|
||||||
|
# can only run locally; the shape half -- phone numbers, uuids, Signal
|
||||||
|
# group ids -- works anywhere and is what CI enforces on every push.
|
||||||
|
- name: no personal data
|
||||||
|
run: ./scripts/check_personal_data.sh --all
|
||||||
|
|
||||||
- name: ruff format
|
- name: ruff format
|
||||||
run: .venv/bin/ruff format --check .
|
run: .venv/bin/ruff format --check .
|
||||||
|
|
||||||
|
|||||||
87
.personal-data-allowlist
Normal file
87
.personal-data-allowlist
Normal file
@ -0,0 +1,87 @@
|
|||||||
|
# Values this public repository is allowed to contain.
|
||||||
|
#
|
||||||
|
# Everything a real person could be identified by is refused by
|
||||||
|
# scripts/check_personal_data.sh, which learns the names from OUTSIDE this
|
||||||
|
# repository (the photo pack, the message database, .env) so that the list of
|
||||||
|
# real names is never in it. What is left is shapes -- phone numbers, uuids,
|
||||||
|
# Signal group ids -- and those need an explicit list of the invented ones the
|
||||||
|
# documents and fixtures are built from. That list is this file.
|
||||||
|
#
|
||||||
|
# One value per line, lower-cased. Add a value here only after checking it
|
||||||
|
# belongs to nobody.
|
||||||
|
|
||||||
|
# Phone numbers. The 555 range is reserved for fiction; the two below are
|
||||||
|
# structurally valid and deliberately not anybody's.
|
||||||
|
+15550000000
|
||||||
|
+48123456789
|
||||||
|
+420111222333
|
||||||
|
+48000000000
|
||||||
|
+48111222333
|
||||||
|
|
||||||
|
# UUIDs used as fixtures.
|
||||||
|
b1c2d3e4-0000-4a00-8a00-000000000001
|
||||||
|
b1c2d3e4-0000-4a00-8a00-000000000002
|
||||||
|
|
||||||
|
# Group ids invented for the offline end-to-end run and the unit tests. Each
|
||||||
|
# one base64-decodes to a sentence saying it is a fixture.
|
||||||
|
group.wlrkbexyzghkr05vzldrdc1ncm91cc1pzc0zmi1ienl0zxmhisfd
|
||||||
|
group.zdjgmfkyagxaqzfuy205mwndmxbaqzb6twkxiwvyumxjeuvositfaelrpt0=
|
||||||
|
group.wtjllxdhdgnozwqtz3jvdxatawqtmzitynl0zxmhisfd
|
||||||
|
|
||||||
|
# The author's own handle, which is the repository owner and appears in the
|
||||||
|
# project URL, the User-Agent and every copyright line.
|
||||||
|
kuhy
|
||||||
|
# The sender name POST /_selftest injects. Not a person.
|
||||||
|
self-test
|
||||||
|
|
||||||
|
# Given names that are also popes. Permitted ONLY in the files that carry the
|
||||||
|
# papal list, so the same name in a test fixture or a document is still
|
||||||
|
# refused -- which is exactly how it leaked the first time.
|
||||||
|
signal_bot/popes.json::marek
|
||||||
|
signal_bot/popes.json::wiktor
|
||||||
|
tests/test_versioning.py::wiktor
|
||||||
|
DOCS-changelog.md::wiktor
|
||||||
|
|
||||||
|
# Invented Signal group ids. Every one of these base64-decodes twice to a
|
||||||
|
# sentence saying what it is, which is how you can tell it belongs to nobody:
|
||||||
|
# group.WlRKbExYZGhkR05vWldRdFozSnZkWEF0YVdRdE16SXRZbmwwWlhNaElTRT0=
|
||||||
|
# -> ZTJlLXdhdGNoZWQtZ3JvdXAtaWQtMzItYnl0ZXMhISE= -> e2e-watched-group-id-32-bytes!!!
|
||||||
|
group.WTI5dVptbG5MV2R5YjNWd0xXbGtMVE15TFdKNWRHVnpJU0VoSVNFaElRPT0=
|
||||||
|
group.WlRKbExYZGhkR05vWldRdFozSnZkWEF0YVdRdE16SXRZbmwwWlhNaElTRT0=
|
||||||
|
group.WlRKbExYTmxZMjl1WkMxbmNtOTFjQzFwWkMwek1pMWllWFJsY3lFaElTRT0=
|
||||||
|
group.WlRKbExYUm9hWEprTFdkeWIzVndMV2xrTFRNeUxXSjVkR1Z6SVNFaElRPT0=
|
||||||
|
group.WlRKbExXWnZkWEowYUMxbmNtOTFjQzFwWkMwek1pMWllWFJsY3lFaElTRT0=
|
||||||
|
group.WTI1c1pDMXZibVV0WjNKdmRYQXRhV1F0TXpJdFlubDBaWE1oSVNFaElTRT0=
|
||||||
|
group.WTI5c1pDMXZibVV0WjNKdmRYQXRhV1F0TXpJdFlubDBaWE1oSVNFaElTRT0=
|
||||||
|
group.WTI5c1pDMTBkMjh0WjNKdmRYQXRhV1F0TXpJdFlubDBaWE1oSVNFaElTRT0=
|
||||||
|
group.WTI5c1pDMTBhSEpsWlMxbmNtOTFjQzFwWkMwek1pMWllWFJsY3lFaElTRT0=
|
||||||
|
group.WTI5c1pDMW1iM1Z5TFdkeWIzVndMV2xrTFRNeUxXSjVkR1Z6SVNFaElTRT0=
|
||||||
|
group.YjNSb1pYSXRaM0p2ZFhBdGFXUXRNekl0WW5sMFpYTWhJU0VoSVNFPQ==
|
||||||
|
group.YzJWamIyNWtMV2R5YjNWd0xXbGtMVE15TFdKNWRHVnpJU0VoSVNFaA==
|
||||||
|
group.ZDJGMFkyaGxaQzFuY205MWNDMXBaQzB6TWkxaWVYUmxjeUVoSVNFaElRPT0=
|
||||||
|
group.WlRKbExYZGhkR05vWldRdA==
|
||||||
|
|
||||||
|
# Invented phone numbers used as fixtures. Reserved / non-allocated ranges.
|
||||||
|
+15550000001
|
||||||
|
+15550000002
|
||||||
|
+15550001111
|
||||||
|
+15550001234
|
||||||
|
+15555550123
|
||||||
|
+15559999999
|
||||||
|
+48987654321
|
||||||
|
signal_bot/versioning.py::wiktor
|
||||||
|
|
||||||
|
# Given names that are also popes, permitted only in the files that carry or
|
||||||
|
# assert the papal list. Anywhere else -- a test fixture, a document, a help
|
||||||
|
# text -- they are still refused, which is the case that leaked.
|
||||||
|
signal_bot/popes.json::jan
|
||||||
|
signal_bot/popes.json::marek
|
||||||
|
signal_bot/popes.json::wiktor
|
||||||
|
signal_bot/versioning.py::jan
|
||||||
|
tool/gen_popes.py::jan
|
||||||
|
tool/gen_popes.py::marek
|
||||||
|
tool/gen_popes.py::wiktor
|
||||||
|
tests/test_versioning.py::jan
|
||||||
|
tests/test_changelog.py::jan
|
||||||
|
tests/test_popes_command.py::jan
|
||||||
|
DOCS-changelog.md::jan
|
||||||
@ -49,6 +49,27 @@ repos:
|
|||||||
# database is gitignored, but three binary SQLite files reached the
|
# database is gitignored, but three binary SQLite files reached the
|
||||||
# history anyway before this gate existed, so .gitignore is evidently
|
# history anyway before this gate existed, so .gitignore is evidently
|
||||||
# not the whole defence. Exceptions go in .binary-allowlist.
|
# not the whole defence. Exceptions go in .binary-allowlist.
|
||||||
|
# This repository is public and leaked once: real names, two phone
|
||||||
|
# numbers and a member's Signal uuid reached GitHub and only came off
|
||||||
|
# it by deleting the repository. The checker learns what to refuse from
|
||||||
|
# OUTSIDE the repo -- the photo pack, the message database, .env -- so
|
||||||
|
# the list of real names is never in the thing being checked.
|
||||||
|
- id: personal-data
|
||||||
|
name: no personal data
|
||||||
|
entry: scripts/check_personal_data.sh
|
||||||
|
language: system
|
||||||
|
types: [text]
|
||||||
|
|
||||||
|
# The same check on the commit message, because that is where two of
|
||||||
|
# the leaks actually lived.
|
||||||
|
- id: personal-data-message
|
||||||
|
name: no personal data in the commit message
|
||||||
|
entry: scripts/check_personal_data.sh --message
|
||||||
|
language: system
|
||||||
|
stages: [commit-msg]
|
||||||
|
always_run: true
|
||||||
|
pass_filenames: true
|
||||||
|
|
||||||
- id: no-binaries
|
- id: no-binaries
|
||||||
name: no binary or image files
|
name: no binary or image files
|
||||||
entry: scripts/check_no_binaries.sh
|
entry: scripts/check_no_binaries.sh
|
||||||
|
|||||||
83
DOCS-privacy.md
Normal file
83
DOCS-privacy.md
Normal file
@ -0,0 +1,83 @@
|
|||||||
|
# What is private, and what stops it leaking
|
||||||
|
|
||||||
|
This repository is public. The bot it describes reads a private group chat,
|
||||||
|
posts from a private photo album, and keeps a database of who said how much.
|
||||||
|
None of that may be in here, and on 2026-08-30 some of it was.
|
||||||
|
|
||||||
|
## What leaked, and how
|
||||||
|
|
||||||
|
Real first names, one full name, two phone numbers and a group member's
|
||||||
|
Signal uuid were in the tree *and* in the public history. They arrived one
|
||||||
|
way every time: **somebody reached for a real name because it was the example
|
||||||
|
in front of them** — a test fixture built from a live database dump, an
|
||||||
|
envelope pasted verbatim out of a running container, a help text written
|
||||||
|
while looking at the actual group.
|
||||||
|
|
||||||
|
Fixing it needed the tree scrubbed, all 56 commits rewritten with
|
||||||
|
`git filter-repo`, and then the repository **deleted and recreated**, because
|
||||||
|
GitHub keeps orphaned commits fetchable by their SHA after a force-push and
|
||||||
|
only the repository going away takes them with it. The closed pull requests
|
||||||
|
and the star went too.
|
||||||
|
|
||||||
|
## What is private, and where it lives
|
||||||
|
|
||||||
|
| | Where | Never in git because |
|
||||||
|
|---|---|---|
|
||||||
|
| the photos and who is in them | `~/.local/share/signal-bot/pack/` | it is outside the repo, and `.gitignore` also names `people.json`, `tags.json`, `media/`, `pack/` and every media extension |
|
||||||
|
| the message database | `signal_bot.sqlite3` | `*.sqlite3` and its `-wal`/`-shm` siblings are ignored |
|
||||||
|
| the account's number, groups, inviters | `.env` | `.env` is ignored; `.env.sample` carries the shape and no values |
|
||||||
|
| everyone's names | the database and the pack manifest | nothing reads them into the repo; the bot logs `pack:p1`, never a name |
|
||||||
|
|
||||||
|
All of it is `700`/`600` on disk. The repository never needs any of it: the
|
||||||
|
tests, the documents and the offline end-to-end run use invented people —
|
||||||
|
alfa, beta, gamma, ala, basia, celina — and invented ids that base64-decode
|
||||||
|
twice into a sentence saying what they are.
|
||||||
|
|
||||||
|
## The gate
|
||||||
|
|
||||||
|
`scripts/check_personal_data.sh` refuses a commit that carries personal data.
|
||||||
|
It runs on staged files, on the commit message, and over the whole tree in
|
||||||
|
CI. Two halves, because they fail differently:
|
||||||
|
|
||||||
|
- **Names cannot be listed here** — a file of real names kept in the repo to
|
||||||
|
check against *is* the leak. So the checker learns them at run time from
|
||||||
|
outside: the photo pack's manifest, the message database, and `.env`. It
|
||||||
|
never prints the word it matched, because this output reaches CI logs.
|
||||||
|
- **Shapes can be listed** — a phone number, a uuid and a Signal group id
|
||||||
|
look like themselves anywhere, so those are matched by pattern and compared
|
||||||
|
against `.personal-data-allowlist`, which holds only invented values.
|
||||||
|
|
||||||
|
On a machine with none of that private data (CI, a fresh clone) the name half
|
||||||
|
has nothing to compare against and **says so** rather than passing quietly.
|
||||||
|
|
||||||
|
It found a real leak the first time it was run: the live test group's id, in
|
||||||
|
two spellings, in `tests/test_config.py` — which four rounds of grepping by
|
||||||
|
hand had missed, because nobody thinks to grep for base64.
|
||||||
|
|
||||||
|
## When it fires
|
||||||
|
|
||||||
|
```
|
||||||
|
tests/test_thing.py:12: a name from your private data
|
||||||
|
```
|
||||||
|
|
||||||
|
Use an invented name. `ala`, `basia`, `celina`, `darek` are already in use;
|
||||||
|
so are `+15550000000` and ids that decode to `e2e-watched-group-id-32-bytes!!!`.
|
||||||
|
|
||||||
|
```
|
||||||
|
DOCS-thing.md:40: phone number '+48123456789' is not in .personal-data-allowlist
|
||||||
|
```
|
||||||
|
|
||||||
|
Either it belongs to somebody — replace it — or it is invented, in which case
|
||||||
|
add it to `.personal-data-allowlist` **after checking that it is**. A line
|
||||||
|
there may be scoped to one file (`signal_bot/popes.json::hadrian`), which is how
|
||||||
|
a given name that is also a pope stays refused in every test fixture that has
|
||||||
|
no business carrying it.
|
||||||
|
|
||||||
|
## What the gate cannot do
|
||||||
|
|
||||||
|
- It does not scan history. Rewriting that is a deliberate, separate act.
|
||||||
|
- It knows only what is on *this* machine. A name nobody has ever typed into
|
||||||
|
the group is a name it cannot recognise.
|
||||||
|
- It does not stop the bot's own logs holding display names —
|
||||||
|
`journalctl --user -u signal-bot` does, and `--vacuum-time=` is how that is
|
||||||
|
emptied.
|
||||||
@ -128,7 +128,7 @@ who made the change. Captured live on 2026-08-29:
|
|||||||
"dataMessage": {
|
"dataMessage": {
|
||||||
"timestamp": 1788006851503,
|
"timestamp": 1788006851503,
|
||||||
"message": null,
|
"message": null,
|
||||||
"groupInfo": { "groupId": "Y29uZmlnLWdy/…", "groupName": "testbot", "type": "UPDATE" }
|
"groupInfo": { "groupId": "Y29uZmlnLWdy…", "groupName": "testbot", "type": "UPDATE" }
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"account": "+420111222333"
|
"account": "+420111222333"
|
||||||
|
|||||||
@ -1,7 +1,7 @@
|
|||||||
# signal-bot
|
# signal-bot
|
||||||
|
|
||||||
A Signal group bot: `!cat` / `!dog` / `!gekon` image commands, `!papież`, and
|
A Signal group bot: `!cat` / `!dog` / `!gekon` image commands, `!papież`, and
|
||||||
group message statistics.
|
group message statistics. The group's own data stays out: `DOCS-privacy.md`.
|
||||||
|
|
||||||
Runs on its **own registered Signal number** (or, if you prefer, as a linked
|
Runs on its **own registered Signal number** (or, if you prefer, as a linked
|
||||||
device on an existing account) via
|
device on an existing account) via
|
||||||
|
|||||||
158
scripts/check_personal_data.py
Executable file
158
scripts/check_personal_data.py
Executable file
@ -0,0 +1,158 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
||||||
|
"""Refuse to commit anything that identifies a real person.
|
||||||
|
|
||||||
|
This repository is public and has already leaked once: real first names, a
|
||||||
|
full name, two phone numbers and a group member's Signal uuid reached GitHub
|
||||||
|
and stayed there until the repository had to be deleted and recreated. Every
|
||||||
|
one of them arrived the same way -- somebody (a human once, this program's
|
||||||
|
author the second time) reached for a real name because it was the example
|
||||||
|
in front of them.
|
||||||
|
|
||||||
|
So it is a gate rather than a rule. Two halves, because they fail differently:
|
||||||
|
|
||||||
|
* **Names**, which cannot be listed here. The forbidden words are read at run
|
||||||
|
time from outside the repository -- the photo pack's manifest, the message
|
||||||
|
database, the local .env -- so the check knows them and the repository
|
||||||
|
never does. On a machine with none of those (CI, a fresh clone) this half
|
||||||
|
simply has nothing to compare against and says so.
|
||||||
|
* **Shapes**, which can. A phone number, a uuid and a Signal group id look
|
||||||
|
like themselves wherever they appear, so those are matched by pattern and
|
||||||
|
compared against the placeholders this repository is allowed to contain.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
scripts/check_personal_data.sh --all # every tracked file
|
||||||
|
scripts/check_personal_data.sh a.py b.md # named files
|
||||||
|
|
||||||
|
The shell wrapper is the entry point; it knows how to list what git tracks,
|
||||||
|
which keeps this half free of a subprocess call it would need a suppression
|
||||||
|
for.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Final
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from personal_data_sources import REPO, forbidden
|
||||||
|
|
||||||
|
ALLOWLIST: Final = REPO / ".personal-data-allowlist"
|
||||||
|
|
||||||
|
# Shapes that identify somebody wherever they turn up.
|
||||||
|
PATTERNS: Final[dict[str, re.Pattern[str]]] = {
|
||||||
|
"phone number": re.compile(r"\+\d{9,15}\b"),
|
||||||
|
"uuid": re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b"),
|
||||||
|
"signal group id": re.compile(r"\bgroup\.[A-Za-z0-9+/]{16,}={0,2}"),
|
||||||
|
}
|
||||||
|
|
||||||
|
SKIP_DIRS: Final = frozenset({".git", ".venv", "__pycache__", ".mypy_cache", ".ruff_cache"})
|
||||||
|
SKIP_SUFFIXES: Final = frozenset({".lock", ".bundle"})
|
||||||
|
|
||||||
|
|
||||||
|
def allowed(path: Path | None = None) -> set[str]:
|
||||||
|
"""Literals this repository may contain, lower-cased.
|
||||||
|
|
||||||
|
A plain line permits a value anywhere. A ``some/path::value`` line
|
||||||
|
permits it only in that file, which is how a name that is also a pope in
|
||||||
|
popes.json stays refused in every test fixture and document that has no
|
||||||
|
business carrying it.
|
||||||
|
"""
|
||||||
|
if not ALLOWLIST.exists():
|
||||||
|
return set()
|
||||||
|
permitted: set[str] = set()
|
||||||
|
where = str(path) if path is not None else ""
|
||||||
|
# The allowlist has to be able to name the values it permits, including
|
||||||
|
# the file-scoped ones, so inside it every listed value counts as
|
||||||
|
# permitted. It is the list of exceptions; it cannot be its own exception.
|
||||||
|
inside = path is not None and path.resolve() == ALLOWLIST
|
||||||
|
for raw in ALLOWLIST.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = raw.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
scope, separator, value = line.partition("::")
|
||||||
|
if not separator:
|
||||||
|
permitted.add(line.lower())
|
||||||
|
elif inside or where.endswith(scope.strip()):
|
||||||
|
permitted.add(value.strip().lower())
|
||||||
|
return permitted
|
||||||
|
|
||||||
|
|
||||||
|
def scan(text: str, words: set[str], path: Path | None = None) -> list[tuple[int, str]]:
|
||||||
|
"""Return (line number, what matched) for every hit in one document."""
|
||||||
|
permitted = allowed(path)
|
||||||
|
words = {word for word in words if word not in permitted}
|
||||||
|
hits: list[tuple[int, str]] = []
|
||||||
|
for number, line in enumerate(text.splitlines(), start=1):
|
||||||
|
lowered = line.lower()
|
||||||
|
for word in words:
|
||||||
|
if re.search(rf"(?<![\w]){re.escape(word)}(?![\w])", lowered):
|
||||||
|
# The word itself is never printed: this output can end up in
|
||||||
|
# a CI log, and naming the name there would be the same leak
|
||||||
|
# by a longer route.
|
||||||
|
hits.append((number, "a name from your private data"))
|
||||||
|
break
|
||||||
|
hits.extend(
|
||||||
|
(number, f"{label} {match!r} is not in .personal-data-allowlist")
|
||||||
|
for label, pattern in PATTERNS.items()
|
||||||
|
for match in pattern.findall(line)
|
||||||
|
if match.lower() not in permitted
|
||||||
|
)
|
||||||
|
return hits
|
||||||
|
|
||||||
|
|
||||||
|
def readable(path: Path) -> str | None:
|
||||||
|
"""The file's text, or None when it is not text at all."""
|
||||||
|
if path.suffix in SKIP_SUFFIXES or any(part in SKIP_DIRS for part in path.parts):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return path.read_text(encoding="utf-8")
|
||||||
|
except OSError, UnicodeDecodeError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
"""Check whatever was asked for, and explain any refusal."""
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
|
||||||
|
)
|
||||||
|
parser.add_argument("--files", nargs="*", type=Path, default=[])
|
||||||
|
parser.add_argument("--message", type=Path, default=None, help="a commit message file")
|
||||||
|
arguments = parser.parse_args()
|
||||||
|
|
||||||
|
words, sources = forbidden(allowed())
|
||||||
|
targets = list(arguments.files)
|
||||||
|
if arguments.message is not None:
|
||||||
|
targets.append(arguments.message)
|
||||||
|
|
||||||
|
failures = 0
|
||||||
|
for path in targets:
|
||||||
|
text = readable(path)
|
||||||
|
if text is None:
|
||||||
|
continue
|
||||||
|
for number, reason in scan(text, words, path):
|
||||||
|
sys.stdout.write(f"{path}:{number}: {reason}\n")
|
||||||
|
failures += 1
|
||||||
|
|
||||||
|
if failures:
|
||||||
|
sys.stdout.write(
|
||||||
|
f"\nRefused: {failures} line(s) carry personal data.\n"
|
||||||
|
"This repository is public. Use an invented name, a placeholder number, or\n"
|
||||||
|
"add the value to .personal-data-allowlist if it is genuinely not anybody's.\n"
|
||||||
|
)
|
||||||
|
return 1
|
||||||
|
if not words:
|
||||||
|
# Said out loud rather than passing quietly: on CI and on a fresh
|
||||||
|
# clone the name half of this check has nothing to compare against,
|
||||||
|
# and a gate that silently checks nothing is worse than no gate.
|
||||||
|
sys.stdout.write("no local private data to compare against; shapes only\n")
|
||||||
|
else:
|
||||||
|
sys.stdout.write(f"clean ({len(words)} name(s) from: {', '.join(sources)})\n")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
73
scripts/check_personal_data.sh
Executable file
73
scripts/check_personal_data.sh
Executable file
@ -0,0 +1,73 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# ============================================================================
|
||||||
|
# Refuse to commit anything that identifies a real person.
|
||||||
|
#
|
||||||
|
# The logic is check_personal_data.py; this half exists to list what git
|
||||||
|
# tracks, which keeps the Python free of a subprocess call it would otherwise
|
||||||
|
# need a lint suppression for.
|
||||||
|
#
|
||||||
|
# scripts/check_personal_data.sh --all every tracked file
|
||||||
|
# scripts/check_personal_data.sh a.py b.md named files
|
||||||
|
# scripts/check_personal_data.sh --message .git/COMMIT_EDITMSG
|
||||||
|
# ============================================================================
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
readonly SCRIPT_DIR
|
||||||
|
REPO_ROOT="$(dirname "$SCRIPT_DIR")"
|
||||||
|
readonly REPO_ROOT
|
||||||
|
readonly CHECKER="${SCRIPT_DIR}/check_personal_data.py"
|
||||||
|
|
||||||
|
# The repo's own interpreter when there is one, so the check runs the same
|
||||||
|
# way under pre-commit, in CI and by hand.
|
||||||
|
PYTHON="${PYTHON:-${REPO_ROOT}/.venv/bin/python}"
|
||||||
|
if [[ ! -x "$PYTHON" ]]; then
|
||||||
|
PYTHON="$(command -v python3)"
|
||||||
|
fi
|
||||||
|
readonly PYTHON
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
echo "Usage: $(basename "$0") [--all | --message FILE | FILE...]"
|
||||||
|
exit 0
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
if [[ $# -eq 0 ]]; then
|
||||||
|
usage
|
||||||
|
fi
|
||||||
|
|
||||||
|
case "$1" in
|
||||||
|
-h | --help)
|
||||||
|
usage
|
||||||
|
;;
|
||||||
|
--message)
|
||||||
|
if [[ $# -lt 2 ]]; then
|
||||||
|
echo "Error: --message needs a file" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
"$PYTHON" "$CHECKER" --message "$2"
|
||||||
|
;;
|
||||||
|
--all)
|
||||||
|
local files=()
|
||||||
|
mapfile -t files < <(git -C "$REPO_ROOT" ls-files)
|
||||||
|
if [[ ${#files[@]} -eq 0 ]]; then
|
||||||
|
echo "Error: git tracks no files here" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
# Prefixed with the repo root so this works from any directory.
|
||||||
|
local paths=()
|
||||||
|
local file
|
||||||
|
for file in "${files[@]}"; do
|
||||||
|
paths+=("${REPO_ROOT}/${file}")
|
||||||
|
done
|
||||||
|
"$PYTHON" "$CHECKER" --files "${paths[@]}"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
"$PYTHON" "$CHECKER" --files "$@"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
155
scripts/personal_data_sources.py
Executable file
155
scripts/personal_data_sources.py
Executable file
@ -0,0 +1,155 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
||||||
|
"""Where the forbidden words come from -- all of it outside this repository.
|
||||||
|
|
||||||
|
Split out of check_personal_data.py at the 250-line cap, and the seam is the
|
||||||
|
honest one: this half knows the private data, that half knows nothing but how
|
||||||
|
to search. A list of real names kept *in* the repository to check against
|
||||||
|
would be the leak the check exists to prevent, so nothing here is ever
|
||||||
|
written down -- it is read at run time from the photo pack, the message
|
||||||
|
database and the local .env, and on a machine with none of those it simply
|
||||||
|
comes back empty.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Final
|
||||||
|
|
||||||
|
REPO: Final = Path(__file__).resolve().parent.parent
|
||||||
|
|
||||||
|
# Three, because real first names are three letters long and one of this
|
||||||
|
# group's is. Two would match inside half the words in the repository; three
|
||||||
|
# needs a handful of allowlist entries for the given names that are also
|
||||||
|
# popes, which is a cost worth paying to catch a real one.
|
||||||
|
MIN_NAME: Final = 3
|
||||||
|
|
||||||
|
|
||||||
|
def _pack_names(directory: Path) -> set[str]:
|
||||||
|
"""Every trigger word in a photo pack manifest -- i.e. every name."""
|
||||||
|
manifest = directory / "people.json"
|
||||||
|
if not manifest.is_file():
|
||||||
|
return set()
|
||||||
|
try:
|
||||||
|
document = json.loads(manifest.read_text(encoding="utf-8"))
|
||||||
|
except OSError, json.JSONDecodeError:
|
||||||
|
return set()
|
||||||
|
entries = [*document.get("people", []), *document.get("combos", [])]
|
||||||
|
return {
|
||||||
|
str(trigger).lstrip("!").strip()
|
||||||
|
for entry in entries
|
||||||
|
for trigger in entry.get("triggers", [])
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _database_names(path: Path) -> set[str]:
|
||||||
|
"""Display names of everybody the bot has ever seen speak."""
|
||||||
|
if not path.is_file():
|
||||||
|
return set()
|
||||||
|
try:
|
||||||
|
connection = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
||||||
|
except sqlite3.Error:
|
||||||
|
return set()
|
||||||
|
# Literal statements, not a table name interpolated into one: a
|
||||||
|
# formatted query is a SQL-injection shape even when every input is a
|
||||||
|
# constant, and this repository takes no lint suppressions.
|
||||||
|
queries = (
|
||||||
|
"SELECT DISTINCT display_name FROM message_counts",
|
||||||
|
"SELECT DISTINCT display_name FROM group_people",
|
||||||
|
"SELECT DISTINCT display_name FROM message_events",
|
||||||
|
)
|
||||||
|
found: set[str] = set()
|
||||||
|
with connection:
|
||||||
|
for query in queries:
|
||||||
|
try:
|
||||||
|
rows = connection.execute(query)
|
||||||
|
except sqlite3.Error:
|
||||||
|
continue
|
||||||
|
found.update(str(row[0]) for row in rows if row[0])
|
||||||
|
connection.close()
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def _database_groups(path: Path) -> set[str]:
|
||||||
|
"""Every real group id the bot has seen, in both spellings.
|
||||||
|
|
||||||
|
Both, because they are the same group written two ways: /v1/groups
|
||||||
|
reports "group.<base64>" and an inbound envelope carries the id that
|
||||||
|
base64 decodes to. A check that knew only the first would have missed the
|
||||||
|
one that was sitting in tests/test_config.py.
|
||||||
|
"""
|
||||||
|
if not path.is_file():
|
||||||
|
return set()
|
||||||
|
try:
|
||||||
|
connection = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
||||||
|
except sqlite3.Error:
|
||||||
|
return set()
|
||||||
|
queries = (
|
||||||
|
"SELECT DISTINCT group_id FROM group_meta",
|
||||||
|
"SELECT DISTINCT group_id FROM message_events",
|
||||||
|
)
|
||||||
|
found: set[str] = set()
|
||||||
|
with connection:
|
||||||
|
for query in queries:
|
||||||
|
try:
|
||||||
|
rows = connection.execute(query)
|
||||||
|
except sqlite3.Error:
|
||||||
|
continue
|
||||||
|
found.update(str(row[0]) for row in rows if row[0])
|
||||||
|
connection.close()
|
||||||
|
inner: set[str] = set()
|
||||||
|
for group in found:
|
||||||
|
payload = group.removeprefix("group.")
|
||||||
|
try:
|
||||||
|
inner.add(base64.b64decode(payload, validate=True).decode("utf-8"))
|
||||||
|
except ValueError, UnicodeDecodeError:
|
||||||
|
continue
|
||||||
|
return found | inner
|
||||||
|
|
||||||
|
|
||||||
|
def _env_values(path: Path) -> set[str]:
|
||||||
|
"""Phone numbers, group ids and inviters from the deployment's .env."""
|
||||||
|
if not path.is_file():
|
||||||
|
return set()
|
||||||
|
interesting = {"PHONE_NUMBER", "GROUP_IDS", "GROUP_ID", "TRUSTED_INVITERS"}
|
||||||
|
found: set[str] = set()
|
||||||
|
for line in path.read_text(encoding="utf-8").splitlines():
|
||||||
|
key, _, value = line.partition("=")
|
||||||
|
if key.strip() in interesting:
|
||||||
|
found.update(part.strip() for part in value.split(",") if part.strip())
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def forbidden(permitted: set[str]) -> tuple[set[str], list[str]]:
|
||||||
|
"""Every literal that must not be committed, and where they came from.
|
||||||
|
|
||||||
|
The sources are all outside the repository on purpose. A list of real
|
||||||
|
names kept *in* the repository to check against would be the leak it is
|
||||||
|
meant to prevent.
|
||||||
|
"""
|
||||||
|
pack = os.environ.get("PHOTO_PACK_DIR", "").strip()
|
||||||
|
directory = (
|
||||||
|
Path(pack)
|
||||||
|
if pack
|
||||||
|
else Path(os.environ.get("XDG_DATA_HOME", Path.home() / ".local" / "share"))
|
||||||
|
/ "signal-bot"
|
||||||
|
/ "pack"
|
||||||
|
)
|
||||||
|
sources: list[str] = []
|
||||||
|
words: set[str] = set()
|
||||||
|
for label, found in (
|
||||||
|
(f"pack manifest ({directory})", _pack_names(directory)),
|
||||||
|
("message database", _database_names(REPO / "signal_bot.sqlite3")),
|
||||||
|
("group ids", _database_groups(REPO / "signal_bot.sqlite3")),
|
||||||
|
(".env", _env_values(REPO / ".env")),
|
||||||
|
):
|
||||||
|
if found:
|
||||||
|
sources.append(label)
|
||||||
|
words |= found
|
||||||
|
return {
|
||||||
|
word.lower() for word in words if len(word) >= MIN_NAME and word.lower() not in permitted
|
||||||
|
}, sources
|
||||||
@ -33,7 +33,7 @@ def group_id_aliases(group_id: str) -> frozenset[str]:
|
|||||||
the *internal* id -- which is exactly what that base64 decodes to. The
|
the *internal* id -- which is exactly what that base64 decodes to. The
|
||||||
two never compare equal, so a bot configured from the API's own output
|
two never compare equal, so a bot configured from the API's own output
|
||||||
silently ignores every message in its own group. Verified live on
|
silently ignores every message in its own group. Verified live on
|
||||||
2026-08-29: ``group.WTI5dVptbG...`` decodes to ``Y29uZmlnLWdy/...``, the id
|
2026-08-29: ``group.WTI5dVptbG5...`` decodes to ``Y29uZmlnLWdy...``, the id
|
||||||
the same endpoint reports as ``internal_id``.
|
the same endpoint reports as ``internal_id``.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
|
|||||||
@ -125,7 +125,7 @@ def default_dir() -> tuple[Path, bool]:
|
|||||||
def options_for(pack: Pack, token: str) -> tuple[frozenset[str], ...] | None:
|
def options_for(pack: Pack, token: str) -> tuple[frozenset[str], ...] | None:
|
||||||
"""What one typed name could mean, or None when it means nothing.
|
"""What one typed name could mean, or None when it means nothing.
|
||||||
|
|
||||||
Accepts the bare spelling as well as the "!" one, because "!celina marek"
|
Accepts the bare spelling as well as the "!" one, because "!celina basia"
|
||||||
is how people actually write it, and refusing the second half over a
|
is how people actually write it, and refusing the second half over a
|
||||||
missing punctuation mark would be a rule nobody can see.
|
missing punctuation mark would be a rule nobody can see.
|
||||||
"""
|
"""
|
||||||
|
|||||||
@ -9,7 +9,7 @@ from signal_bot.stats.render_day import MAX_LINES, DayView, render_day, summary_
|
|||||||
# The real group, so the rendered example is a case that can actually occur.
|
# The real group, so the rendered example is a case that can actually occur.
|
||||||
GROUP = [
|
GROUP = [
|
||||||
("Ala", 76, 12),
|
("Ala", 76, 12),
|
||||||
("Jan Nowak", 42, 3),
|
("Tomasz Nowak", 42, 3),
|
||||||
("kuhy", 38, 0),
|
("kuhy", 38, 0),
|
||||||
("Basia", 33, 1),
|
("Basia", 33, 1),
|
||||||
("Franciszek Kowalczyk", 16, 0),
|
("Franciszek Kowalczyk", 16, 0),
|
||||||
@ -44,7 +44,7 @@ def test_the_nightly_block_for_a_real_day() -> None:
|
|||||||
assert rendered == (
|
assert rendered == (
|
||||||
"📊 Wtorek, 1 września — 220 wiadomości (16 📷)\n"
|
"📊 Wtorek, 1 września — 220 wiadomości (16 📷)\n"
|
||||||
"👑 Ala — 76 (12 📷)\n"
|
"👑 Ala — 76 (12 📷)\n"
|
||||||
"2. Jan Nowak — 42 (3 📷)\n"
|
"2. Tomasz Nowak — 42 (3 📷)\n"
|
||||||
"3. kuhy — 38\n"
|
"3. kuhy — 38\n"
|
||||||
"4. Basia — 33 (1 📷)\n"
|
"4. Basia — 33 (1 📷)\n"
|
||||||
"5. Franciszek Kowalczyk — 16\n"
|
"5. Franciszek Kowalczyk — 16\n"
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user