kuhyx/signal-bot
1
0
Fork 0
mirror of https://github.com/kuhyx/signal-bot.git synced 2026-11-10 18:17:02 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

feat: two commands that answer with somebody else's work

Browse Source 
!sztuka posts a work of art, skewed 60% Polish; !wiersz posts one short
public-domain Polish poem, whole. Both follow the rule the rest of the bot
already keeps: the repository commits identifiers, and the words arrive over
the wire when somebody asks.

!wiersz was written the other way first, and the gate caught it. A pack of
short public-domain poems was harvested, name-filtered and committed, and
check_personal_data.sh refused it: three lines carried a real group member's
given name, and two still did after a filter that had discarded 31% of the
corpus. That is structural, not a tuning problem -- Polish verse capitalises
every line, so a line-initial capital carries no signal, and that is exactly
where a name sits. The pack now holds Wolne Lektury slugs; the poem, its
title and its author are fetched at post time and appear nowhere in git.
It is also the better version: with no name filter there are no false
positives, so the pool is larger and the attribution is a full name again.

The gate then refused the art category list too, over a painter whose given
name is somebody's. That is the popes.json case exactly, so it takes the
same answer: a file-scoped allowlist entry, still refused everywhere else.

The Commons harvest is now three modules by what they are, not by where the
line count fell: commons_api is the conversation with Commons, commons_harvest
turns categories into a pool of licensed page ids, commons_cache makes an
interrupted run resumable. Along the way it gained licences in the same
request that lists a category (the second pass was 1,500 requests at 2.3s
each), maxlag instead of a guessed sleep, 5xx retries because one 504 used to
destroy a whole run, and Wikidata for works by creators of Polish
citizenship: the Polish pool grew 15,221 -> 20,092.

Batching categories through CirrusSearch looked like a further 6x and was
measured out again: 2,404 files where the authoritative enumeration returns
2,429, a different ~1% missing on every run.

The offline run is 19s where it was four minutes, almost all of it the
deafness gate waiting out three thirty-second liveness probes to prove
something provable in three; the tick is configuration now, and e2e enforces
a 40s budget on itself so that never silently comes back.
This commit is contained in:
Krzysztof Rudnicki 2026-08-31 21:32:07 +02:00
parent 08e72a5173
commit 08599e6056
41 changed files with 82655 additions and 216 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.env.sample
View File

@ -59,6 +59,10 @@ CAT_API_URL=
DOG_API_URL=
GECKO_API_URL=
COMMONS_API_URL=
# Template; {slug} is substituted per poem.
POEM_TEXT_URL=
# Seconds between liveness probes; lower only for tests.
LIVENESS_TICK_SECONDS=
# Optional. Where !papież keeps the 267 papal portraits it has downloaded.
# Default: $CACHE_DIRECTORY/portraits under systemd (the unit sets

4
.gitignore vendored
View File

@ -200,3 +200,7 @@ bottles/
node_modules/
web/coverage/
*.tsbuildinfo
# Disposable resume cache for tool/gen_art.py; see commons_api.load_cache.
tool/.commons_cache.json
tool/.commons_cache.tmp

6
.personal-data-allowlist
View File

@ -85,3 +85,9 @@ tests/test_versioning.py::jan
tests/test_changelog.py::jan
tests/test_popes_command.py::jan
DOCS-changelog.md::jan
# Polish painters whose given name is also somebody's. Permitted ONLY in the
# category list that harvests their work, exactly as a pope's name is
# permitted only in popes.json -- the same name in a test fixture or a
# document is still refused.
tool/art_categories.json::jan

7
DOCS-changelog.md
View File

@ -71,6 +71,13 @@ this repo: describe what changes for them. 0.0.1 shipped a bullet about
`/healthz`, an endpoint bound to 127.0.0.1 that nobody in a Signal group can
reach, and it was rightly called out.
## 0.2.5 — Lucius I
papież nr 22 z 267, pontyfikat 253-254
- dodano !sztuka — losowe dzieło sztuki, częściej polskie niż nie
- dodano !wiersz — losowy krótki wiersz, w całości, z domeny publicznej
## 0.2.4 — Cornelius
papież nr 21 z 267, pontyfikat 251-253

7
DOCS-commands.md
View File

@ -15,6 +15,8 @@ This file is the trigger reference — every spelling that resolves, which
| cat | `!kot` `!koty` `!kots` `!cat` `!cats` `!meow` `!miau` + styled variants |
| dog | `!pies` `!psy` `!dog` `!dogs` `!woof` `!szczek` + styled variants |
| gecko | `!gekon` `!gekony` `!geko` `!gecko` `!geckon` `!geckos` |
| art | `!sztuka` `!sztukę` `!sztuki` `!art` |
| poem | `!wiersz` `!wiersze` `!wiersza` `!wierszyk` `!poem` `!poems` `!poezja` `!poezje` `!poetry` |
| trap | `!pułapka` `!pulapka` `!pułapki` `!pulapki` `!pułapkę` `!pulapke` `!trap` `!traps` `!trapy` `!trapper` `!sidła` `!sidla` `!sidło` `!sidlo` `!wnyk` `!wnyki` `!potrzask` `!zasadzka` `!klatka` `!samołówka` `!samolowka` `!beartrap` `!bear_trap` `!snare` `!snares` `!mousetrap` |
| pope | `!papież` `!papiez` `!papieze` `!papieża` `!papa` `!pope` `!popes` `!pontifex` + styled variants |
| stats_today | `!staty` `!statystyki` `!stats` `!dziś` `!dzis` |
@ -37,7 +39,10 @@ one. It is subtracted from the personal-data gate's word list by
`_repo_vocabulary()`, so it must not be name-shaped; and
`tests/test_nearest.py` fails if it lands within edit distance of a
*different* command's trigger, because the near-miss suggestion below would
then be a coin toss.
then be a coin toss. `!poems` is the one trigger in the table that is only
just legal: it sits two edits from `!pies`, and passes solely because `!poem`
is one edit away and wins the tie. Deleting `!poem` would fail the suite at
`!poems`, nowhere near where the change was made.
## A word nothing claims

31
DOCS-privacy.md
View File

@ -74,6 +74,37 @@ there may be scoped to one file (`signal_bot/popes.json::hadrian`), which is how
a given name that is also a pope stays refused in every test fixture that has
no business carrying it.
## Why no command commits the words it posts
Every content command posts what it just fetched. `commons.caption_for` says
why that matters: *the text is fetched at run time and posted to the group;
none of it is ever written into this repository, which is why it may be free
text where `traps.json` may not.*
`!wiersz` was written the other way first, and it is worth recording because
the other way looked entirely reasonable. A pack of several hundred short
public-domain Polish poems was harvested, filtered for names, and committed —
and `scripts/check_personal_data.sh` refused it. Three lines carried a real
group member's given name. After a further filter that discarded 31% of the
corpus, two still did.
The reason is structural rather than a matter of filtering harder. Polish
verse capitalises **every** line, so a capital at the start of a line carries
no information — and that is exactly where a name in a poem tends to sit, in a
dedication (`Basi`), an epigraph credit (`Shakspeare.`) or simply mid-sentence
across a line break. A filter strict enough to catch those rejects all verse.
So the pack holds Wolne Lektury **slugs**, and the poem, its title and its
author arrive over the wire when somebody types the command. `Maria
Konopnicka` is posted into the group every week and appears nowhere in git.
The same shape as `traps.json`'s page ids and `art.json`'s, for the same
reason: an identifier cannot carry a name, and free text eventually will.
The failure mode this avoids is the nasty one — it appears **only locally**.
CI has no private data to compare against, so CI stays green while every
commit on the maintainer's machine is refused, over data nobody can edit out
by hand.
## What the gate cannot do
- It does not scan history. Rewriting that is a deliberate, separate act.

6
README.md
View File

@ -1,7 +1,7 @@
# signal-bot
A Signal group bot: `!cat` / `!dog` / `!gekon` / `!pułapka` image commands,
`!papież`, and group statistics. The group's own data stays out: `DOCS-privacy.md`.
A Signal group bot: `!cat` / `!dog` / `!gekon` / `!pułapka` / `!sztuka` /
`!wiersz` / `!papież`, and group statistics. Data stays out: `DOCS-privacy.md`.
Runs on its **own registered Signal number** (or, if you prefer, as a linked
device on an existing account) via
@ -56,7 +56,7 @@ expect:
- { kind: image, to: reply }
```
Give each scenario its own `uuid`: one bot process serves them all and the
It enforces a 40-second budget on itself. Give each scenario its own `uuid`: one bot process serves them all and the
cooldown lives in SQLite, so a shared actor leaks state between files.
## Run it for real

17
fake_signal/cli.py
View File

@ -18,7 +18,22 @@ import httpx
from fake_signal.scenario import Scenario, check, load_scenario
SETTLE_SECONDS = 0.35
# Measured, not guessed. A surplus send is *always* detected whatever this is,
# because collect() re-reads after the expected count arrives -- injecting a
# deliberate double-send proved that at every value tried, six runs, all red.
# What this buys is **attribution**: whether the surplus is blamed on the
# scenario that caused it or on the innocent one that follows. Measured with a
# surplus delayed by hand:
#
# surplus at +0.05s / +0.25s blamed correctly at 0.35 and at 0.15
# surplus at +0.50s blamed correctly at 0.35, wrongly at 0.15
#
# 0.15 halves the replay (35s -> 19s, against a 40s budget) and only
# misattributes a surplus arriving half a second late, which took an explicit
# asyncio.sleep to construct -- a real double-send comes from the same handler
# microseconds apart. Raise it with --settle if a slow machine starts blaming
# the wrong scenario; that symptom is what this number is for.
SETTLE_SECONDS = 0.15
# How long a scenario may wait for the sends it expects before giving up.
#
# A single fixed settle cannot cover both a reply that is a dictionary lookup

17
fake_signal/routes_fixtures.py
View File

@ -44,6 +44,23 @@ def build_fixture_router() -> APIRouter:
}
return {"query": {"pages": [page]}}
# Wolne Lektury serves the poem as plain text in a fixed shape: author,
# blank, title, the poem, then a footer behind a lone rule. The slug is
# echoed into the title so a scenario proves the bot asked for the poem
# it drew rather than a hard-coded one. The public-domain sentence is
# here because poem_command refuses anything without it.
@router.get("/poem-text")
async def poem_text(request: Request) -> Response:
slug = request.query_params.get("slug", "")
body = (
f"Adam Testowy\n\n{slug}\n\n\n"
"Pierwsza linijka wiersza,\n"
"druga linijka wiersza.\n\n"
"-----\n"
"Ten utwór jest w domenie publicznej.\n"
)
return Response(content=body, media_type="text/plain; charset=utf-8")
@router.get("/image.png")
async def image() -> Response:
return Response(content=PIXEL_PNG, media_type="image/png")

5
fake_signal/scenarios/art_command.yaml Normal file
View File

@ -0,0 +1,5 @@
name: "!sztuka posts a work of art, captioned with what it is"
inject:
- { text: "!sztuka", uuid: uuid-gallery, name: Gallery }
expect:
- { kind: image, to: reply }

5
fake_signal/scenarios/poem_command.yaml Normal file
View File

@ -0,0 +1,5 @@
name: "!wiersz answers with the poem itself, no attachment and no network"
inject:
- { text: "!wiersz", uuid: uuid-poet, name: Poet }
expect:
- { kind: text, to: reply }

79
scripts/e2e.sh
View File

@ -98,6 +98,7 @@ validate_requirements() {
main() {
validate_requirements
TEMP_DIR="$(mktemp -d)"
STARTED_AT="$(date +%s)"
echo "== booting the fake Signal server on :${FAKE_PORT}"
export FAKE_PORT
@ -138,10 +139,12 @@ main() {
DOG_API_URL="${FAKE_URL}/_fixtures/dog-api" \
GECKO_API_URL="${FAKE_URL}/_fixtures/gecko-api" \
COMMONS_API_URL="${FAKE_URL}/_fixtures/commons-api" \
POEM_TEXT_URL="${FAKE_URL}/_fixtures/poem-text?slug={slug}" \
PORTRAIT_CACHE_DIR="${TEMP_DIR}/portraits" \
PHOTO_PACK_DIR="${TEMP_DIR}/pack" \
BIND_PORT="$BOT_PORT" \
SILENCE_SECONDS=0 \
LIVENESS_TICK_SECONDS=1 \
"$PYTHON" -m signal_bot &
BOT_PID=$!
wait_for_url "http://127.0.0.1:${BOT_PORT}/healthz" "bot"
@ -153,6 +156,7 @@ main() {
echo "== replaying scenarios"
"$PYTHON" -m fake_signal.cli \
"$REPO_ROOT"/fake_signal/scenarios/*.yaml \
--settle "${E2E_SETTLE:-0.15}" \
--base-url "$FAKE_URL" \
--group "watched=${WATCHED_GROUP_INTERNAL}" \
--group "reply=${WATCHED_GROUP}" \
@ -166,82 +170,9 @@ main() {
prove_the_add_photo_invariants
prove_the_deafness_gate
enforce_the_time_budget "$STARTED_AT"
}
# The three claims a reply cannot make on its own: nothing was written when
# nothing was stored, nothing kept its metadata, and no second copy survived.
prove_the_add_photo_invariants() {
local pack="${TEMP_DIR}/pack"
echo "== checking every stored file was actually stripped"
"$PYTHON" "$REPO_ROOT/tool/e2e_media.py" --verify-pack "$pack"
echo "== checking a typed label never reached custom.json as a key"
if grep -qE '"(wakacje|piesek)"[[:space:]]*:' "${pack}/custom.json"; then
echo "Error: a typed label became a key in custom.json" >&2
exit 1
fi
echo "== checking !dodaj with no attachment minted nothing"
if grep -q "piesek" "${pack}/custom.json"; then
echo "Error: a mint survived a message that carried no attachment" >&2
exit 1
fi
echo "== checking the refused uploads were not stored"
if grep -q "att-junk" "${pack}/tags.json"; then
echo "Error: a refused file reached tags.json" >&2
exit 1
fi
echo "== checking signal-cli's own copies were deleted"
local left
left="$(curl -sf "${FAKE_URL}/_control/attachments")"
if [[ "$left" == *"att-person"* || "$left" == *"att-custom"* ]]; then
echo "Error: an ingested attachment is still in the container store: ${left}" >&2
exit 1
fi
echo "== checking the adder was recorded for a future !undo_photo"
if [[ ! -s "${pack}/adds.json" ]]; then
echo "Error: adds.json was never written" >&2
exit 1
fi
}
# The gate that did not exist on 2026-08-29, proven against the real process:
# with signal-cli unable to reach Signal, /healthz must go red rather than
# keep answering 200 while nothing arrives.
prove_the_deafness_gate() {
local health="http://127.0.0.1:${BOT_PORT}/healthz"
echo "== cutting the fake's connection to Signal"
curl -sf -X POST "${FAKE_URL}/_control/connection" \
-H 'Content-Type: application/json' -d '{"alive": false}' >/dev/null
local code="" waited=0
while ((waited < 150)); do
code="$(curl -s -o /dev/null -w '%{http_code}' "$health")"
[[ "$code" == "503" ]] && break
sleep 1
waited=$((waited + 1))
done
[[ "$code" == "503" ]] || die "/healthz still answers ${code} with the connection cut"
curl -s "$health" | grep -q "Closed unexpectedly" ||
die "/healthz went red without saying why"
echo "== /healthz turned 503 and named the reason after ${waited}s"
echo "== restoring the connection"
curl -sf -X POST "${FAKE_URL}/_control/connection" \
-H 'Content-Type: application/json' -d '{"alive": true}' >/dev/null
waited=0
while ((waited < 150)); do
code="$(curl -s -o /dev/null -w '%{http_code}' "$health")"
[[ "$code" == "200" ]] && break
sleep 1
waited=$((waited + 1))
done
[[ "$code" == "200" ]] || die "/healthz stayed ${code} after the connection came back"
echo "== /healthz recovered on its own after ${waited}s"
}
main "$@"

101
scripts/e2e_proofs.sh Executable file
View File

@ -0,0 +1,101 @@
#!/bin/bash
# ============================================================================
# The assertions scripts/e2e.sh makes after the scenarios have replayed.
#
# Split out of e2e.sh at the 250-line cap. A filing change, not an interface
# change: these are sourced, so they still run against the same fake server
# and the same bot process the scenarios used.
# ============================================================================
prove_the_add_photo_invariants() {
local pack="${TEMP_DIR}/pack"
echo "== checking every stored file was actually stripped"
"$PYTHON" "$REPO_ROOT/tool/e2e_media.py" --verify-pack "$pack"
echo "== checking a typed label never reached custom.json as a key"
if grep -qE '"(wakacje|piesek)"[[:space:]]*:' "${pack}/custom.json"; then
echo "Error: a typed label became a key in custom.json" >&2
exit 1
fi
echo "== checking !dodaj with no attachment minted nothing"
if grep -q "piesek" "${pack}/custom.json"; then
echo "Error: a mint survived a message that carried no attachment" >&2
exit 1
fi
echo "== checking the refused uploads were not stored"
if grep -q "att-junk" "${pack}/tags.json"; then
echo "Error: a refused file reached tags.json" >&2
exit 1
fi
echo "== checking signal-cli's own copies were deleted"
local left
left="$(curl -sf "${FAKE_URL}/_control/attachments")"
if [[ "$left" == *"att-person"* || "$left" == *"att-custom"* ]]; then
echo "Error: an ingested attachment is still in the container store: ${left}" >&2
exit 1
fi
echo "== checking the adder was recorded for a future !undo_photo"
if [[ ! -s "${pack}/adds.json" ]]; then
echo "Error: adds.json was never written" >&2
exit 1
fi
}
# The gate that did not exist on 2026-08-29, proven against the real process:
# with signal-cli unable to reach Signal, /healthz must go red rather than
# keep answering 200 while nothing arrives.
prove_the_deafness_gate() {
local health="http://127.0.0.1:${BOT_PORT}/healthz"
echo "== cutting the fake's connection to Signal"
curl -sf -X POST "${FAKE_URL}/_control/connection" \
-H 'Content-Type: application/json' -d '{"alive": false}' >/dev/null
local code="" waited=0
while ((waited < 150)); do
code="$(curl -s -o /dev/null -w '%{http_code}' "$health")"
[[ "$code" == "503" ]] && break
sleep 1
waited=$((waited + 1))
done
[[ "$code" == "503" ]] || die "/healthz still answers ${code} with the connection cut"
curl -s "$health" | grep -q "Closed unexpectedly" ||
die "/healthz went red without saying why"
echo "== /healthz turned 503 and named the reason after ${waited}s"
echo "== restoring the connection"
curl -sf -X POST "${FAKE_URL}/_control/connection" \
-H 'Content-Type: application/json' -d '{"alive": true}' >/dev/null
waited=0
while ((waited < 150)); do
code="$(curl -s -o /dev/null -w '%{http_code}' "$health")"
[[ "$code" == "200" ]] && break
sleep 1
waited=$((waited + 1))
done
[[ "$code" == "200" ]] || die "/healthz stayed ${code} after the connection came back"
echo "== /healthz recovered on its own after ${waited}s"
}
# A budget, not a benchmark. This run was four minutes until the liveness
# tick became configurable, and nothing announced the cost -- slower is the
# one regression a green suite never reports. Exceeding it fails the run;
# whoever exceeded it decides whether the check earned its seconds or the
# budget moves. E2E_BUDGET_SECONDS raises it for a loaded machine.
enforce_the_time_budget() {
local started="$1"
local budget="${E2E_BUDGET_SECONDS:-40}"
local elapsed=$(( $(date +%s) - started ))
echo "== end to end took ${elapsed}s (budget ${budget}s)"
if ((elapsed > budget)); then
echo "Error: the offline run took ${elapsed}s, over its ${budget}s budget." >&2
echo " Either the new check earns its seconds and the budget moves," >&2
echo " or something got slower and this is the report of it." >&2
exit 1
fi
}

19
signal_bot/aliases.json
View File

@ -1,5 +1,5 @@
{
"_comment": "Command trigger words, including the styled Unicode variants people actually type. These live as data, not source, because they are data: adding an alias should never mean touching Python.",
"_comment": "Command trigger words, including the styled Unicode variants people actually type. These live as data, not source, because they are data: adding an alias should never mean touching Python. One trap worth knowing before you edit: !poems sits two edits from !pies and passes test_no_two_commands_are_within_reach_of_each_other only because !poem is one edit away and wins the tie. Removing !poem would fail the suite at !poems, which is nowhere near where the change was made.",
"cat": [
"!kot",
"!koty",
@ -58,6 +58,12 @@
"!snares",
"!mousetrap"
],
"art": [
"!sztuka",
"!sztukę",
"!sztuki",
"!art"
],
"pope": [
"!papież",
"!papiez",
@ -69,6 +75,17 @@
"!pontifex",
"!𝗽𝗮𝗽𝗶𝗲𝘇"
],
"poem": [
"!wiersz",
"!wiersze",
"!wiersza",
"!wierszyk",
"!poem",
"!poems",
"!poezja",
"!poezje",
"!poetry"
],
"stats_today": [
"!staty",
"!statystyki",

79211
signal_bot/art.json Normal file
View File

File diff suppressed because it is too large Load Diff

137
signal_bot/art_command.py Normal file
View File

@ -0,0 +1,137 @@
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""The !sztuka command: one work of art, and what it is.
A near-copy of !pułapka, deliberately: same committed pool of Commons page
ids, same single imageinfo request, same live-fetched caption, same absence
of a cache. What ``traps_command`` says about each of those holds here.
The one thing that is new is the skew. The pool is two lists rather than one,
and ``POLISH_SHARE`` per cent of posts come from the Polish one **whatever
the two sizes are** -- the pool is chosen before the work is. That is worth
saying out loud because the other reading is defensible and this is not it:
if the Polish pool holds two thousand works and the world pool twenty
thousand, an individual Polish work is drawn about ten times as often as an
individual foreign one. Skewing the output is the entire point; a merged pool
weighted by size would produce a command that is 90% foreign and technically
correct.
Nothing is cached, for the reason ``traps_command`` gives: a uniform draw
from tens of thousands warms a cache essentially never, so the disk would
fill with pictures nobody asks for twice.
"""
from __future__ import annotations
import json
import secrets
from base64 import b64encode
from pathlib import Path
from typing import TYPE_CHECKING, Final
from signal_bot.animals import load_aliases
from signal_bot.commons import caption_for, imageinfo_params, page_of
from signal_bot.endpoints import THUMB_WIDTH
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
from signal_bot.context import CommandContext
from signal_bot.decorators import Decorator
from signal_bot.http_client import ImageFetcher
from signal_bot.registry import CommandRegistry, Handler
COMMAND_NAME: Final = "art"
DESCRIPTION: Final = "losowe dzieło sztuki"
ART_FILE: Final = Path(__file__).with_name("art.json")
# Per cent of posts drawn from the Polish pool.
POLISH_SHARE: Final = 60
PER_CENT: Final = 100
def load_art(path: Path = ART_FILE) -> tuple[tuple[int, ...], tuple[int, ...]]:
"""Read both pools.
The file holds Commons page ids -- bare integers -- and that is checked
here rather than assumed, for the reason ``traps_command.load_traps``
gives. It matters more for art than for traps: a Commons painting
filename is very often literally somebody's name, so a pool of filenames
would be a list of names committed to a public repository.
Args:
path: Where to read the pools from. Defaults to the shipped file.
Returns:
The Polish pool and the world pool.
Raises:
ValueError: If either pool is empty or holds anything but integers.
"""
document = json.loads(path.read_text(encoding="utf-8"))
pools: list[tuple[int, ...]] = []
for name in ("polish", "world"):
pool = document[name]
if not pool or not all(isinstance(page, int) for page in pool):
msg = f"{path} must hold a non-empty list of Commons page ids under {name!r}"
raise ValueError(msg)
pools.append(tuple(pool))
return pools[0], pools[1]
def random_art(
polish: Sequence[int],
world: Sequence[int],
randbelow: Callable[[int], int] = secrets.randbelow,
) -> int:
"""Return one page id, skewed towards the Polish pool.
secrets rather than random for the reason animals.paged_url gives: the
choice is not security-sensitive, but ruff's ALL rule set flags `random`
and this repository forbids the noqa that would silence it.
``randbelow`` is a parameter because ``secrets.randbelow`` cannot be
seeded, and a test that asserted the 60/40 split by taking real draws
would either flake or need thousands of iterations. Injecting it lets the
test name the branch it means.
"""
pool = polish if randbelow(PER_CENT) < POLISH_SHARE else world
return pool[randbelow(len(pool))]
def art_handler(
fetcher: ImageFetcher,
api_url: str,
pools: tuple[Sequence[int], Sequence[int]] | None = None,
) -> Handler:
"""Build the handler that posts one random work of art."""
polish, world = load_art() if pools is None else (tuple(pools[0]), tuple(pools[1]))
async def handler(ctx: CommandContext) -> None:
# One request, because the caption and the picture have to describe
# the same file; two would be a chance for them to disagree.
document = await fetcher.fetch_json(
api_url,
imageinfo_params(THUMB_WIDTH, page_id=random_art(polish, world), describe=True),
)
page = page_of(document)
image = await fetcher.fetch_image(str(page["imageinfo"][0]["thumburl"]))
await ctx.reply_image(b64encode(image).decode("ascii"), caption_for(page))
return handler
def register_art_command(
registry: CommandRegistry,
fetcher: ImageFetcher,
api_url: str,
*,
decorate: Decorator | None = None,
aliases: dict[str, list[str]] | None = None,
pools: tuple[Sequence[int], Sequence[int]] | None = None,
) -> None:
"""Register !sztuka under every trigger aliases.json gives it."""
table = load_aliases() if aliases is None else aliases
handler = art_handler(fetcher, api_url, pools)
if decorate is not None:
handler = decorate(handler)
registry.register(COMMAND_NAME, table[COMMAND_NAME], handler, DESCRIPTION)

4
signal_bot/bot.py
View File

@ -16,6 +16,7 @@ from typing import TYPE_CHECKING
from signal_bot.animals import load_aliases, register_animal_commands
from signal_bot.announce import announce_all
from signal_bot.art_command import register_art_command
from signal_bot.changelog import load as load_changelog
from signal_bot.decorators import cooldown
from signal_bot.dispatcher import Dispatcher
@ -27,6 +28,7 @@ from signal_bot.pack_command import register_pack_commands
from signal_bot.pack_holder import PackHolder
from signal_bot.pack_load import load as load_pack
from signal_bot.pack_random import register_random_command
from signal_bot.poem_command import register_poem_command
from signal_bot.popes_command import register_pope_command
from signal_bot.portraits import PortraitCache
from signal_bot.registry import CommandRegistry
@ -200,6 +202,8 @@ def build_bot(
)
register_pope_command(registry, portraits, decorate=rate_limit)
register_trap_command(registry, fetcher, config.commons_api_url, decorate=rate_limit)
register_art_command(registry, fetcher, config.commons_api_url, decorate=rate_limit)
register_poem_command(registry, fetcher, config.poem_text_url, decorate=rate_limit)
register_stats_commands(registry, decorate=rate_limit)
# Rate-limited like every other command, and unlike !add_photo, whose
# "NO CAP, NO LIMITS" is a deliberate exception rather than a default.

9
signal_bot/config.py
View File

@ -15,6 +15,7 @@ from signal_bot.endpoints import (
DEFAULT_COMMONS_API_URL,
DEFAULT_DOG_API_URL,
DEFAULT_GECKO_API_URL,
DEFAULT_POEM_TEXT_URL,
)
# Re-exported: every caller has always imported these from config, and the
@ -32,6 +33,7 @@ __all__ = [
"DEFAULT_COMMONS_API_URL",
"DEFAULT_DOG_API_URL",
"DEFAULT_GECKO_API_URL",
"DEFAULT_POEM_TEXT_URL",
"Config",
"ConfigError",
"canonical_group_id",
@ -58,6 +60,11 @@ DEFAULT_LOG_LEVEL: Final = "INFO"
# health. Silence is normal -- a group can be quiet all evening -- so this is
# only the point at which silence becomes worth *asking* about.
DEFAULT_SILENCE_SECONDS: Final = 120
# How often the liveness probe runs. Overridable for the same reason the API
# urls are: scripts/e2e.sh has to prove the deafness gate really fires, and at
# the production tick that assertion costs three ticks to go red plus one to
# recover -- two minutes of a four-minute run spent watching a clock.
DEFAULT_LIVENESS_TICK_SECONDS: Final = 30
_WS_SCHEMES: Final = {"http": "ws", "https": "wss"}
@ -81,6 +88,7 @@ class Config:
dog_api_url: str = DEFAULT_DOG_API_URL
gecko_api_url: str = DEFAULT_GECKO_API_URL
commons_api_url: str = DEFAULT_COMMONS_API_URL
poem_text_url: str = DEFAULT_POEM_TEXT_URL
# Where the /healthz readiness probe binds. Read here rather than in
# __main__ so that .env governs it: __main__ runs before anything has
# loaded a .env file, so an os.getenv() there would silently ignore it.
@ -96,6 +104,7 @@ class Config:
# Lowered in the e2e run so the deafness gate can be proven against the
# real process rather than only in unit tests.
silence_seconds: int = DEFAULT_SILENCE_SECONDS
liveness_tick_seconds: int = DEFAULT_LIVENESS_TICK_SECONDS
# The private photo albums, empty for an installation without any. Only
# the paths are config; everything inside them -- who the people are,
# what fires their photos -- is data this repository never sees.

6
signal_bot/config_env.py
View File

@ -25,6 +25,7 @@ from signal_bot.config import (
DEFAULT_BIND_PORT,
DEFAULT_COOLDOWN_SECONDS,
DEFAULT_DATABASE_PATH,
DEFAULT_LIVENESS_TICK_SECONDS,
DEFAULT_LOG_LEVEL,
DEFAULT_REPORT_HOUR,
DEFAULT_REPORT_MINUTE,
@ -39,6 +40,7 @@ from signal_bot.endpoints import (
DEFAULT_COMMONS_API_URL,
DEFAULT_DOG_API_URL,
DEFAULT_GECKO_API_URL,
DEFAULT_POEM_TEXT_URL,
)
from signal_bot.groups import ConfigError, parse_group_ids
from signal_bot.pack import PEOPLE_FILE, default_dirs
@ -172,10 +174,14 @@ def load_config(env: dict[str, str] | None = None, *, use_dotenv: bool = True) -
dog_api_url=env.get("DOG_API_URL", "").strip() or DEFAULT_DOG_API_URL,
gecko_api_url=env.get("GECKO_API_URL", "").strip() or DEFAULT_GECKO_API_URL,
commons_api_url=env.get("COMMONS_API_URL", "").strip() or DEFAULT_COMMONS_API_URL,
poem_text_url=env.get("POEM_TEXT_URL", "").strip() or DEFAULT_POEM_TEXT_URL,
bind_host=env.get("BIND_HOST", "").strip() or DEFAULT_BIND_HOST,
bind_port=_bind_port(env),
log_level=_log_level(env),
trusted_inviters=parse_trusted_inviters(env),
silence_seconds=_positive_int(env, "SILENCE_SECONDS", DEFAULT_SILENCE_SECONDS),
liveness_tick_seconds=_positive_int(
env, "LIVENESS_TICK_SECONDS", DEFAULT_LIVENESS_TICK_SECONDS
),
photo_pack_dirs=_photo_pack_dirs(),
)

5
signal_bot/endpoints.py
View File

@ -28,6 +28,11 @@ DEFAULT_GECKO_API_URL: Final = (
# redirects; imageinfo hands back the final URL and a thumbnail in one hop,
# which is also what keeps a 4000px original off Signal.
DEFAULT_COMMONS_API_URL: Final = "https://commons.wikimedia.org/w/api.php"
# Wolne Lektury serves each book's plain text at a URL derived from its slug,
# so this is a template rather than an address. Overridable like the rest, so
# scripts/e2e.sh can point !wiersz at the fake and the offline run stays
# offline -- it is the only command whose upstream is not an image API.
DEFAULT_POEM_TEXT_URL: Final = "https://wolnelektury.pl/media/book/txt/{slug}.txt"
# Wide enough to look like a photograph rather than an avatar, small enough
# that the whole 267-portrait cache stays well under 100 MB.
THUMB_WIDTH: Final = 640

3
signal_bot/help_command.py
View File

@ -38,8 +38,9 @@ HEADING: Final = "Komendy"
# Anything missing from here still gets printed, in a block of its own -- see
# the module docstring.
GROUPS: Final[tuple[tuple[str, tuple[str, ...]], ...]] = (
("Obrazki", ("cat", "dog", "gecko", "trap")),
("Obrazki", ("cat", "dog", "gecko", "trap", "art")),
("Papież", ("pope",)),
("Wiersze", ("poem",)),
("Propozycje", ("suggest",)),
(
"Statystyki",

12
signal_bot/http_client.py
View File

@ -82,6 +82,18 @@ class ImageFetcher:
response.raise_for_status()
return response.json()
async def fetch_text(self, url: str) -> str:
"""Fetch one plain-text document.
!wiersz is the only command whose upstream serves neither JSON nor an
image: Wolne Lektury hands back the poem as text/plain. It goes
through this client like everything else, so the offline environment
and the tests serve it the same way they serve a cat.
"""
response = await self._client.get(url)
response.raise_for_status()
return response.text
async def fetch_image(self, image_url: str) -> bytes:
"""Fetch the bytes at an already-resolved image URL.

5
signal_bot/liveness.py
View File

@ -31,10 +31,9 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
TICK_SECONDS: Final = 30.0
# Measured against the real failure: the broken endpoint answered in 3.9s
# once and hung past 20s another time. 15s is above the observed failure
# latency and below TICK_SECONDS, so probes can never pile up.
# latency and below config.liveness_tick_seconds, so probes cannot pile up.
PROBE_TIMEOUT_SECONDS: Final = 15.0
# signal-cli's own reconnect loop backs off in 100..3200ms, so the failure is
# stable within a second. Three probes 30s apart is not "waiting it out", it
@ -131,7 +130,7 @@ class Liveness:
"""
completed = 0
while iterations is None or completed < iterations:
await self._sleep(TICK_SECONDS)
await self._sleep(self._config.liveness_tick_seconds)
await self.check()
completed += 1

159
signal_bot/poem_command.py Normal file
View File

@ -0,0 +1,159 @@
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""The !wiersz command: one short public-domain Polish poem, whole.
Shaped exactly like !pułapka, and for the same reason. The repository commits
a pool of **identifiers** -- Commons page ids there, Wolne Lektury slugs here
-- and the words arrive over the wire at the moment somebody asks. What
``commons.caption_for`` says is the rule the whole bot follows:
The text is fetched at run time and posted to the group; none of it is
ever written into this repository, which is why it may be free text
where traps.json may not.
This command was written the other way first, committing the poems
themselves, and that is worth recording because it looked entirely
reasonable and failed hard. ``scripts/check_personal_data.py`` refused the
pack: three lines carried a real group member's name, and two still did after
a filter that had already discarded 31% of the corpus. The names were at the
start of a line, and a filter cannot reject those -- Polish verse capitalises
every line, so a line-initial capital is the norm and not a signal. Committing
several hundred Polish poems means committing several hundred Polish given
names, and one collision refuses every commit on the maintainer's machine
while CI, which has no private data to compare against, stays green.
Slugs fixed that structurally rather than by degree, and the version that
survived is also the better one: no name filter means no false positives, so
the pool is far larger and poems that were being thrown away for containing
an ordinary word come back.
There is no cap on the rendered message, which is a decision rather than an
oversight. ``commons.CAPTION_LIMIT`` bounds a caption at 220 characters
because a caption is a label on a picture. A poem is the message, truncating
one was ruled out when the length filter was chosen, and the worst case is
bounded anyway by the caps the harvest applied: 24 lines and 1200 characters,
plus a title, an author and a link.
"""
from __future__ import annotations
import json
import secrets
from pathlib import Path
from typing import TYPE_CHECKING, Final
from signal_bot.animals import load_aliases
from signal_bot.wolnelektury import CATALOGUE, parse
if TYPE_CHECKING:
from collections.abc import Sequence
from signal_bot.context import CommandContext
from signal_bot.decorators import Decorator
from signal_bot.http_client import ImageFetcher
from signal_bot.registry import CommandRegistry, Handler
from signal_bot.wolnelektury import Poem
COMMAND_NAME: Final = "poem"
DESCRIPTION: Final = "losowy wiersz"
POEMS_FILE: Final = Path(__file__).with_name("poems.json")
# Whole poems only, never an excerpt: a truncated poem is worse than no poem,
# so these reject at harvest time rather than cutting at post time. They live
# here rather than in the generator because they are the promise the pack
# makes -- every committed slug is a poem this short -- and the tests assert
# the shipped pack against them.
MAX_LINES: Final = 24
MAX_CHARS: Final = 1200
def load_slugs(path: Path = POEMS_FILE) -> tuple[str, ...]:
"""Read the slug pool.
The file holds Wolne Lektury slugs and nothing else, which is checked
here rather than assumed -- the whole privacy argument for this command
rests on the pack carrying no poem text, and this is the guard that keeps
it that way when somebody regenerates it.
Args:
path: Where to read the pool from. Defaults to the shipped file.
Returns:
Every slug in the pool.
Raises:
ValueError: If the pool is empty or holds anything but non-empty
strings.
"""
document = json.loads(path.read_text(encoding="utf-8"))
slugs = document["poems"]
if not slugs or not all(isinstance(slug, str) and slug for slug in slugs):
msg = f"{path} must hold a non-empty list of Wolne Lektury slugs"
raise ValueError(msg)
return tuple(slugs)
def random_slug(slugs: Sequence[str]) -> str:
"""Return one slug, uniformly at random.
No memory of what came before, the same as !pułapka and for the same
reason: a pool this size is not one a group works through, so a
no-repeat window would be bookkeeping that never changes an answer.
secrets rather than random for the reason animals.paged_url gives: the
choice is not security-sensitive, but ruff's ALL rule set flags `random`
and this repository forbids the noqa that would silence it.
"""
return slugs[secrets.randbelow(len(slugs))]
def render(poem: Poem, slug: str) -> str:
"""The message as the group reads it: title, poem, who wrote it, where.
The author's full name is safe to post and would not have been safe to
commit -- see the module docstring. Posting it is the whole point of
fetching it.
"""
return f"{poem.title}\n\n{poem.text}\n\n— {poem.author}\n{CATALOGUE.format(slug=slug)}"
def poem_handler(
fetcher: ImageFetcher,
text_url: str,
slugs: Sequence[str] | None = None,
) -> Handler:
"""Build the handler that posts one random poem.
Raises:
ValueError: If Wolne Lektury served something that is not the poem
the harvest committed. The dispatcher logs and stays silent, the
same as any other upstream failure -- a half-parsed poem is not
something to post.
"""
pool = load_slugs() if slugs is None else tuple(slugs)
async def handler(ctx: CommandContext) -> None:
slug = random_slug(pool)
poem = parse(await fetcher.fetch_text(text_url.format(slug=slug)))
if poem is None:
msg = f"wolnelektury.pl no longer serves a usable poem at {slug!r}"
raise ValueError(msg)
await ctx.reply(render(poem, slug))
return handler
def register_poem_command(
registry: CommandRegistry,
fetcher: ImageFetcher,
text_url: str,
*,
decorate: Decorator | None = None,
aliases: dict[str, list[str]] | None = None,
slugs: Sequence[str] | None = None,
) -> None:
"""Register !wiersz under every trigger aliases.json gives it."""
table = load_aliases() if aliases is None else aliases
handler = poem_handler(fetcher, text_url, slugs)
if decorate is not None:
handler = decorate(handler)
registry.register(COMMAND_NAME, table[COMMAND_NAME], handler, DESCRIPTION)

1185
signal_bot/poems.json Normal file
View File

File diff suppressed because it is too large Load Diff

105
signal_bot/wolnelektury.py Normal file
View File

@ -0,0 +1,105 @@
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Reading Wolne Lektury's plain-text books.
Shared by ``poem_command``, which reads one at the moment somebody types
``!wiersz``, and by ``tool/gen_poems.py``, which reads five thousand to decide
which slugs are worth committing. One parser, because the harvest's idea of a
poem and the handler's have to be the same idea -- a slug committed as "short
enough" that the handler then renders differently would be a bug nothing
catches.
**Nothing this module returns is ever written into the repository**, and that
is the whole design of !wiersz rather than an incidental property. The first
attempt committed the poems themselves, and ``scripts/check_personal_data.py``
refused the pack: Polish verse is full of Polish given names, three of them
collided with a real group member, and two still did after a filter that had
already thrown away 31% of the corpus. The names sat at the start of a line,
which is exactly where a filter cannot look, because Polish verse capitalises
every line. So the pack holds slugs and the text arrives over the wire, which
is what ``commons.caption_for`` already says about every other command:
The text is fetched at run time and posted to the group; none of it is
ever written into this repository, which is why it may be free text
where traps.json may not.
The format is regular enough to parse and irregular enough to check: the
author on the first line, the title on the third, an optional ``tłum.``
credit under that, the poem, and a footer behind a lone rule.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Final
# Where a reader goes for the annotated version. A link, never fetched, so it
# is a constant rather than a configurable endpoint.
CATALOGUE: Final = "https://wolnelektury.pl/katalog/lektura/{slug}/"
# The sentence Wolne Lektury puts in the footer of a public-domain work, and
# does not put in the footer of a CC-licensed one. The corpus is not a licence:
# it hosts living authors too, and a third of it came back non-free.
PUBLIC_DOMAIN: Final = "Ten utwór jest w domenie publicznej."
# The credit line that marks a translation, directly under the title. A poem
# translated into Polish is not a Polish poem.
TRANSLATED: Final = "tłum."
FOOTER: Final = "\n-----"
# author, blank, title -- so the body, or a translator credit, starts here.
HEADER_LINES: Final = 3
@dataclass(frozen=True)
class Poem:
"""One poem as Wolne Lektury serves it."""
title: str
author: str
text: str
@property
def lines(self) -> int:
"""How many lines of verse, blanks not counted.
Blank lines are stanza breaks, and a stanza break is not a line of
poetry -- counting them would make a four-stanza poem look twice as
long as it reads.
"""
return len([line for line in self.text.splitlines() if line.strip()])
def parse(raw: str) -> Poem | None:
"""One book's plain text, or None when it is not a poem we can use.
Returns None rather than raising, and rather than guessing, for three
separate reasons that all end the same way -- this book is skipped:
* it is too short to have the header at all, i.e. not the shape we know;
* it carries a ``tłum.`` credit, so it is a translation;
* it is not in the public domain by its own footer's account.
A guess here would be committed by the harvest or posted to a group by
the handler, so anything unfamiliar is dropped instead.
Args:
raw: The body of a ``media/book/txt/<slug>.txt`` response.
Returns:
The poem, or None if it is unusable.
"""
if PUBLIC_DOMAIN not in raw:
return None
lines = raw.split(FOOTER, maxsplit=1)[0].splitlines()
if len(lines) <= HEADER_LINES:
return None
rest = lines[HEADER_LINES:]
if rest[0].strip().startswith(TRANSLATED):
return None
author = lines[0].strip()
title = lines[2].strip()
text = "\n".join(rest).strip()
if not author or not title or not text:
return None
return Poem(title=title, author=author, text=text)

233
tests/test_art_command.py Normal file
View File

@ -0,0 +1,233 @@
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""The !sztuka command: one work of art, skewed Polish, captioned."""
from __future__ import annotations
import base64
import json
from typing import TYPE_CHECKING
import httpx
import pytest
from signal_bot.art_command import (
ART_FILE,
COMMAND_NAME,
PER_CENT,
POLISH_SHARE,
load_art,
random_art,
register_art_command,
)
from signal_bot.context import CommandContext
from signal_bot.decorators import DISPLAY_MARGIN_SECONDS, cooldown
from signal_bot.http_client import ImageFetcher
from signal_bot.models import parse_envelope
from signal_bot.registry import CommandRefusedError, CommandRegistry
from tests.conftest import WATCHED_GROUP, envelope
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
from pathlib import Path
from signal_bot.config import Config
from signal_bot.store import Store
from signal_bot.transport_fake import FakeTransport
IMAGE = b"art-bytes"
API = "https://commons.example/w/api.php"
ALIASES = {COMMAND_NAME: ["!sztuka", "!sztukę", "!art"]}
POLISH = (1001, 1002, 1003)
WORLD = (2001, 2002, 2003)
POOLS = (POLISH, WORLD)
def _fetcher(seen: list[httpx.URL] | None = None) -> ImageFetcher:
def handler(request: httpx.Request) -> httpx.Response:
if "commons" in str(request.url):
if seen is not None:
seen.append(request.url)
page = {
"title": "File:Bitwa pod Grunwaldem.jpg",
"imageinfo": [
{
"thumburl": "https://cdn/art.jpg",
"descriptionurl": "https://commons.wikimedia.org/wiki/File:Grunwald.jpg",
"extmetadata": {
"ImageDescription": {"value": "Bitwa pod <b>Grunwaldem</b>"},
},
}
],
}
return httpx.Response(200, json={"query": {"pages": [page]}})
return httpx.Response(200, content=IMAGE)
return ImageFetcher(httpx.AsyncClient(transport=httpx.MockTransport(handler)))
def _ctx(transport: FakeTransport, store: Store, config: Config, text: str) -> CommandContext:
message = parse_envelope(envelope(text=text))
assert message is not None
return CommandContext(
message=message,
group_id=WATCHED_GROUP,
transport=transport,
store=store,
config=config,
command_name=COMMAND_NAME,
now=100.0,
day="2026-08-31",
)
@pytest.mark.parametrize("trigger", ["!sztuka", "!sztukę", "!art"])
def test_every_alias_resolves(trigger: str) -> None:
registry = CommandRegistry()
register_art_command(registry, _fetcher(), API, aliases=ALIASES, pools=POOLS)
command = registry.resolve(trigger)
assert command is not None
assert command.name == COMMAND_NAME
def test_the_real_alias_file_carries_the_command() -> None:
registry = CommandRegistry()
register_art_command(registry, _fetcher(), API, pools=POOLS)
# The Polish spelling is the primary one !pomoc prints.
assert registry.resolve("!sztuka") is not None
assert registry.resolve("!art") is not None
async def test_it_posts_an_image_captioned_with_what_the_work_is(
transport: FakeTransport, store: Store, config: Config
) -> None:
registry = CommandRegistry()
register_art_command(registry, _fetcher(), API, aliases=ALIASES, pools=POOLS)
command = registry.resolve("!sztuka")
assert command is not None
await command.handler(_ctx(transport, store, config, "!sztuka"))
sent = transport.sent[0]
assert sent.recipient == WATCHED_GROUP
assert sent.image_base64 is not None
assert base64.b64decode(sent.image_base64) == IMAGE
assert len(transport.sent) == 1
assert sent.caption == (
"Bitwa pod Grunwaldem\nhttps://commons.wikimedia.org/wiki/File:Grunwald.jpg"
)
async def test_it_names_the_file_by_page_id_and_never_by_title(
transport: FakeTransport, store: Store, config: Config
) -> None:
# A Commons painting filename is very often literally somebody's name, so
# this is the assertion that keeps the query -- and the pool -- on ids.
seen: list[httpx.URL] = []
registry = CommandRegistry()
register_art_command(registry, _fetcher(seen), API, aliases=ALIASES, pools=POOLS)
command = registry.resolve("!sztuka")
assert command is not None
await command.handler(_ctx(transport, store, config, "!sztuka"))
assert len(seen) == 1
assert seen[0].params["pageids"] in {str(page) for page in POLISH + WORLD}
assert "titles" not in seen[0].params
async def test_the_cooldown_decorator_applies(
transport: FakeTransport, store: Store, config: Config
) -> None:
registry = CommandRegistry()
register_art_command(
registry,
_fetcher(),
API,
decorate=cooldown(10, "WAIT {seconds}"),
aliases=ALIASES,
pools=POOLS,
)
store.record_use(WATCHED_GROUP, "uuid-alice", COMMAND_NAME, 100.0, "2026-08-31")
command = registry.resolve("!sztuka")
assert command is not None
with pytest.raises(CommandRefusedError):
await command.handler(_ctx(transport, store, config, "!sztuka"))
assert transport.texts == [f"WAIT {10 + DISPLAY_MARGIN_SECONDS}"]
def _stub(first: int, second: int = 0) -> Callable[[int], int]:
"""A randbelow whose first answer picks the pool and second picks the work."""
answers = iter((first, second))
def randbelow(_: int) -> int:
return next(answers)
return randbelow
@pytest.mark.parametrize(
("roll", "expected"),
[(0, POLISH), (POLISH_SHARE - 1, POLISH), (POLISH_SHARE, WORLD), (PER_CENT - 1, WORLD)],
)
def test_the_share_decides_which_pool_is_drawn_from(roll: int, expected: Sequence[int]) -> None:
"""The 60/40 branch, named rather than sampled.
secrets.randbelow cannot be seeded, so asserting the split by taking real
draws would either flake or need thousands of iterations. The chooser is
injectable precisely so this test can say which side it means.
"""
assert random_art(POLISH, WORLD, _stub(roll)) == expected[0]
def test_the_draw_actually_varies() -> None:
"""0.0.3 posted one identical gecko forever; that is what this catches."""
drawn = {random_art(POLISH, WORLD) for _ in range(40)}
assert len(drawn) > 1
def test_both_shipped_pools_are_page_ids_and_nothing_else() -> None:
"""The personal-data invariant, as a unit test.
A filename here would carry artists' and sitters' names into a public
repository, and one of them colliding with a label somebody minted from
the chat wedges check_personal_data.py permanently. Integers cannot.
"""
document = json.loads(ART_FILE.read_text(encoding="utf-8"))
for name in ("polish", "world"):
raw = document[name]
assert raw
assert all(isinstance(page, int) for page in raw)
assert len(set(raw)) == len(raw)
assert raw == sorted(raw)
def test_the_two_shipped_pools_are_disjoint() -> None:
# A work in both pools would be drawn at both sets of odds, which makes
# POLISH_SHARE a number that does not mean what it says.
polish, world = load_art()
assert not set(polish) & set(world)
def test_the_shipped_pools_are_large_enough_to_be_worth_drawing_from() -> None:
# A regeneration that silently truncated one pool would still pass every
# other test here, and would quietly make the skew a draw from nothing.
polish, world = load_art()
assert len(polish) >= 300
assert len(world) >= 300
@pytest.mark.parametrize("broken", [[], ["1001.jpg"], [None]])
@pytest.mark.parametrize("pool", ["polish", "world"])
def test_a_pool_that_is_not_page_ids_is_refused(
tmp_path: Path, pool: str, broken: list[object]
) -> None:
document: dict[str, list[object]] = {"polish": [1001], "world": [2001]}
document[pool] = broken
path = tmp_path / "art.json"
path.write_text(json.dumps(document), encoding="utf-8")
with pytest.raises(ValueError, match="page ids"):
load_art(path)

14
tests/test_bot.py
View File

@ -73,6 +73,20 @@ def test_build_bot_registers_the_animal_commands(
assert bot.dispatcher.registry.resolve("!pies") is not None
def test_build_bot_registers_the_culture_commands(
config: Config, store: Store, transport: FakeTransport, directory: GroupDirectory
) -> None:
"""!sztuka and !wiersz reach the live registry, off the shipped packs.
Both read a committed data file at registration time rather than at the
moment somebody types the trigger, so a missing or malformed pack is a
startup failure. This is the test that would catch one.
"""
bot = _bot(config, store, transport, directory)
assert bot.dispatcher.registry.resolve("!sztuka") is not None
assert bot.dispatcher.registry.resolve("!wiersz") is not None
async def test_listen_dispatches_every_frame(
config: Config, store: Store, transport: FakeTransport, directory: GroupDirectory
) -> None:

7
tests/test_config.py
View File

@ -67,6 +67,13 @@ def test_overrides_are_honoured() -> None:
assert config.cooldown_seconds == 0
def test_the_liveness_tick_is_overridable() -> None:
# scripts/e2e.sh turns this down: at the production tick, proving the
# deafness gate costs three ticks to go red plus one to recover.
assert load_config(dict(BASE_ENV)).liveness_tick_seconds == 30
assert load_config(dict(BASE_ENV) | {"LIVENESS_TICK_SECONDS": "1"}).liveness_tick_seconds == 1
def test_blank_optionals_fall_back_to_defaults() -> None:
env = dict(BASE_ENV) | {"API_BASE_URL": " ", "DATABASE_PATH": "", "REPORT_HOUR": ""}
config = load_config(env)

15
tests/test_fake_server.py
View File

@ -10,6 +10,7 @@ from fastapi.testclient import TestClient
from fake_signal.server import create_app, default_state
from fake_signal.state import PIXEL_PNG, State
from signal_bot.wolnelektury import parse
@pytest.fixture
@ -93,6 +94,20 @@ def test_fixture_apis_mirror_the_real_response_shapes(client: TestClient) -> Non
assert page["imageinfo"][0]["thumburl"].endswith("/_fixtures/image.png")
assert "url" not in page["imageinfo"][0]
assert page["title"] == "File:8-St.jpg"
poem = client.get("/_fixtures/poem-text", params={"slug": "impas"})
# Wolne Lektury serves text/plain in a fixed shape, and poem_command
# refuses anything without the public-domain sentence -- so the fixture
# has to carry it or !wiersz answers nothing and the scenario would fail
# for a reason that looks like the bot's fault.
assert poem.headers["content-type"].startswith("text/plain")
parsed = parse(poem.text)
assert parsed is not None
# The slug is echoed back as the title, so a scenario proves the bot
# asked for the poem it drew rather than a hard-coded one.
assert parsed.title == "impas"
assert parsed.author == "Adam Testowy"
assert "linijka wiersza" in parsed.text
assert client.get("/_fixtures/image.png").content == PIXEL_PNG
# The square URL is deliberately absent: the bot must rewrite it.
assert client.get("/_fixtures/photos/large.png").content == PIXEL_PNG

6
tests/test_help_command.py
View File

@ -9,7 +9,8 @@ import httpx
import pytest
from signal_bot.animals import register_animal_commands
from signal_bot.config import DEFAULT_COMMONS_API_URL
from signal_bot.art_command import register_art_command
from signal_bot.config import DEFAULT_COMMONS_API_URL, DEFAULT_POEM_TEXT_URL
from signal_bot.context import CommandContext
from signal_bot.help_command import (
ALBUM_HEADING,
@ -28,6 +29,7 @@ from signal_bot.pack_command import register_pack_commands
from signal_bot.pack_holder import PackHolder
from signal_bot.pack_load import load as load_pack
from signal_bot.pack_random import register_random_command
from signal_bot.poem_command import register_poem_command
from signal_bot.popes_command import register_pope_command
from signal_bot.portraits import PortraitCache
from signal_bot.registry import CommandRegistry
@ -66,6 +68,8 @@ def full_registry() -> CommandRegistry:
register_animal_commands(registry, fetcher)
register_pope_command(registry, PortraitCache(fetcher, DEFAULT_COMMONS_API_URL))
register_trap_command(registry, fetcher, DEFAULT_COMMONS_API_URL, traps=(1,))
register_art_command(registry, fetcher, DEFAULT_COMMONS_API_URL, pools=((1,), (2,)))
register_poem_command(registry, fetcher, DEFAULT_POEM_TEXT_URL, slugs=("impas",))
register_stats_commands(registry)
register_suggest_command(registry)
register_help_command(registry)

246
tests/test_poem_command.py Normal file
View File

@ -0,0 +1,246 @@
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""The !wiersz command: one short public-domain Polish poem, whole."""
from __future__ import annotations
import json
from typing import TYPE_CHECKING
import httpx
import pytest
from signal_bot.context import CommandContext
from signal_bot.decorators import DISPLAY_MARGIN_SECONDS, cooldown
from signal_bot.http_client import ImageFetcher
from signal_bot.models import parse_envelope
from signal_bot.poem_command import (
COMMAND_NAME,
MAX_CHARS,
MAX_LINES,
POEMS_FILE,
load_slugs,
random_slug,
register_poem_command,
render,
)
from signal_bot.registry import CommandRefusedError, CommandRegistry
from signal_bot.wolnelektury import PUBLIC_DOMAIN, Poem
from tests.conftest import WATCHED_GROUP, envelope
if TYPE_CHECKING:
from pathlib import Path
from signal_bot.config import Config
from signal_bot.store import Store
from signal_bot.transport_fake import FakeTransport
TEXT_URL = "https://lektury.example/txt/{slug}.txt"
ALIASES = {COMMAND_NAME: ["!wiersz", "!wiersze", "!poem"]}
POOL = ("bol-tabuli", "impas", "nikt-nic")
BODY = "Pierwsza linijka wiersza,\ndruga linijka wiersza."
def _fetcher(seen: list[httpx.URL] | None = None, *, body: str = BODY) -> ImageFetcher:
def handler(request: httpx.Request) -> httpx.Response:
if seen is not None:
seen.append(request.url)
raw = f"Maria Konopnicka\n\nA co wam śpiewać\n\n\n{body}\n\n-----\n{PUBLIC_DOMAIN}\n"
return httpx.Response(200, text=raw)
return ImageFetcher(httpx.AsyncClient(transport=httpx.MockTransport(handler)))
def _ctx(transport: FakeTransport, store: Store, config: Config, text: str) -> CommandContext:
message = parse_envelope(envelope(text=text))
assert message is not None
return CommandContext(
message=message,
group_id=WATCHED_GROUP,
transport=transport,
store=store,
config=config,
command_name=COMMAND_NAME,
now=100.0,
day="2026-08-31",
)
@pytest.mark.parametrize("trigger", ["!wiersz", "!wiersze", "!poem"])
def test_every_alias_resolves(trigger: str) -> None:
registry = CommandRegistry()
register_poem_command(registry, _fetcher(), TEXT_URL, aliases=ALIASES, slugs=POOL)
command = registry.resolve(trigger)
assert command is not None
assert command.name == COMMAND_NAME
def test_the_real_alias_file_carries_the_command() -> None:
registry = CommandRegistry()
register_poem_command(registry, _fetcher(), TEXT_URL, slugs=POOL)
# The Polish spelling is the primary one !pomoc prints.
assert registry.resolve("!wiersz") is not None
assert registry.resolve("!poem") is not None
async def test_it_posts_the_poem_its_author_and_where_it_came_from(
transport: FakeTransport, store: Store, config: Config
) -> None:
registry = CommandRegistry()
register_poem_command(registry, _fetcher(), TEXT_URL, aliases=ALIASES, slugs=("impas",))
command = registry.resolve("!wiersz")
assert command is not None
await command.handler(_ctx(transport, store, config, "!wiersz"))
expected = (
f"A co wam śpiewać\n\n{BODY}\n\n— Maria Konopnicka\n"
"https://wolnelektury.pl/katalog/lektura/impas/"
)
assert transport.texts == [expected]
# Text, not an attachment: this is the one content command that posts no
# picture, and sending an empty image would be a silent wrong turn.
assert len(transport.sent) == 1
assert transport.sent[0].image_base64 is None
# ...and it still quotes the asker, like every other reply.
assert transport.sent[0].quote is not None
async def test_the_full_author_name_is_posted_though_it_is_never_committed(
transport: FakeTransport, store: Store, config: Config
) -> None:
"""The point of fetching rather than committing, as an assertion.
"Maria" is a Polish given name. Posting it is fine; committing several
hundred like it is what made check_personal_data.py refuse the first
version of this pack. See the module docstring.
"""
registry = CommandRegistry()
register_poem_command(registry, _fetcher(), TEXT_URL, aliases=ALIASES, slugs=POOL)
command = registry.resolve("!wiersz")
assert command is not None
await command.handler(_ctx(transport, store, config, "!wiersz"))
assert "Maria Konopnicka" in transport.texts[0]
async def test_it_asks_for_the_slug_it_drew(
transport: FakeTransport, store: Store, config: Config
) -> None:
seen: list[httpx.URL] = []
registry = CommandRegistry()
register_poem_command(registry, _fetcher(seen), TEXT_URL, aliases=ALIASES, slugs=("impas",))
command = registry.resolve("!wiersz")
assert command is not None
await command.handler(_ctx(transport, store, config, "!wiersz"))
assert len(seen) == 1
assert str(seen[0]) == "https://lektury.example/txt/impas.txt"
async def test_an_upstream_that_stops_serving_a_poem_is_not_posted(
transport: FakeTransport, store: Store, config: Config
) -> None:
"""A half-parsed poem is not something to put in a group.
The dispatcher turns this into a log line and silence, which is what
every other command already does when its upstream misbehaves.
"""
registry = CommandRegistry()
fetcher = ImageFetcher(
httpx.AsyncClient(transport=httpx.MockTransport(lambda _: httpx.Response(200, text="")))
)
register_poem_command(registry, fetcher, TEXT_URL, aliases=ALIASES, slugs=POOL)
command = registry.resolve("!wiersz")
assert command is not None
with pytest.raises(ValueError, match="no longer serves"):
await command.handler(_ctx(transport, store, config, "!wiersz"))
assert transport.sent == []
async def test_the_cooldown_decorator_applies(
transport: FakeTransport, store: Store, config: Config
) -> None:
registry = CommandRegistry()
register_poem_command(
registry,
_fetcher(),
TEXT_URL,
decorate=cooldown(10, "WAIT {seconds}"),
aliases=ALIASES,
slugs=POOL,
)
store.record_use(WATCHED_GROUP, "uuid-alice", COMMAND_NAME, 100.0, "2026-08-31")
command = registry.resolve("!wiersz")
assert command is not None
with pytest.raises(CommandRefusedError):
await command.handler(_ctx(transport, store, config, "!wiersz"))
assert transport.texts == [f"WAIT {10 + DISPLAY_MARGIN_SECONDS}"]
def test_the_draw_actually_varies() -> None:
"""0.0.3 posted one identical gecko forever; that is what this catches."""
assert len({random_slug(POOL) for _ in range(40)}) > 1
def test_render_puts_the_poem_before_the_attribution() -> None:
# The order is the point: a group reads the poem, then who wrote it.
text = render(Poem(title="Impas", author="Bolesław Leśmian", text=BODY), "impas")
assert text.index("Pierwsza") < text.index("— Bolesław") < text.index("wolnelektury.pl")
def test_the_shipped_pack_holds_slugs_and_no_poetry() -> None:
"""The privacy invariant this whole command is shaped around.
Committing the poems themselves is what check_personal_data.py refused:
three lines carried a real group member's name, and two still did after
a filter that had already discarded 31% of the corpus. A slug cannot
carry a name that way. This is the test that keeps the pack to slugs
when somebody regenerates it.
"""
raw = json.loads(POEMS_FILE.read_text(encoding="utf-8"))["poems"]
assert raw
assert all(isinstance(slug, str) and slug for slug in raw)
# A slug is a lowercase hyphenated handle, never prose: no capitals, no
# whitespace, no punctuation that a line of verse would carry.
assert all(slug == slug.lower() for slug in raw)
assert not any(set(slug) & set(" \n.,;:!?\"'") for slug in raw)
assert len(set(raw)) == len(raw)
assert raw == sorted(raw)
# What this does NOT claim: that no slug contains a word that is also
# somebody's name. "Zosia i jej mopsy" is a Konopnicka poem and its slug
# is zosia-i-jej-mopsy, because a capitalised *first* word is how every
# title starts and cannot be told from a name. That residue is acceptable
# where a pack of poems was not, and the difference is repair: the real
# check is scripts/check_personal_data.sh, pre-commit runs it on every
# commit, and a colliding slug is one line to delete out of 1180. The
# text version wedged the repository with "data nobody can edit out by
# hand"; this is a one-line edit and a regeneration.
def test_the_shipped_pack_is_large_enough_to_be_worth_drawing_from() -> None:
# A regeneration that silently truncated would still pass every other
# test here, and would quietly shrink the command's whole world.
assert len(load_slugs()) >= 300
def test_the_caps_are_the_promise_the_pack_makes() -> None:
# Whole poems only. The caps are enforced at harvest time, so they live
# next to the command rather than in the generator that applies them.
assert MAX_LINES == 24
assert MAX_CHARS == 1200
@pytest.mark.parametrize("broken", [[], ["", "impas"], [None], [3]])
def test_a_pool_that_is_not_slugs_is_refused(tmp_path: Path, broken: list[object]) -> None:
path = tmp_path / "poems.json"
path.write_text(json.dumps({"poems": broken}), encoding="utf-8")
with pytest.raises(ValueError, match="slugs"):
load_slugs(path)

74
tests/test_wolnelektury.py Normal file
View File

@ -0,0 +1,74 @@
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Reading Wolne Lektury's plain text, which both !wiersz and the harvest do."""
from __future__ import annotations
import pytest
from signal_bot.wolnelektury import PUBLIC_DOMAIN, Poem, parse
BODY = "Pierwsza linijka wiersza,\ndruga linijka wiersza."
def _book(
*,
header: str = "Maria Konopnicka\n\nA co wam śpiewać\n",
body: str = BODY,
footer: str = PUBLIC_DOMAIN,
) -> str:
return f"{header}\n\n{body}\n\n\n-----\n{footer}\n"
def test_it_reads_the_author_the_title_and_the_poem() -> None:
poem = parse(_book())
assert poem == Poem(title="A co wam śpiewać", author="Maria Konopnicka", text=BODY)
def test_the_footer_is_not_part_of_the_poem() -> None:
# The rule and everything after it is Wolne Lektury's own boilerplate --
# a licence notice and a funding credit -- and posting it would double
# the length of every reply.
poem = parse(_book())
assert poem is not None
assert "wolnelektury" not in poem.text.lower()
assert PUBLIC_DOMAIN not in poem.text
def test_a_work_that_is_not_public_domain_is_refused() -> None:
"""The corpus is not a licence: it hosts living authors too.
Measured on the real catalogue, a third of it came back non-free, so
this is the check that keeps !wiersz to the public domain rather than a
hope about where the poems came from.
"""
assert parse(_book(footer="Ten utwór jest udostępniony na licencji CC BY-SA.")) is None
def test_a_translation_is_refused() -> None:
# A poem translated into Polish is not a Polish poem, and the credit
# line under the title is the only place the text says so.
header = "Tristan Derème\n\n[Ach! rzucić sieci zdarte…]\ntłum. Stefan Napierski"
assert parse(_book(header=header)) is None
@pytest.mark.parametrize("raw", ["", "jeden\n\ndwa\n", "\n\n\n"])
def test_anything_not_shaped_like_a_book_is_refused(raw: str) -> None:
# Skipped rather than guessed at: a guess here gets posted to a group.
assert parse(f"{raw}\n-----\n{PUBLIC_DOMAIN}\n") is None
@pytest.mark.parametrize("missing", ["author", "title", "body"])
def test_a_book_missing_a_field_is_refused(missing: str) -> None:
parts = {"author": "Maria Konopnicka", "title": "A co wam śpiewać", "body": BODY}
parts[missing] = ""
assert (
parse(_book(header=f"{parts['author']}\n\n{parts['title']}\n", body=parts["body"])) is None
)
def test_blank_lines_are_stanza_breaks_and_not_lines_of_poetry() -> None:
# A four-stanza poem would otherwise look twice as long as it reads,
# and the length cap is what decides whether it ships at all.
poem = parse(_book(body="jeden\n\ndwa\n\ntrzy"))
assert poem is not None
assert poem.lines == 3

120
tool/art_categories.json Normal file
View File

@ -0,0 +1,120 @@
{
"_comment": "Which Commons categories !sztuka draws from, in two pools. Data, not source, for the reason aliases.json is: changing the policy should never mean touching Python. Every seed below was checked against Commons before being written down, and gen_art.py checks them all again before each harvest -- fourteen names in the first draft (Polish paintings, Polish sculptures, Paintings by Polish painters, Polish drawings, Polish prints, Polish posters, Drawings by Polish artists, Sculptures by Polish sculptors, Polish decorative arts, Stained glass windows in Poland, Old master prints, Textile art, Icons (religion), Witkacy) all look entirely plausible and none of them exists, so a typo would otherwise remove a whole artist or medium in silence.",
"_pools_comment": "The pools are separate because the draw is weighted, not merged: art_command.POLISH_SHARE per cent of posts come from 'polish' whatever the two sizes are. They are also walked disjointly -- a category reached from a Polish seed is never walked again from a world seed, and any file that still turns up in both is subtracted from the world pool -- so no work can sit in both and be drawn at both odds. The consequence worth knowing: 'polish' means reachable from a Polish seed, not painted by a Pole. A Matejko filed only under Category:Portrait paintings and not under his own category counts as world art, so the real Polish share of the corpus is a little higher than POLISH_SHARE makes the output.",
"_polish_comment": "Museums, movements and media first, then artists whose own category is the granular unit Commons actually files paintings under. Bacciarelli and Bellotto were Italians who spent their careers painting in Warsaw, and belong to Polish art history for that reason rather than by birth.",
"polish": [
"Category:Art in Poland",
"Category:Art of Poland",
"Category:Paintings in Poland",
"Category:Paintings from Poland",
"Category:Paintings of Poland",
"Category:17th-century paintings from Poland",
"Category:18th-century paintings from Poland",
"Category:19th-century paintings from Poland",
"Category:20th-century paintings from Poland",
"Category:Paintings in the National Museum in Warsaw",
"Category:Paintings in the National Museum in Kraków",
"Category:Paintings in the National Museum in Gdańsk",
"Category:Paintings in the National Museum in Poznań",
"Category:Paintings in the National Museum in Wrocław",
"Category:Sculptures in Poland",
"Category:Sculptures from Poland",
"Category:Sculptures in Warsaw",
"Category:Sculptures in Kraków",
"Category:Drawings from Poland",
"Category:Prints from Poland",
"Category:Stained glass in Poland",
"Category:Folk art in Poland",
"Category:Baroque art in Poland",
"Category:Art Nouveau in Poland",
"Category:Young Poland",
"Category:Artists from Poland",
"Category:Painters from Poland",
"Category:Sculptors from Poland",
"Category:Jan Matejko",
"Category:Józef Chełmoński",
"Category:Jacek Malczewski",
"Category:Stanisław Wyspiański",
"Category:Aleksander Gierymski",
"Category:Maksymilian Gierymski",
"Category:Julian Fałat",
"Category:Leon Wyczółkowski",
"Category:Olga Boznańska",
"Category:Władysław Podkowiński",
"Category:Józef Brandt",
"Category:Henryk Siemiradzki",
"Category:Wojciech Kossak",
"Category:Juliusz Kossak",
"Category:Piotr Michałowski",
"Category:Tadeusz Makowski",
"Category:Zofia Stryjeńska",
"Category:Teodor Axentowicz",
"Category:Wlastimil Hofman",
"Category:Xawery Dunikowski",
"Category:Jerzy Nowosielski",
"Category:Zdzisław Beksiński",
"Category:Tamara de Lempicka",
"Category:Marcello Bacciarelli",
"Category:Bernardo Bellotto"
],
"_world_comment": "Media before movements before museums, so that the pool is not simply European oil painting: engraving, lithograph, woodcut, fresco, mosaic, tapestry, ceramic, icon, relief, illuminated manuscript, calligraphy and ukiyo-e are all in it.",
"world": [
"Category:Oil paintings",
"Category:Drawings",
"Category:Engravings",
"Category:Lithographs",
"Category:Etchings",
"Category:Woodcuts",
"Category:Watercolor paintings",
"Category:Pastels",
"Category:Frescos",
"Category:Mosaics",
"Category:Tapestries",
"Category:Ceramics",
"Category:Calligraphy",
"Category:Icons",
"Category:Illuminated manuscripts",
"Category:Ukiyo-e",
"Category:Art Nouveau posters",
"Category:Bronze sculptures",
"Category:Marble sculptures",
"Category:Terracotta sculptures",
"Category:Reliefs",
"Category:Statues",
"Category:Busts",
"Category:Altarpieces",
"Category:Triptychs",
"Category:Portrait paintings",
"Category:Self-portraits",
"Category:Landscape paintings",
"Category:Genre paintings",
"Category:History paintings",
"Category:Religious paintings",
"Category:Renaissance paintings",
"Category:Baroque paintings",
"Category:Impressionist paintings",
"Category:Post-Impressionist paintings",
"Category:Symbolist paintings",
"Category:Expressionist paintings",
"Category:Paintings in the Louvre",
"Category:Paintings in the Metropolitan Museum of Art",
"Category:Paintings in the National Gallery, London",
"Category:Paintings in the Uffizi Gallery"
],
"_excluded_comment": "Pruned as whole subtrees, never enqueued and never read. Book illustrations is not objectionable, it is simply eleven thousand five hundred files -- more than every other world seed put together -- and a uniform draw would turn !sztuka into a book-illustration command. Erotic art is excluded for the reason the trap generator excludes booby traps: the command posts an attachment into a group chat and this list is the only place its content can be decided at all.",
"excluded": [
"Category:Book illustrations",
"Category:Erotic art"
],
"prune_patterns": [
"Erotic",
"Nude",
"Pornograph",
"Photographs of",
"Graves of",
"Tombs of",
"Signatures of",
"Graffiti"
],
"_prune_comment": "Any subcategory whose title contains one of these is pruned wholesale. A pattern rather than a list because Commons mints a new '<thing>' category whenever somebody needs one. 'Photographs of' is here because an artist's own category holds photographs of the artist as well as their work, and a portrait photograph of a sculptor is not a sculpture; the same rule drops the graves, tombs and signatures those categories also collect."
}

179
tool/commons_api.py Executable file
View File

@ -0,0 +1,179 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Talking to Wikimedia Commons, for the generators that harvest from it.
Extracted from ``gen_traps.py`` when ``gen_art.py`` arrived and would
otherwise have carried a second copy of the same retry loop, the same
continuation handling and the same licence filter. Every constant here was
measured by that first generator; the comments say what by.
Nothing in this module knows what a trap or a painting is. It knows how to
ask Commons a question without being rate-limited into silence, and how to
tell whether a file may be reposted.
"""
from __future__ import annotations
import time
from typing import Any, Final
import httpx
API: Final = "https://commons.wikimedia.org/w/api.php"
# The extension is what tells Signal how to render the attachment, so an
# unknown one is dropped rather than guessed. Same set as portraits.SUFFIXES.
SUFFIXES: Final = frozenset({".jpg", ".jpeg", ".png", ".gif"})
# Commons hosts non-free content too. !gekon filters licences at its query and
# endpoints.py calls that filter load-bearing; this is the same idea, on the
# only field Commons exposes for it. Matched case-folded, by prefix.
LICENCES: Final = ("public domain", "cc0", "cc by", "cc-by", "no restrictions")
# imageinfo takes fifty page ids per request, joined with "|". A comma is
# rejected outright ("badinteger"), which is worth writing down because the
# error names the parameter rather than the separator.
BATCH: Final = 50
# Wikimedia's etiquette asks for requests in series, and a strictly serial
# walk of a hundred-thousand-file harvest is the best part of an hour. Three
# is the compromise: enough to hide the round trip, few enough to stay a
# polite read against a project that serves everybody. The 429 handling in
# get() is the safety net rather than the plan -- if Commons does push back,
# every worker backs off exponentially and the harvest slows instead of
# failing. Only the per-category *file* queries run this way; claiming a
# category still happens serially, so `seen` and the pool ordering it
# guarantees are untouched.
WORKERS: Final = 3
TOO_MANY_REQUESTS: Final = 429
SERVER_ERROR: Final = 500
RETRIES: Final = 6
PAUSE: Final = 0.4
# Wikimedia's documented way to be a good citizen, and better than guessing a
# sleep: the server reports its own replication lag and refuses work when it
# is behind, so the throttle is feedback instead of superstition. Five seconds
# is the value their API docs give for interactive tools.
MAXLAG: Final = "5"
def get(params: dict[str, str], user_agent: str) -> dict[str, Any]:
"""One API call, retried with a widening pause when Commons says slow down.
Commons answers 429 after a few hundred requests even at a steady 0.4s,
and a harvest that silently loses a category is worse than a slow one.
Args:
params: The query, minus the action/format boilerplate.
user_agent: Wikimedia's policy requires a descriptive agent with a
contact URL; the generic client default is rate-limited hard and
may be blocked outright.
Returns:
The decoded response body.
Raises:
RuntimeError: If Commons answered 429 to every attempt.
"""
query = {
"action": "query",
"format": "json",
"formatversion": "2",
"maxlag": MAXLAG,
**params,
}
for attempt in range(RETRIES):
time.sleep(PAUSE)
try:
response = httpx.get(
API, params=query, headers={"User-Agent": user_agent}, timeout=120.0
)
except httpx.TransportError:
# A dropped connection is not an answer; treat it like a 5xx.
time.sleep(2.0**attempt)
continue
# 429 is Commons asking for less; 5xx is Commons failing under the
# weight of a generator+imageinfo query, which is expensive enough to
# time out on the biggest categories. Both are transient and both
# used to be fatal -- a single 504 on Category:Drawings threw away a
# nine-minute harvest, which is what put this branch here.
if response.status_code == TOO_MANY_REQUESTS or response.status_code >= SERVER_ERROR:
time.sleep(2.0**attempt)
continue
response.raise_for_status()
document: dict[str, Any] = response.json()
# maxlag is reported in-band with HTTP 200, not as a status code.
if isinstance(document.get("error"), dict) and document["error"].get("code") == "maxlag":
time.sleep(2.0**attempt)
continue
return document
msg = f"Commons kept refusing {params}"
raise RuntimeError(msg)
def members(category: str, kind: str, user_agent: str) -> list[dict[str, Any]]:
"""Every member of one category, following continuations."""
found: list[dict[str, Any]] = []
carry_on: str | None = None
while True:
params = {
"list": "categorymembers",
"cmtitle": category,
"cmtype": kind,
"cmlimit": "500",
}
if carry_on is not None:
params["cmcontinue"] = carry_on
document = get(params, user_agent)
found += document.get("query", {}).get("categorymembers", [])
carry_on = document.get("continue", {}).get("cmcontinue")
if carry_on is None:
return found
def licence_of(page: dict[str, Any]) -> str:
"""The licence short name for one imageinfo page, or "" if it has none.
Every field is checked rather than indexed, because MediaWiki serialises
an **empty map as an empty list** -- a PHP array quirk that
formatversion=2 does not fully hide. A file whose extmetadata is empty
arrives as ``"extmetadata": []``, and calling .get() on that threw
AttributeError forty thousand files into a harvest and lost the run. The
traps corpus happened not to contain one; the art corpus does.
"""
info = page.get("imageinfo")
if not isinstance(info, list) or not info or not isinstance(info[0], dict):
return ""
meta = info[0].get("extmetadata")
if not isinstance(meta, dict):
return ""
short = meta.get("LicenseShortName")
if not isinstance(short, dict):
return ""
return str(short.get("value", ""))
def verify_seeds(seeds: tuple[str, ...], user_agent: str) -> None:
"""Fail before harvesting if any seed category does not exist.
A category that is not there is not an error to Commons -- it is an empty
answer -- so a plausible-looking typo removes a whole subject silently.
Five of the trap generator's first-draft names were wrong this way, and
fourteen of the art generator's were.
Raises:
RuntimeError: If any seed is missing, naming all of them at once.
"""
missing: list[str] = []
for start in range(0, len(seeds), BATCH):
document = get({"titles": "|".join(seeds[start : start + BATCH])}, user_agent)
missing += [
str(page["title"])
for page in document.get("query", {}).get("pages", [])
if page.get("missing")
]
if missing:
msg = f"these seed categories do not exist on Commons: {sorted(missing)}"
raise RuntimeError(msg)

55
tool/commons_cache.py Executable file
View File

@ -0,0 +1,55 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Where an interrupted harvest leaves the work it already did.
Its own module because it is its own concern: neither talking to Commons
(commons_api) nor deciding which files belong in a pool (commons_harvest),
but making the second survivable. A harvest is an hour of somebody else's
bandwidth, and killing one used to throw all of it away -- three times in a
single afternoon, which is what prompted this.
The unit is the category. Whatever finished stays finished, so a second run
pays only for what the first did not reach, and a completed harvest re-runs
in seconds.
"""
from __future__ import annotations
import json
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from pathlib import Path
def load_cache(path: Path) -> dict[str, dict[int, str]]:
"""Categories already walked, from a previous run of this generator.
A harvest is an hour of somebody else's bandwidth, and killing it used to
throw all of that away -- three times in one afternoon, which is what
prompted this. The unit is the category rather than the run: whatever was
finished stays finished, so a second run pays only for what the first did
not reach, and a completed harvest re-runs in seconds.
JSON object keys are strings, so the page ids come back as strings and are
converted here rather than at every use.
"""
if not path.is_file():
return {}
try:
document = json.loads(path.read_text(encoding="utf-8"))
except OSError, json.JSONDecodeError:
# A half-written cache is a reason to redo the work, never a reason
# to fail: the whole point of this file is that it is disposable.
return {}
return {
category: {int(page): str(title) for page, title in files.items()}
for category, files in document.items()
}
def save_cache(path: Path, cache: dict[str, dict[int, str]]) -> None:
"""Write the cache, atomically, so an interrupted save cannot poison it."""
temporary = path.with_suffix(".tmp")
temporary.write_text(json.dumps(cache), encoding="utf-8")
temporary.replace(path)

222
tool/commons_harvest.py Executable file
View File

@ -0,0 +1,222 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Harvesting files out of Commons categories, licence and all.
Split from ``commons_api`` at the 250-line cap. That module is the transport
-- one request, retried politely -- and this one is everything built on it:
which categories a pool draws from, which files may be reposted, and the
cache that lets an interrupted harvest resume instead of starting over.
"""
from __future__ import annotations
import sys
from concurrent.futures import ThreadPoolExecutor
from functools import partial
from pathlib import Path
from typing import Any
from commons_api import (
BATCH,
LICENCES,
SUFFIXES,
WORKERS,
get,
licence_of,
members,
)
def licensed_members(category: str, *, user_agent: str) -> dict[int, str]:
"""Every usable, repostable file in one category, in a single pass.
``list=categorymembers`` names the files and ``prop=imageinfo`` reports
the licence, and MediaWiki will do both in one request if the first is
run as a *generator* for the second. Doing them separately -- which is
what this module did first, and what gen_traps still does -- costs an
extra request per fifty files, and those requests are the slow kind:
measured at 2.3s each against 0.75s for a plain listing, because
extmetadata is expensive. On a 75,000-file harvest that second pass was
forty-six minutes on its own.
Verified equivalent to ``members`` + ``licensed`` page id for page id, on
a painter's category of 387 files and on Category:Ukiyo-e of 311.
"""
kept: dict[int, str] = {}
carry: dict[str, str] = {}
while True:
document = get(
{
"generator": "categorymembers",
"gcmtitle": category,
"gcmtype": "file",
"gcmlimit": "500",
"prop": "imageinfo",
"iiprop": "extmetadata",
"iiextmetadatafilter": "LicenseShortName",
**carry,
},
user_agent,
)
for page in document.get("query", {}).get("pages", []):
title = str(page["title"])
if Path(title).suffix.lower() not in SUFFIXES:
continue
if licence_of(page).casefold().startswith(LICENCES):
kept[int(page["pageid"])] = title
carry = document.get("continue", {})
if not carry:
return kept
def _fetch_licensed(
wanted: list[str],
user_agent: str,
cache: dict[str, dict[int, str]] | None,
) -> dict[int, str]:
"""Every licensed file in these categories, reusing whatever is cached.
Split out of walk because it is the only part that is concurrent and the
only part that touches the cache; walk stays a description of which
categories get visited, which is what makes the pool ordering readable.
"""
files: dict[int, str] = {}
if cache is not None:
for category in [name for name in wanted if name in cache]:
files.update(cache[category])
wanted = [name for name in wanted if name not in cache]
# A quotation mark would end the incategory: term early and silently
# change the query, so those few categories keep the per-category path.
# Per category, via list=categorymembers, because that is the
# authoritative enumeration. Asking CirrusSearch instead (incategory:
# ORed across a batch) looked like a 6x win and was measured to be
# neither: it returned 2404 files where this returns 2429, dropped a
# different ~1% on every run, and was slower once the overflow path
# started firing. A search index is ranked and eventually consistent;
# deep pagination over one silently loses rows. Kept as a comment so the
# next person does not spend an afternoon rediscovering it.
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
found_each = pool.map(partial(licensed_members, user_agent=user_agent), wanted)
for category, found in zip(wanted, found_each, strict=True):
files.update(found)
if cache is not None:
cache[category] = found
return files
def walk(
seeds: tuple[str, ...],
excluded: frozenset[str],
patterns: tuple[str, ...],
user_agent: str,
seen: set[str] | None = None,
*,
with_licences: bool = False,
cache: dict[str, dict[int, str]] | None = None,
) -> dict[int, str]:
"""Page id to title, for every usable file one step from the seeds.
Depth is one, and that is measured rather than chosen: recursing three
levels from *Category:Fish traps* reaches Alfred Sisley's net paintings
and Hawaiian fisherman watercolours. Immediate subcategories only.
Args:
seeds: Categories to start from.
excluded: Categories pruned as whole subtrees.
patterns: A subcategory whose title contains one of these is pruned.
user_agent: Passed through to every request.
seen: Categories already walked. Pass one set across two calls to keep
two pools disjoint -- a file in both would be drawn at both odds.
with_licences: Filter licences in the same request that lists the
files, rather than in a second pass. Faster by far -- see
licensed_members -- and the caller then must not call
``licensed`` again.
cache: Category to its files, from load_cache. Anything already in
here is not fetched again, which is what makes an interrupted
harvest resumable instead of wasted. Mutated as the walk goes.
Returns:
Every file found, page id to title.
"""
files: dict[int, str] = {}
visited = set(excluded) if seen is None else seen | set(excluded)
for number, seed in enumerate(seeds, start=1):
sys.stderr.write(f"[{number}/{len(seeds)}] {seed} ({len(files)} files so far)\n")
sys.stderr.flush()
branch = [seed, *(sub["title"] for sub in members(seed, "subcat", user_agent))]
wanted: list[str] = []
for category in branch:
if category in visited or any(pattern in category for pattern in patterns):
continue
visited.add(category)
wanted.append(category)
if with_licences:
files.update(_fetch_licensed(wanted, user_agent, cache))
continue
for category in wanted:
for entry in members(category, "file", user_agent):
title = str(entry["title"])
if Path(title).suffix.lower() in SUFFIXES:
files[int(entry["pageid"])] = title
if seen is not None:
seen |= visited
return files
def _titles_page(titles: list[str], user_agent: str) -> dict[str, Any]:
"""One imageinfo lookup for a batch of file titles."""
return get(
{
"titles": "|".join(titles),
"prop": "imageinfo",
"iiprop": "extmetadata",
"iiextmetadatafilter": "LicenseShortName",
},
user_agent,
)
def licensed_titles(titles: list[str], user_agent: str) -> dict[int, str]:
"""Resolve Commons file titles to page ids, keeping only repostable ones.
The mirror image of ``licensed``: that one starts from page ids the walk
found, this one starts from titles somebody else named -- Wikidata, which
knows the filename and not the id. Same licence rule, same batch size.
"""
kept: dict[int, str] = {}
batches = [titles[start : start + BATCH] for start in range(0, len(titles), BATCH)]
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
documents = list(pool.map(partial(_titles_page, user_agent=user_agent), batches))
for document in documents:
for page in document.get("query", {}).get("pages", []):
if page.get("missing"):
continue
title = str(page["title"])
if Path(title).suffix.lower() not in SUFFIXES:
continue
if licence_of(page).casefold().startswith(LICENCES):
kept[int(page["pageid"])] = title
return kept
def licensed(files: dict[int, str], user_agent: str) -> dict[int, str]:
"""Drop everything whose licence does not permit reposting it."""
kept: dict[int, str] = {}
ids = sorted(files)
sys.stderr.write(f"checking licences on {len(ids)} files\n")
sys.stderr.flush()
for start in range(0, len(ids), BATCH):
chunk = ids[start : start + BATCH]
document = get(
{
"prop": "imageinfo",
"iiprop": "extmetadata",
"iiextmetadatafilter": "LicenseShortName",
"pageids": "|".join(str(page) for page in chunk),
},
user_agent,
)
for page in document.get("query", {}).get("pages", []):
if licence_of(page).casefold().startswith(LICENCES):
kept[int(page["pageid"])] = files[int(page["pageid"])]
return kept

142
tool/gen_art.py Executable file
View File

@ -0,0 +1,142 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Regenerate ``signal_bot/art.json`` from Wikimedia Commons.
Run this by hand; the output is committed, so the bot never searches Commons
at the moment somebody types ``!sztuka``. Everything ``gen_traps.py`` says
about why the file holds **page ids rather than filenames** applies here with
more force, not less: a Commons painting filename is very often literally a
person's name -- ``Portret Stanisława Potockiego.jpg`` -- and committing
tens of thousands of those would collide with a label somebody minted from
the chat, at which point ``scripts/check_personal_data.py`` refuses every
commit for good. An integer cannot carry a name.
Two pools, written as two lists. The draw is weighted rather than merged
(see ``art_command.POLISH_SHARE``), so they must stay separate here, and
they are walked with one shared ``seen`` set so that no file lands in both.
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from typing import Final
import commons_api
import commons_cache
import commons_harvest
import wikidata
CATEGORIES_FILE: Final = Path(__file__).with_name("art_categories.json")
USER_AGENT: Final = "signal-bot art generator (https://github.com/kuhyx/signal-bot)"
# Per pool, not in total. A world pool of twenty thousand next to a Polish
# pool of eleven would still satisfy a combined floor while making the Polish
# 60% a draw from almost nothing, which is the failure this catches.
MINIMUM: Final = 300
# Where an interrupted harvest leaves what it already fetched. Disposable and
# gitignored: delete it to force a full re-read of Commons.
CACHE_FILE: Final = Path(__file__).with_name(".commons_cache.json")
def categories() -> tuple[tuple[str, ...], tuple[str, ...], frozenset[str], tuple[str, ...]]:
"""Return the Polish seeds, the world seeds, the pruned set and patterns."""
document = json.loads(CATEGORIES_FILE.read_text(encoding="utf-8"))
return (
tuple(document["polish"]),
tuple(document["world"]),
frozenset(document["excluded"]),
tuple(document["prune_patterns"]),
)
def harvest() -> tuple[dict[int, str], dict[int, str]]:
"""Both pools, walked disjointly and licence-filtered in one pass.
Polish first, and that ordering is the whole point of sharing ``seen``:
*Category:Oil paintings* reaches Matejko too, and whichever pool walks a
category first owns it. Walking Polish first means a Polish painting
stays Polish rather than being counted as world art.
"""
polish_seeds, world_seeds, excluded, patterns = categories()
commons_api.verify_seeds(polish_seeds + world_seeds, USER_AGENT)
seen: set[str] = set()
cache = commons_cache.load_cache(CACHE_FILE)
if cache:
sys.stderr.write(f"resuming: {len(cache)} categories already fetched\n")
sys.stderr.write("== polish pool\n")
polish = commons_harvest.walk(
polish_seeds, excluded, patterns, USER_AGENT, seen, with_licences=True, cache=cache
)
commons_cache.save_cache(CACHE_FILE, cache)
# Wikidata knows which works were made by Poles, which is the question
# the categories only approximate. Additive: it reaches works nobody
# filed under a Polish category, the categories reach works nobody made
# a Wikidata item for.
sys.stderr.write("== polish pool, via wikidata\n")
titles = wikidata.polish_artwork_titles(USER_AGENT)
sys.stderr.write(f"wikidata named {len(titles)} artworks by Polish creators\n")
before = len(polish)
polish.update(commons_harvest.licensed_titles(titles, USER_AGENT))
sys.stderr.write(f"polish pool {before} -> {len(polish)} after wikidata\n")
sys.stderr.write("== world pool\n")
world = commons_harvest.walk(
world_seeds, excluded, patterns, USER_AGENT, seen, with_licences=True, cache=cache
)
commons_cache.save_cache(CACHE_FILE, cache)
# A file can still be in two categories that were walked by different
# pools; seen only makes the *categories* disjoint. Subtract explicitly.
for page in polish:
world.pop(page, None)
# No second licence pass: walk(with_licences=True) already filtered, in
# the same requests that listed the files. That second pass was 1,500
# requests at 2.3s each -- forty-six minutes -- on the previous run.
return polish, world
def _rejection(polish: dict[int, str], world: dict[int, str]) -> str:
"""Return why this harvest must not be written, or an empty string."""
for label, pool in (("polish", polish), ("world", world)):
if len(pool) < MINIMUM:
return f"got {len(pool)} {label} works, expected at least {MINIMUM}"
return ""
def main() -> int:
"""Write art.json, refusing to write a harvest that failed its own checks."""
parser = argparse.ArgumentParser(description="Regenerate signal_bot/art.json.")
parser.add_argument(
"--titles",
action="store_true",
help="print every kept title; the committed file is integers, so this is the review",
)
arguments = parser.parse_args()
polish, world = harvest()
rejection = _rejection(polish, world)
if rejection:
sys.stderr.write(f"refusing to write: {rejection}\n")
return 1
if arguments.titles:
for label, pool in (("polish", polish), ("world", world)):
for page in sorted(pool):
sys.stdout.write(f"{label}\t{page}\t{pool[page]}\n")
out = Path(__file__).resolve().parent.parent / "signal_bot" / "art.json"
document = {
"source": str(CATEGORIES_FILE.name),
"polish": sorted(polish),
"world": sorted(world),
}
out.write_text(json.dumps(document, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
sys.stdout.write(f"wrote {len(polish)} polish + {len(world)} world works to {out}\n")
return 0
if __name__ == "__main__":
raise SystemExit(main())

203
tool/gen_poems.py Executable file
View File

@ -0,0 +1,203 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Regenerate ``signal_bot/poems.json`` from Wolne Lektury.
Run this by hand; the output is committed, so the bot never searches the
catalogue at the moment somebody types ``!wiersz`` -- it fetches exactly one
known poem.
**The file holds slugs, not poems, and that is load-bearing.** The first
version of this generator committed the text, and
``scripts/check_personal_data.py`` refused the pack: Polish verse is full of
Polish given names, three of them collided with a real group member, and two
still did after a filter that had already discarded 31% of the corpus,
because the names sat at the start of a line where no filter can look. A slug
cannot carry a name the way a poem can, and everything a reader sees --
title, author, the poem itself -- is fetched at post time and never written
here. ``signal_bot/poem_command.py`` tells the longer version.
What is left to decide at harvest time is which poems are *worth* committing,
and that needs the text, so this reads all five thousand once:
* a work is dropped unless its own footer says *Ten utwór jest w domenie
publicznej* -- Wolne Lektury hosts living authors too, and a third of the
catalogue came back non-free;
* a translation is dropped -- the txt carries a ``tłum.`` credit under the
title, and a poem translated into Polish is not a Polish poem;
* anything longer than the caps is dropped, whole poems only;
* a title carrying a proper noun is dropped. This one is belt-and-braces
rather than load-bearing now: the slug is derived from the title, so
filtering the title is what keeps a dedication like *Do Zosi* from
committing ``do-zosi``. It costs about a tenth of the pool and removes the
only free text left in the file.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
import time
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from functools import partial
from pathlib import Path
from typing import Final
import httpx
from signal_bot.poem_command import MAX_CHARS, MAX_LINES
from signal_bot.wolnelektury import parse
API: Final = "https://wolnelektury.pl/api/genres/{genre}/books/?format=json"
TEXT: Final = "https://wolnelektury.pl/media/book/txt/{slug}.txt"
USER_AGENT: Final = "signal-bot poems generator (https://github.com/kuhyx/signal-bot)"
# Lyric forms only. "Poemat" is deliberately absent: a poemat is book-length
# by definition and every one would fail the caps anyway, costing a few
# hundred requests to learn nothing.
GENRES: Final = ("wiersz", "sonet", "fraszka", "wiersz-sylabotoniczny")
# A capitalised word that is not the title's first and does not open a
# bracket. Catches the dedications -- "Do Zosi", "Na śmierć Jana", "O
# Mickiewiczu" -- while keeping the very common "IV (Jak ptaki, kiedy
# odlatywać poczną...)" incipit titles, which are a large slice of the
# catalogue and carry no name at all.
TITLE_PROPER_NOUN: Final = re.compile(r"\w[\s,]+([A-ZĄĆĘŁŃÓŚŹŻ][a-ząćęłńóśźż]+)")
# A pool smaller than this is a truncated harvest, and committing it would
# quietly shrink the command's whole world.
MINIMUM: Final = 300
# Five thousand sequential fetches spends half an hour waiting on latency
# rather than on Wolne Lektury. Four workers against a static text endpoint
# is roughly ten requests a second -- polite for a volunteer-run library, and
# it turns the harvest from thirty-odd minutes into six. httpx.Client is
# thread-safe and they share its one connection pool.
WORKERS: Final = 4
TOO_MANY_REQUESTS: Final = 429
RETRIES: Final = 5
PAUSE: Final = 0.1
def _get(url: str, client: httpx.Client) -> str:
"""One request, retried with a widening pause when the site says slow down.
The client is passed in rather than made here, for the reason app.py
keeps one AsyncClient for the whole bot: a bare ``httpx.get`` is a fresh
TCP connection and a fresh TLS handshake every time, and across five
thousand poems that handshake is most of the harvest rather than most of
the politeness. One reused connection is both faster and less to ask of a
volunteer-run library.
"""
for attempt in range(RETRIES):
time.sleep(PAUSE)
response = client.get(url, headers={"User-Agent": USER_AGENT}, timeout=60.0)
if response.status_code == TOO_MANY_REQUESTS:
time.sleep(2.0**attempt)
continue
response.raise_for_status()
return response.text
msg = f"wolnelektury.pl kept answering {TOO_MANY_REQUESTS} for {url}"
raise RuntimeError(msg)
def catalogue(client: httpx.Client) -> set[str]:
"""Every lyric slug in the catalogue.
A set because the genres overlap heavily -- a sonnet is also a wiersz --
and fetching the same poem four times would be four times the requests
for the same answer.
"""
found: set[str] = set()
for genre in GENRES:
books = json.loads(_get(API.format(genre=genre), client))
found |= {str(book["slug"]) for book in books}
sys.stderr.write(f"{genre}: {len(books)} books ({len(found)} distinct so far)\n")
sys.stderr.flush()
return found
def _fetch(slug: str, client: httpx.Client) -> tuple[str, str | None]:
"""One slug and its text, or None when the fetch failed.
Failure comes back as a value rather than an exception because this runs
in a worker thread: one dead slug out of five thousand should cost that
slug, not the harvest.
"""
try:
return slug, _get(TEXT.format(slug=slug), client)
except httpx.HTTPError:
return slug, None
def consider(slug: str, raw: str) -> tuple[str | None, str]:
"""The slug if it is worth committing, or None and the reason it is not."""
poem = parse(raw)
if poem is None:
return None, "not public domain, translated or malformed"
if poem.lines > MAX_LINES or len(poem.text) > MAX_CHARS:
return None, "too long"
if TITLE_PROPER_NOUN.search(poem.title):
return None, "title carries a proper noun"
return slug, ""
def harvest(client: httpx.Client) -> tuple[list[str], Counter[str]]:
"""Every slug worth committing, and a tally of why the rest were not.
``map`` rather than ``as_completed``: it yields in submission order, so
the progress line means what it says and the result does not depend on
which worker happened to finish first.
"""
slugs = sorted(catalogue(client))
kept: list[str] = []
why: Counter[str] = Counter()
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
for number, (slug, raw) in enumerate(
pool.map(partial(_fetch, client=client), slugs), start=1
):
if number % 500 == 0:
sys.stderr.write(f"[{number}/{len(slugs)}] kept {len(kept)}\n")
sys.stderr.flush()
if raw is None:
why["no text"] += 1
continue
wanted, reason = consider(slug, raw)
if wanted is None:
why[reason] += 1
else:
kept.append(wanted)
return kept, why
def main() -> int:
"""Write poems.json, refusing to write a harvest that failed its own checks."""
parser = argparse.ArgumentParser(description="Regenerate signal_bot/poems.json.")
parser.add_argument("--slugs", action="store_true", help="print every kept slug")
arguments = parser.parse_args()
with httpx.Client(follow_redirects=True) as client:
kept, why = harvest(client)
for reason, count in why.most_common():
sys.stderr.write(f"rejected {count}: {reason}\n")
if len(kept) < MINIMUM:
sys.stderr.write(f"refusing to write: got {len(kept)}, expected at least {MINIMUM}\n")
return 1
if arguments.slugs:
for slug in kept:
sys.stdout.write(f"{slug}\n")
out = Path(__file__).resolve().parent.parent / "signal_bot" / "poems.json"
document = {"source": "wolnelektury.pl", "poems": sorted(kept)}
out.write_text(json.dumps(document, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
sys.stdout.write(f"wrote {len(kept)} slugs to {out}\n")
return 0
if __name__ == "__main__":
raise SystemExit(main())

140
tool/gen_traps.py
View File

@ -14,9 +14,9 @@ refuse every commit for good, over data nobody can edit out by hand. An
integer cannot carry a name. ``prop=imageinfo`` accepts ``pageids=`` and puts
``thumburl`` at the same place ``titles=`` does, so nothing downstream cares.
Depth is one, and that is also measured: recursing three levels from
*Category:Fish traps* reaches Alfred Sisley's net paintings and Hawaiian
fisherman watercolours. Immediate subcategories only.
The API plumbing -- the retry loop, the continuation handling, the licence
filter and the depth-of-one rule -- lives in ``commons_api.py``, which
``gen_art.py`` shares.
"""
from __future__ import annotations
@ -24,40 +24,20 @@ from __future__ import annotations
import argparse
import json
import sys
import time
from pathlib import Path
from typing import Any, Final
from typing import Final
import httpx
import commons_api
import commons_harvest
CATEGORIES_FILE: Final = Path(__file__).with_name("trap_categories.json")
API: Final = "https://commons.wikimedia.org/w/api.php"
# Wikimedia's policy requires a descriptive agent with a contact URL; the
# generic client default is rate-limited hard and may be blocked outright.
USER_AGENT: Final = "signal-bot traps generator (https://github.com/kuhyx/signal-bot)"
# The extension is what tells Signal how to render the attachment, so an
# unknown one is dropped rather than guessed. Same set as portraits.SUFFIXES.
SUFFIXES: Final = frozenset({".jpg", ".jpeg", ".png", ".gif"})
# Commons hosts non-free content too. !gekon filters licences at its query and
# endpoints.py calls that filter load-bearing; this is the same idea, on the
# only field Commons exposes for it. Matched case-folded, by prefix.
LICENCES: Final = ("public domain", "cc0", "cc by", "cc-by", "no restrictions")
# imageinfo takes fifty page ids per request, joined with "|". A comma is
# rejected outright ("badinteger"), which is worth writing down because the
# error names the parameter rather than the separator.
BATCH: Final = 50
# A harvest smaller than this is a truncated one, and committing it would
# quietly shrink the command's whole world.
MINIMUM: Final = 500
TOO_MANY_REQUESTS: Final = 429
RETRIES: Final = 6
PAUSE: Final = 0.4
def categories() -> tuple[tuple[str, ...], frozenset[str], tuple[str, ...]]:
"""Return the seeds, the pruned set and the pruned patterns."""
@ -69,113 +49,11 @@ def categories() -> tuple[tuple[str, ...], frozenset[str], tuple[str, ...]]:
)
def _get(params: dict[str, str]) -> dict[str, Any]:
"""One API call, retried with a widening pause when Commons says slow down.
Commons answers 429 after a few hundred requests even at a steady 0.4s,
and a harvest that silently loses a category is worse than a slow one.
"""
query = {"action": "query", "format": "json", "formatversion": "2", **params}
for attempt in range(RETRIES):
time.sleep(PAUSE)
response = httpx.get(API, params=query, headers={"User-Agent": USER_AGENT}, timeout=60.0)
if response.status_code == TOO_MANY_REQUESTS:
time.sleep(2.0**attempt)
continue
response.raise_for_status()
document: dict[str, Any] = response.json()
return document
msg = f"Commons kept answering {TOO_MANY_REQUESTS} for {params}"
raise RuntimeError(msg)
def _members(category: str, kind: str) -> list[dict[str, Any]]:
"""Every member of one category, following continuations."""
found: list[dict[str, Any]] = []
carry_on: str | None = None
while True:
params = {
"list": "categorymembers",
"cmtitle": category,
"cmtype": kind,
"cmlimit": "500",
}
if carry_on is not None:
params["cmcontinue"] = carry_on
document = _get(params)
found += document.get("query", {}).get("categorymembers", [])
carry_on = document.get("continue", {}).get("cmcontinue")
if carry_on is None:
return found
def verify_seeds(seeds: tuple[str, ...]) -> None:
"""Fail before harvesting if any seed category does not exist.
A category that is not there is not an error to Commons -- it is an empty
answer -- so a plausible-looking typo removes a whole subject silently.
Five of the first draft's names were wrong this way.
Raises:
RuntimeError: If any seed is missing, naming all of them at once.
"""
missing: list[str] = []
for start in range(0, len(seeds), BATCH):
document = _get({"titles": "|".join(seeds[start : start + BATCH])})
missing += [
str(page["title"])
for page in document.get("query", {}).get("pages", [])
if page.get("missing")
]
if missing:
msg = f"these seed categories do not exist on Commons: {sorted(missing)}"
raise RuntimeError(msg)
def harvest() -> dict[int, str]:
"""Page id to title, for every usable file one step from the seeds."""
seeds, excluded, patterns = categories()
verify_seeds(seeds)
files: dict[int, str] = {}
seen: set[str] = set(excluded)
for number, seed in enumerate(seeds, start=1):
sys.stderr.write(f"[{number}/{len(seeds)}] {seed} ({len(files)} files so far)\n")
sys.stderr.flush()
branch = [seed, *(sub["title"] for sub in _members(seed, "subcat"))]
for category in branch:
if category in seen or any(pattern in category for pattern in patterns):
continue
seen.add(category)
for entry in _members(category, "file"):
title = str(entry["title"])
if Path(title).suffix.lower() in SUFFIXES:
files[int(entry["pageid"])] = title
return files
def licensed(files: dict[int, str]) -> dict[int, str]:
"""Drop everything whose licence does not permit reposting it."""
kept: dict[int, str] = {}
ids = sorted(files)
sys.stderr.write(f"checking licences on {len(ids)} files\n")
sys.stderr.flush()
for start in range(0, len(ids), BATCH):
chunk = ids[start : start + BATCH]
document = _get(
{
"prop": "imageinfo",
"iiprop": "extmetadata",
"iiextmetadatafilter": "LicenseShortName",
"pageids": "|".join(str(page) for page in chunk),
}
)
for page in document.get("query", {}).get("pages", []):
info = page.get("imageinfo") or [{}]
short = info[0].get("extmetadata", {}).get("LicenseShortName", {}).get("value", "")
name = str(short).casefold()
if name.startswith(LICENCES):
kept[int(page["pageid"])] = files[int(page["pageid"])]
return kept
commons_api.verify_seeds(seeds, USER_AGENT)
return commons_harvest.walk(seeds, excluded, patterns, USER_AGENT)
def _rejection(traps: dict[int, str]) -> str:
@ -195,7 +73,7 @@ def main() -> int:
)
arguments = parser.parse_args()
traps = licensed(harvest())
traps = commons_harvest.licensed(harvest(), USER_AGENT)
rejection = _rejection(traps)
if rejection:
sys.stderr.write(f"refusing to write: {rejection}\n")

66
tool/wikidata.py Executable file
View File

@ -0,0 +1,66 @@
#!/usr/bin/env python3
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
"""Polish artworks, as Wikidata knows them.
The category walk in ``commons_api`` answers "what is filed near a Polish
category", which is a proxy. Wikidata answers the question actually being
asked -- "what was made by somebody with Polish citizenship" -- because it
models the artwork, its creator and the creator's citizenship as data rather
than as folder membership. One SPARQL query returns about six thousand of
them in six seconds.
It does not replace the category walk, it adds to it. Wikidata only knows
about works somebody has made an item for, which is a fraction of Commons;
the categories reach the rest. Used together, the Polish pool means both
"by a Pole" and "filed as Polish art".
The query service is public and needs no account, unlike Quarry and the
Commons Query Service, both of which require an OAuth login.
"""
from __future__ import annotations
import urllib.parse
from typing import Any, Final
import httpx
ENDPOINT: Final = "https://query.wikidata.org/sparql"
# Poland; "country of citizenship"; "creator"; "image".
# The classes are listed rather than reached through a subclass-of* path,
# because that path over every work of art times the service out: measured,
# it fails where this returns in six seconds.
QUERY: Final = """
SELECT DISTINCT ?image WHERE {
VALUES ?class { wd:Q3305213 wd:Q860861 wd:Q93184 wd:Q11060274 wd:Q18761202 wd:Q1473346 }
?work wdt:P31 ?class ; wdt:P18 ?image ; wdt:P170 ?creator .
?creator wdt:P27 wd:Q36 .
}
"""
TIMEOUT: Final = 180.0
def polish_artwork_titles(user_agent: str) -> list[str]:
"""Commons file titles for artworks by creators of Polish citizenship.
Returns:
Titles in Commons' own form -- ``File:Something.jpg`` -- ready to be
handed to ``commons_api.licensed_titles``. P18 gives a Special:FilePath
URL, so the name is percent-decoded and its underscores turned back
into spaces here rather than at every use.
"""
response = httpx.get(
ENDPOINT,
params={"query": QUERY, "format": "json"},
headers={"User-Agent": user_agent, "Accept": "application/sparql-results+json"},
timeout=TIMEOUT,
)
response.raise_for_status()
document: dict[str, Any] = response.json()
titles: list[str] = []
for row in document["results"]["bindings"]:
name = urllib.parse.unquote(str(row["image"]["value"]).rsplit("/", 1)[-1])
titles.append("File:" + name.replace("_", " "))
return titles