mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-10 22:57:01 +01:00
Krzysztof kuhy Rudnicki
08599e6056
!sztuka posts a work of art, skewed 60% Polish; !wiersz posts one short public-domain Polish poem, whole. Both follow the rule the rest of the bot already keeps: the repository commits identifiers, and the words arrive over the wire when somebody asks. !wiersz was written the other way first, and the gate caught it. A pack of short public-domain poems was harvested, name-filtered and committed, and check_personal_data.sh refused it: three lines carried a real group member's given name, and two still did after a filter that had discarded 31% of the corpus. That is structural, not a tuning problem -- Polish verse capitalises every line, so a line-initial capital carries no signal, and that is exactly where a name sits. The pack now holds Wolne Lektury slugs; the poem, its title and its author are fetched at post time and appear nowhere in git. It is also the better version: with no name filter there are no false positives, so the pool is larger and the attribution is a full name again. The gate then refused the art category list too, over a painter whose given name is somebody's. That is the popes.json case exactly, so it takes the same answer: a file-scoped allowlist entry, still refused everywhere else. The Commons harvest is now three modules by what they are, not by where the line count fell: commons_api is the conversation with Commons, commons_harvest turns categories into a pool of licensed page ids, commons_cache makes an interrupted run resumable. Along the way it gained licences in the same request that lists a category (the second pass was 1,500 requests at 2.3s each), maxlag instead of a guessed sleep, 5xx retries because one 504 used to destroy a whole run, and Wikidata for works by creators of Polish citizenship: the Polish pool grew 15,221 -> 20,092. Batching categories through CirrusSearch looked like a further 6x and was measured out again: 2,404 files where the authoritative enumeration returns 2,429, a different ~1% missing on every run. The offline run is 19s where it was four minutes, almost all of it the deafness gate waiting out three thirty-second liveness probes to prove something provable in three; the tick is configuration now, and e2e enforces a 40s budget on itself so that never silently comes back.
56 lines
2.1 KiB
Python
Executable File
56 lines
2.1 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
|
"""Where an interrupted harvest leaves the work it already did.
|
|
|
|
Its own module because it is its own concern: neither talking to Commons
|
|
(commons_api) nor deciding which files belong in a pool (commons_harvest),
|
|
but making the second survivable. A harvest is an hour of somebody else's
|
|
bandwidth, and killing one used to throw all of it away -- three times in a
|
|
single afternoon, which is what prompted this.
|
|
|
|
The unit is the category. Whatever finished stays finished, so a second run
|
|
pays only for what the first did not reach, and a completed harvest re-runs
|
|
in seconds.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
from typing import TYPE_CHECKING
|
|
|
|
if TYPE_CHECKING:
|
|
from pathlib import Path
|
|
|
|
|
|
def load_cache(path: Path) -> dict[str, dict[int, str]]:
|
|
"""Categories already walked, from a previous run of this generator.
|
|
|
|
A harvest is an hour of somebody else's bandwidth, and killing it used to
|
|
throw all of that away -- three times in one afternoon, which is what
|
|
prompted this. The unit is the category rather than the run: whatever was
|
|
finished stays finished, so a second run pays only for what the first did
|
|
not reach, and a completed harvest re-runs in seconds.
|
|
|
|
JSON object keys are strings, so the page ids come back as strings and are
|
|
converted here rather than at every use.
|
|
"""
|
|
if not path.is_file():
|
|
return {}
|
|
try:
|
|
document = json.loads(path.read_text(encoding="utf-8"))
|
|
except OSError, json.JSONDecodeError:
|
|
# A half-written cache is a reason to redo the work, never a reason
|
|
# to fail: the whole point of this file is that it is disposable.
|
|
return {}
|
|
return {
|
|
category: {int(page): str(title) for page, title in files.items()}
|
|
for category, files in document.items()
|
|
}
|
|
|
|
|
|
def save_cache(path: Path, cache: dict[str, dict[int, str]]) -> None:
|
|
"""Write the cache, atomically, so an interrupted save cannot poison it."""
|
|
temporary = path.with_suffix(".tmp")
|
|
temporary.write_text(json.dumps(cache), encoding="utf-8")
|
|
temporary.replace(path)
|