mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-10 19:37:03 +01:00
Krzysztof kuhy Rudnicki
08599e6056
!sztuka posts a work of art, skewed 60% Polish; !wiersz posts one short public-domain Polish poem, whole. Both follow the rule the rest of the bot already keeps: the repository commits identifiers, and the words arrive over the wire when somebody asks. !wiersz was written the other way first, and the gate caught it. A pack of short public-domain poems was harvested, name-filtered and committed, and check_personal_data.sh refused it: three lines carried a real group member's given name, and two still did after a filter that had discarded 31% of the corpus. That is structural, not a tuning problem -- Polish verse capitalises every line, so a line-initial capital carries no signal, and that is exactly where a name sits. The pack now holds Wolne Lektury slugs; the poem, its title and its author are fetched at post time and appear nowhere in git. It is also the better version: with no name filter there are no false positives, so the pool is larger and the attribution is a full name again. The gate then refused the art category list too, over a painter whose given name is somebody's. That is the popes.json case exactly, so it takes the same answer: a file-scoped allowlist entry, still refused everywhere else. The Commons harvest is now three modules by what they are, not by where the line count fell: commons_api is the conversation with Commons, commons_harvest turns categories into a pool of licensed page ids, commons_cache makes an interrupted run resumable. Along the way it gained licences in the same request that lists a category (the second pass was 1,500 requests at 2.3s each), maxlag instead of a guessed sleep, 5xx retries because one 504 used to destroy a whole run, and Wikidata for works by creators of Polish citizenship: the Polish pool grew 15,221 -> 20,092. Batching categories through CirrusSearch looked like a further 6x and was measured out again: 2,404 files where the authoritative enumeration returns 2,429, a different ~1% missing on every run. The offline run is 19s where it was four minutes, almost all of it the deafness gate waiting out three thirty-second liveness probes to prove something provable in three; the tick is configuration now, and e2e enforces a 40s budget on itself so that never silently comes back.
106 lines
4.1 KiB
Python
106 lines
4.1 KiB
Python
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
|
"""Reading Wolne Lektury's plain-text books.
|
|
|
|
Shared by ``poem_command``, which reads one at the moment somebody types
|
|
``!wiersz``, and by ``tool/gen_poems.py``, which reads five thousand to decide
|
|
which slugs are worth committing. One parser, because the harvest's idea of a
|
|
poem and the handler's have to be the same idea -- a slug committed as "short
|
|
enough" that the handler then renders differently would be a bug nothing
|
|
catches.
|
|
|
|
**Nothing this module returns is ever written into the repository**, and that
|
|
is the whole design of !wiersz rather than an incidental property. The first
|
|
attempt committed the poems themselves, and ``scripts/check_personal_data.py``
|
|
refused the pack: Polish verse is full of Polish given names, three of them
|
|
collided with a real group member, and two still did after a filter that had
|
|
already thrown away 31% of the corpus. The names sat at the start of a line,
|
|
which is exactly where a filter cannot look, because Polish verse capitalises
|
|
every line. So the pack holds slugs and the text arrives over the wire, which
|
|
is what ``commons.caption_for`` already says about every other command:
|
|
|
|
The text is fetched at run time and posted to the group; none of it is
|
|
ever written into this repository, which is why it may be free text
|
|
where traps.json may not.
|
|
|
|
The format is regular enough to parse and irregular enough to check: the
|
|
author on the first line, the title on the third, an optional ``tłum.``
|
|
credit under that, the poem, and a footer behind a lone rule.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass
|
|
from typing import Final
|
|
|
|
# Where a reader goes for the annotated version. A link, never fetched, so it
|
|
# is a constant rather than a configurable endpoint.
|
|
CATALOGUE: Final = "https://wolnelektury.pl/katalog/lektura/{slug}/"
|
|
|
|
# The sentence Wolne Lektury puts in the footer of a public-domain work, and
|
|
# does not put in the footer of a CC-licensed one. The corpus is not a licence:
|
|
# it hosts living authors too, and a third of it came back non-free.
|
|
PUBLIC_DOMAIN: Final = "Ten utwór jest w domenie publicznej."
|
|
|
|
# The credit line that marks a translation, directly under the title. A poem
|
|
# translated into Polish is not a Polish poem.
|
|
TRANSLATED: Final = "tłum."
|
|
|
|
FOOTER: Final = "\n-----"
|
|
|
|
# author, blank, title -- so the body, or a translator credit, starts here.
|
|
HEADER_LINES: Final = 3
|
|
|
|
|
|
@dataclass(frozen=True)
|
|
class Poem:
|
|
"""One poem as Wolne Lektury serves it."""
|
|
|
|
title: str
|
|
author: str
|
|
text: str
|
|
|
|
@property
|
|
def lines(self) -> int:
|
|
"""How many lines of verse, blanks not counted.
|
|
|
|
Blank lines are stanza breaks, and a stanza break is not a line of
|
|
poetry -- counting them would make a four-stanza poem look twice as
|
|
long as it reads.
|
|
"""
|
|
return len([line for line in self.text.splitlines() if line.strip()])
|
|
|
|
|
|
def parse(raw: str) -> Poem | None:
|
|
"""One book's plain text, or None when it is not a poem we can use.
|
|
|
|
Returns None rather than raising, and rather than guessing, for three
|
|
separate reasons that all end the same way -- this book is skipped:
|
|
|
|
* it is too short to have the header at all, i.e. not the shape we know;
|
|
* it carries a ``tłum.`` credit, so it is a translation;
|
|
* it is not in the public domain by its own footer's account.
|
|
|
|
A guess here would be committed by the harvest or posted to a group by
|
|
the handler, so anything unfamiliar is dropped instead.
|
|
|
|
Args:
|
|
raw: The body of a ``media/book/txt/<slug>.txt`` response.
|
|
|
|
Returns:
|
|
The poem, or None if it is unusable.
|
|
"""
|
|
if PUBLIC_DOMAIN not in raw:
|
|
return None
|
|
lines = raw.split(FOOTER, maxsplit=1)[0].splitlines()
|
|
if len(lines) <= HEADER_LINES:
|
|
return None
|
|
rest = lines[HEADER_LINES:]
|
|
if rest[0].strip().startswith(TRANSLATED):
|
|
return None
|
|
author = lines[0].strip()
|
|
title = lines[2].strip()
|
|
text = "\n".join(rest).strip()
|
|
if not author or not title or not text:
|
|
return None
|
|
return Poem(title=title, author=author, text=text)
|