mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-10 18:07:29 +01:00
Krzysztof kuhy Rudnicki
08599e6056
!sztuka posts a work of art, skewed 60% Polish; !wiersz posts one short public-domain Polish poem, whole. Both follow the rule the rest of the bot already keeps: the repository commits identifiers, and the words arrive over the wire when somebody asks. !wiersz was written the other way first, and the gate caught it. A pack of short public-domain poems was harvested, name-filtered and committed, and check_personal_data.sh refused it: three lines carried a real group member's given name, and two still did after a filter that had discarded 31% of the corpus. That is structural, not a tuning problem -- Polish verse capitalises every line, so a line-initial capital carries no signal, and that is exactly where a name sits. The pack now holds Wolne Lektury slugs; the poem, its title and its author are fetched at post time and appear nowhere in git. It is also the better version: with no name filter there are no false positives, so the pool is larger and the attribution is a full name again. The gate then refused the art category list too, over a painter whose given name is somebody's. That is the popes.json case exactly, so it takes the same answer: a file-scoped allowlist entry, still refused everywhere else. The Commons harvest is now three modules by what they are, not by where the line count fell: commons_api is the conversation with Commons, commons_harvest turns categories into a pool of licensed page ids, commons_cache makes an interrupted run resumable. Along the way it gained licences in the same request that lists a category (the second pass was 1,500 requests at 2.3s each), maxlag instead of a guessed sleep, 5xx retries because one 504 used to destroy a whole run, and Wikidata for works by creators of Polish citizenship: the Polish pool grew 15,221 -> 20,092. Batching categories through CirrusSearch looked like a further 6x and was measured out again: 2,404 files where the authoritative enumeration returns 2,429, a different ~1% missing on every run. The offline run is 19s where it was four minutes, almost all of it the deafness gate waiting out three thirty-second liveness probes to prove something provable in three; the tick is configuration now, and e2e enforces a 40s budget on itself so that never silently comes back.
67 lines
2.5 KiB
Python
Executable File
67 lines
2.5 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
|
"""Polish artworks, as Wikidata knows them.
|
|
|
|
The category walk in ``commons_api`` answers "what is filed near a Polish
|
|
category", which is a proxy. Wikidata answers the question actually being
|
|
asked -- "what was made by somebody with Polish citizenship" -- because it
|
|
models the artwork, its creator and the creator's citizenship as data rather
|
|
than as folder membership. One SPARQL query returns about six thousand of
|
|
them in six seconds.
|
|
|
|
It does not replace the category walk, it adds to it. Wikidata only knows
|
|
about works somebody has made an item for, which is a fraction of Commons;
|
|
the categories reach the rest. Used together, the Polish pool means both
|
|
"by a Pole" and "filed as Polish art".
|
|
|
|
The query service is public and needs no account, unlike Quarry and the
|
|
Commons Query Service, both of which require an OAuth login.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import urllib.parse
|
|
from typing import Any, Final
|
|
|
|
import httpx
|
|
|
|
ENDPOINT: Final = "https://query.wikidata.org/sparql"
|
|
|
|
# Poland; "country of citizenship"; "creator"; "image".
|
|
# The classes are listed rather than reached through a subclass-of* path,
|
|
# because that path over every work of art times the service out: measured,
|
|
# it fails where this returns in six seconds.
|
|
QUERY: Final = """
|
|
SELECT DISTINCT ?image WHERE {
|
|
VALUES ?class { wd:Q3305213 wd:Q860861 wd:Q93184 wd:Q11060274 wd:Q18761202 wd:Q1473346 }
|
|
?work wdt:P31 ?class ; wdt:P18 ?image ; wdt:P170 ?creator .
|
|
?creator wdt:P27 wd:Q36 .
|
|
}
|
|
"""
|
|
|
|
TIMEOUT: Final = 180.0
|
|
|
|
|
|
def polish_artwork_titles(user_agent: str) -> list[str]:
|
|
"""Commons file titles for artworks by creators of Polish citizenship.
|
|
|
|
Returns:
|
|
Titles in Commons' own form -- ``File:Something.jpg`` -- ready to be
|
|
handed to ``commons_api.licensed_titles``. P18 gives a Special:FilePath
|
|
URL, so the name is percent-decoded and its underscores turned back
|
|
into spaces here rather than at every use.
|
|
"""
|
|
response = httpx.get(
|
|
ENDPOINT,
|
|
params={"query": QUERY, "format": "json"},
|
|
headers={"User-Agent": user_agent, "Accept": "application/sparql-results+json"},
|
|
timeout=TIMEOUT,
|
|
)
|
|
response.raise_for_status()
|
|
document: dict[str, Any] = response.json()
|
|
titles: list[str] = []
|
|
for row in document["results"]["bindings"]:
|
|
name = urllib.parse.unquote(str(row["image"]["value"]).rsplit("/", 1)[-1])
|
|
titles.append("File:" + name.replace("_", " "))
|
|
return titles
|