mirror of
https://github.com/kuhyx/signal-bot.git
synced 2026-11-10 19:17:02 +01:00
feat: a word with two meanings gets the one made of steel
!pułapka (and !trap, !sidła, !wnyk, !potrzask, 26 spellings in all) posts a photograph of the literal device -- a bear trap, a mousetrap, a snare, a fish weir -- captioned with what Commons says it is. The pool is 2218 Commons page ids, harvested by tool/gen_traps.py from a curated seed list and committed. Page ids and not filenames on purpose: a filename carries photographers' and sitters' names, and one of those colliding with a label somebody mints from the chat would wedge check_personal_data.py permanently, over data nobody can edit by hand. _repo_vocabulary() subtracts aliases.json and nothing else, so it would not have caught it. An integer cannot carry a name. Nothing is cached. !papież caches because 267 portraits are a closed set a group works through; a uniform draw from 2218 warms a cache essentially never, so this fetches per call the way !kot and !pies do. What the harvest measured, and now enforces: - five plausible seed categories do not exist (Snares, Man traps, Hunting traps, Wasp traps, Animal trapping). A missing category is an empty answer, not an error, so verify_seeds() asks Commons before harvesting and fails loudly naming all of them. - depth three from Category:Fish traps reaches Alfred Sisley's net paintings. Depth one, from seeds picked by hand. - "<thing> in art" categories are paintings: Mousetraps in art put a Robert Campin altarpiece in the pool. Pruned by pattern, since Commons mints a new one whenever somebody needs it. Booby traps, the Wolfsangel, camera traps and animals dying in traps are pruned by name. - Commons rate-limits hard, so a partial harvest must not be committed silently: 429 backs off, and a short pool refuses to write at all. Descriptions are HTML and sometimes say nothing, so the tags come out, the Fisheries records' trailing catalogue list is cut, a description naming only the country falls back to the filename, and the file's own page is always linked. !łapka is not an alias: it resolves to !papa at distance two, which test_nearest.py fails on, and rightly -- the suggestion would be a coin toss. signal_bot/commons.py is the imageinfo query both Commons commands now share; portraits.py keeps its behaviour and its tests unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01416TegJ3sa8xL5qGGgrygP
This commit is contained in:
parent
262b874bd9
commit
14d422295f
@ -15,6 +15,7 @@ This file is the trigger reference — every spelling that resolves, which
|
||||
| cat | `!kot` `!koty` `!kots` `!cat` `!cats` `!meow` `!miau` + styled variants |
|
||||
| dog | `!pies` `!psy` `!dog` `!dogs` `!woof` `!szczek` + styled variants |
|
||||
| gecko | `!gekon` `!gekony` `!geko` `!gecko` `!geckon` `!geckos` |
|
||||
| trap | `!pułapka` `!pulapka` `!pułapki` `!pulapki` `!pułapkę` `!pulapke` `!trap` `!traps` `!trapy` `!trapper` `!sidła` `!sidla` `!sidło` `!sidlo` `!wnyk` `!wnyki` `!potrzask` `!zasadzka` `!klatka` `!samołówka` `!samolowka` `!beartrap` `!bear_trap` `!snare` `!snares` `!mousetrap` |
|
||||
| pope | `!papież` `!papiez` `!papieze` `!papieża` `!papa` `!pope` `!popes` `!pontifex` + styled variants |
|
||||
| stats_today | `!staty` `!statystyki` `!stats` `!dziś` `!dzis` |
|
||||
| stats_yesterday | `!wczoraj` `!yesterday` |
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
# signal-bot
|
||||
|
||||
A Signal group bot: `!cat` / `!dog` / `!gekon` image commands, `!papież`, and
|
||||
group message statistics. The group's own data stays out: `DOCS-privacy.md`.
|
||||
A Signal group bot: `!cat` / `!dog` / `!gekon` / `!pułapka` image commands,
|
||||
`!papież`, and group statistics. The group's own data stays out: `DOCS-privacy.md`.
|
||||
|
||||
Runs on its **own registered Signal number** (or, if you prefer, as a linked
|
||||
device on an existing account) via
|
||||
|
||||
5
fake_signal/scenarios/trap_command.yaml
Normal file
5
fake_signal/scenarios/trap_command.yaml
Normal file
@ -0,0 +1,5 @@
|
||||
name: "!pułapka posts a photograph of the device, captioned with what it is"
|
||||
inject:
|
||||
- { text: "!pułapka", uuid: uuid-trap, name: Trapper }
|
||||
expect:
|
||||
- { kind: image, to: reply }
|
||||
@ -30,6 +30,34 @@
|
||||
"!geckon",
|
||||
"!geckos"
|
||||
],
|
||||
"trap": [
|
||||
"!pułapka",
|
||||
"!pulapka",
|
||||
"!pułapki",
|
||||
"!pulapki",
|
||||
"!pułapkę",
|
||||
"!pulapke",
|
||||
"!trap",
|
||||
"!traps",
|
||||
"!trapy",
|
||||
"!trapper",
|
||||
"!sidła",
|
||||
"!sidla",
|
||||
"!sidło",
|
||||
"!sidlo",
|
||||
"!wnyk",
|
||||
"!wnyki",
|
||||
"!potrzask",
|
||||
"!zasadzka",
|
||||
"!klatka",
|
||||
"!samołówka",
|
||||
"!samolowka",
|
||||
"!beartrap",
|
||||
"!bear_trap",
|
||||
"!snare",
|
||||
"!snares",
|
||||
"!mousetrap"
|
||||
],
|
||||
"pope": [
|
||||
"!papież",
|
||||
"!papiez",
|
||||
|
||||
@ -35,6 +35,7 @@ from signal_bot.stats import periods
|
||||
from signal_bot.stats.commands import register_stats_commands
|
||||
from signal_bot.stats.nightly import days_to_report, mark_reported
|
||||
from signal_bot.suggest_command import register_suggest_command
|
||||
from signal_bot.traps_command import register_trap_command
|
||||
from signal_bot.versioning import POPE
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@ -198,6 +199,7 @@ def build_bot(
|
||||
endpoints=config.image_api_urls,
|
||||
)
|
||||
register_pope_command(registry, portraits, decorate=rate_limit)
|
||||
register_trap_command(registry, fetcher, config.commons_api_url, decorate=rate_limit)
|
||||
register_stats_commands(registry, decorate=rate_limit)
|
||||
# Rate-limited like every other command, and unlike !add_photo, whose
|
||||
# "NO CAP, NO LIMITS" is a deliberate exception rather than a default.
|
||||
|
||||
126
signal_bot/commons.py
Normal file
126
signal_bot/commons.py
Normal file
@ -0,0 +1,126 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
||||
"""What a Wikimedia Commons image fetch needs, shared by its two callers.
|
||||
|
||||
`!papież` names a file by title and caches the result; `!pułapka` names one by
|
||||
page id and does not cache anything. Everything between those two ends -- the
|
||||
query, the width, the place `thumburl` turns up in the answer -- is identical,
|
||||
so it lives here rather than twice.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import html
|
||||
import re
|
||||
from typing import Any, Final
|
||||
|
||||
from signal_bot.http_client import ImageFetchError
|
||||
|
||||
# Where imageinfo puts the thumbnail URL in its reply. The same path whether
|
||||
# the file was named by title or by page id.
|
||||
THUMB_PATH: Final = ("query", "pages", 0, "imageinfo", 0, "thumburl")
|
||||
# Commons hosts these four for the 267 portraits: 245 jpg, 15 png, 6 gif,
|
||||
# 1 jpeg. An unknown suffix is refused rather than guessed, because the
|
||||
# extension is what tells Signal how to render the attachment.
|
||||
SUFFIXES: Final = frozenset({".jpg", ".jpeg", ".png", ".gif"})
|
||||
|
||||
|
||||
# Long enough to say what the thing is, short enough not to bury the photo.
|
||||
CAPTION_LIMIT: Final = 220
|
||||
_TAGS: Final = re.compile(r"<[^>]+>")
|
||||
# The Fisheries records put their prose first and then an HTML list of
|
||||
# catalogue fields ("Subject: Fishing nets", "Tag: Fisheries Techniques").
|
||||
# Cutting at the list keeps the sentence and drops the filing cabinet.
|
||||
_LIST: Final = "<ul"
|
||||
# Commons templates that fill the description field with something that is
|
||||
# not a description. Measured on a hand-woven Maltese fish trap: its whole
|
||||
# description named only the country, where the filename said what the
|
||||
# object was. Treated as no description at all.
|
||||
_BOILERPLATE: Final = ("this photo has been taken in the country",)
|
||||
|
||||
|
||||
def imageinfo_params(
|
||||
width: int,
|
||||
*,
|
||||
title: str | None = None,
|
||||
page_id: int | None = None,
|
||||
describe: bool = False,
|
||||
) -> dict[str, str]:
|
||||
"""Build the imageinfo query for one Commons file.
|
||||
|
||||
Args:
|
||||
width: Thumbnail width to ask for, in pixels.
|
||||
title: Full page title, "File:" prefix included.
|
||||
page_id: Commons page id. Preferred where the caller stores a list,
|
||||
because an integer cannot carry somebody's name into the
|
||||
repository the way a filename can.
|
||||
describe: Also ask for the file's description, so one request
|
||||
answers both what the picture is and where it lives.
|
||||
|
||||
Returns:
|
||||
Query parameters, ready for ImageFetcher.fetch_bytes.
|
||||
|
||||
Raises:
|
||||
ValueError: If both or neither of title and page_id were given.
|
||||
Silently preferring one would make a caller's typo look like a
|
||||
missing file, which is the same shape as a real 404.
|
||||
"""
|
||||
if (title is None) == (page_id is None):
|
||||
msg = "name a Commons file by exactly one of title or page_id"
|
||||
raise ValueError(msg)
|
||||
selector = {"titles": title} if title is not None else {"pageids": str(page_id)}
|
||||
described = (
|
||||
{"iiprop": "url|extmetadata", "iiextmetadatafilter": "ImageDescription"}
|
||||
if describe
|
||||
else {"iiprop": "url"}
|
||||
)
|
||||
return {
|
||||
"action": "query",
|
||||
"format": "json",
|
||||
"formatversion": "2",
|
||||
"prop": "imageinfo",
|
||||
"iiurlwidth": str(width),
|
||||
**described,
|
||||
**selector,
|
||||
}
|
||||
|
||||
|
||||
def page_of(document: object) -> dict[str, Any]:
|
||||
"""Return the single page an imageinfo answer describes.
|
||||
|
||||
Raises:
|
||||
ImageFetchError: If Commons answered without one -- a deleted or
|
||||
renumbered page id comes back as `{"missing": true}` and has no
|
||||
imageinfo at all.
|
||||
"""
|
||||
pages = document.get("query", {}).get("pages", []) if isinstance(document, dict) else []
|
||||
if not pages or not pages[0].get("imageinfo"):
|
||||
msg = f"Commons named no image: {document!r:.200}"
|
||||
raise ImageFetchError(msg)
|
||||
page: dict[str, Any] = pages[0]
|
||||
return page
|
||||
|
||||
|
||||
def caption_for(page: dict[str, Any]) -> str:
|
||||
"""Return the line that rides along with the photograph.
|
||||
|
||||
Commons descriptions are HTML, sometimes several languages deep, and
|
||||
sometimes only the filename repeated back. So the tags come out, the
|
||||
catalogue list is cut, and a description that says nothing falls back to
|
||||
the title -- but the file's own page is always linked, because that is
|
||||
what answers "what am I even looking at" when the rest does not.
|
||||
|
||||
The text is fetched at run time and posted to the group; none of it is
|
||||
ever written into this repository, which is why it may be free text
|
||||
where traps.json may not.
|
||||
"""
|
||||
info = page["imageinfo"][0]
|
||||
raw = str(info.get("extmetadata", {}).get("ImageDescription", {}).get("value", ""))
|
||||
text = " ".join(html.unescape(_TAGS.sub(" ", raw.split(_LIST, 1)[0])).split())
|
||||
if text.lower().startswith(_BOILERPLATE):
|
||||
text = ""
|
||||
if not text:
|
||||
text = str(page.get("title", "")).removeprefix("File:").rsplit(".", 1)[0]
|
||||
if len(text) > CAPTION_LIMIT:
|
||||
text = text[:CAPTION_LIMIT].rstrip() + "…"
|
||||
url = str(info.get("descriptionurl", ""))
|
||||
return f"{text}\n{url}" if url else text
|
||||
@ -22,11 +22,12 @@ DEFAULT_GECKO_API_URL: Final = (
|
||||
"?taxon_id=371057&photos=true&quality_grade=research"
|
||||
"&photo_license=cc-by,cc-by-sa,cc0&per_page=1"
|
||||
)
|
||||
# Commons resolves a portrait *filename* to a real upload.wikimedia.org URL.
|
||||
# Special:FilePath would be shorter but answers 302, and this repo's httpx
|
||||
# client does not follow redirects; imageinfo hands back the final URL and a
|
||||
# thumbnail in one hop, which is also what keeps a 4000px original off Signal.
|
||||
# Commons resolves a file -- named by title for !papiez, by page id for
|
||||
# !pulapka -- to a real upload.wikimedia.org URL. Special:FilePath would be
|
||||
# shorter but answers 302, and this repo's httpx client does not follow
|
||||
# redirects; imageinfo hands back the final URL and a thumbnail in one hop,
|
||||
# which is also what keeps a 4000px original off Signal.
|
||||
DEFAULT_COMMONS_API_URL: Final = "https://commons.wikimedia.org/w/api.php"
|
||||
# Wide enough to look like a photograph rather than an avatar, small enough
|
||||
# that the whole 267-portrait cache stays well under 100 MB.
|
||||
PORTRAIT_WIDTH: Final = 640
|
||||
THUMB_WIDTH: Final = 640
|
||||
|
||||
@ -38,7 +38,7 @@ HEADING: Final = "Komendy"
|
||||
# Anything missing from here still gets printed, in a block of its own -- see
|
||||
# the module docstring.
|
||||
GROUPS: Final[tuple[tuple[str, tuple[str, ...]], ...]] = (
|
||||
("Obrazki", ("cat", "dog", "gecko")),
|
||||
("Obrazki", ("cat", "dog", "gecko", "trap")),
|
||||
("Papież", ("pope",)),
|
||||
("Propozycje", ("suggest",)),
|
||||
(
|
||||
|
||||
@ -65,17 +65,32 @@ class ImageFetcher:
|
||||
Raises:
|
||||
ImageFetchError: The JSON did not contain a usable URL.
|
||||
"""
|
||||
response = await self._client.get(api_url, params=params)
|
||||
response.raise_for_status()
|
||||
image_url = _walk(response.json(), url_path)
|
||||
image_url = _walk(await self.fetch_json(api_url, params), url_path)
|
||||
if rewrite is not None:
|
||||
image_url = image_url.replace(*rewrite)
|
||||
return await self.fetch_image(image_url)
|
||||
|
||||
# These bytes are never written to a path built from the remote URL:
|
||||
# the original bot named a local file after the URL's last path
|
||||
# segment and wrote it into the process CWD, which is both path
|
||||
# traversal and unbounded. portraits.py does cache to disk, but under
|
||||
# a name it derives from the pope's ordinal, never from upstream.
|
||||
async def fetch_json(self, api_url: str, params: dict[str, str] | None = None) -> object:
|
||||
"""Fetch one JSON document from an API.
|
||||
|
||||
Separate from fetch_bytes because a Commons answer carries more than
|
||||
the picture: !pułapka needs the description in the same reply, and
|
||||
asking twice would be both wasteful and a chance for the two halves
|
||||
to disagree.
|
||||
"""
|
||||
response = await self._client.get(api_url, params=params)
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
|
||||
async def fetch_image(self, image_url: str) -> bytes:
|
||||
"""Fetch the bytes at an already-resolved image URL.
|
||||
|
||||
These bytes are never written to a path built from the remote URL:
|
||||
the original bot named a local file after the URL's last path
|
||||
segment and wrote it into the process CWD, which is both path
|
||||
traversal and unbounded. portraits.py does cache to disk, but under
|
||||
a name it derives from the pope's ordinal, never from upstream.
|
||||
"""
|
||||
image = await self._client.get(image_url)
|
||||
image.raise_for_status()
|
||||
return image.content
|
||||
|
||||
@ -42,7 +42,7 @@ MEDIA_DIR: Final = "media"
|
||||
|
||||
# What Signal is told an attachment is. Closed, and checked at load time: the
|
||||
# suffix is the only thing that tells Signal how to render the file, so an
|
||||
# unknown one is refused rather than guessed. Mirrors portraits._SUFFIXES.
|
||||
# unknown one is refused rather than guessed. Mirrors commons.SUFFIXES.
|
||||
MEDIA_TYPES: Final[dict[str, str]] = {
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
|
||||
@ -12,22 +12,16 @@ from __future__ import annotations
|
||||
import base64
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Final
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from signal_bot.endpoints import PORTRAIT_WIDTH
|
||||
from signal_bot.commons import SUFFIXES, THUMB_PATH, imageinfo_params
|
||||
from signal_bot.endpoints import THUMB_WIDTH
|
||||
from signal_bot.http_client import ImageFetchError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from signal_bot.http_client import ImageFetcher
|
||||
from signal_bot.versioning import Pope
|
||||
|
||||
# Where imageinfo puts the thumbnail URL in its reply.
|
||||
_THUMB_PATH: Final = ("query", "pages", 0, "imageinfo", 0, "thumburl")
|
||||
# Commons hosts these four for the 267 portraits: 245 jpg, 15 png, 6 gif,
|
||||
# 1 jpeg. An unknown suffix is refused rather than guessed, because the
|
||||
# extension is what tells Signal how to render the attachment.
|
||||
_SUFFIXES: Final = frozenset({".jpg", ".jpeg", ".png", ".gif"})
|
||||
|
||||
|
||||
def default_cache_dir() -> Path:
|
||||
"""Return the portrait directory, preferring the one systemd hands us.
|
||||
@ -68,7 +62,7 @@ class PortraitCache:
|
||||
data in a way an integer is not.
|
||||
"""
|
||||
suffix = Path(pope.portrait).suffix.lower()
|
||||
if suffix not in _SUFFIXES:
|
||||
if suffix not in SUFFIXES:
|
||||
msg = f"portrait for #{pope.ordinal} has an unusable suffix: {pope.portrait!r}"
|
||||
raise ImageFetchError(msg)
|
||||
return self._dir / f"{pope.ordinal:03d}{suffix}"
|
||||
@ -86,16 +80,8 @@ class PortraitCache:
|
||||
|
||||
image = await self._fetcher.fetch_bytes(
|
||||
self._api_url,
|
||||
_THUMB_PATH,
|
||||
params={
|
||||
"action": "query",
|
||||
"format": "json",
|
||||
"formatversion": "2",
|
||||
"prop": "imageinfo",
|
||||
"iiprop": "url",
|
||||
"iiurlwidth": str(PORTRAIT_WIDTH),
|
||||
"titles": f"File:{pope.portrait}",
|
||||
},
|
||||
THUMB_PATH,
|
||||
params=imageinfo_params(THUMB_WIDTH, title=f"File:{pope.portrait}"),
|
||||
)
|
||||
_write(path, image)
|
||||
return image
|
||||
|
||||
2223
signal_bot/traps.json
Normal file
2223
signal_bot/traps.json
Normal file
File diff suppressed because it is too large
Load Diff
116
signal_bot/traps_command.py
Normal file
116
signal_bot/traps_command.py
Normal file
@ -0,0 +1,116 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
||||
"""The !pułapka command: a photograph of the device, and what it is.
|
||||
|
||||
The joke is literalism -- whatever the word meant to whoever typed it, what
|
||||
comes back is a bear trap, a mousetrap, a snare, a cage, a fish weir. The
|
||||
caption is not an apology for the joke, it is the answer to the next
|
||||
question: Commons has plenty of files whose subject is genuinely unguessable
|
||||
("Molchreuse", "K Pañayacu 0811"), and a newt trap nobody can identify is
|
||||
just a photograph of a basket.
|
||||
|
||||
Nothing is cached. !papież caches because 267 portraits are a closed set a
|
||||
group works through; a uniform draw from a couple of thousand traps warms a
|
||||
cache essentially never, so the disk would fill with images nobody asks for
|
||||
twice. This works the way !kot and !pies do instead: fetch, post, forget.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import secrets
|
||||
from base64 import b64encode
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Final
|
||||
|
||||
from signal_bot.animals import load_aliases
|
||||
from signal_bot.commons import caption_for, imageinfo_params, page_of
|
||||
from signal_bot.endpoints import THUMB_WIDTH
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
from signal_bot.context import CommandContext
|
||||
from signal_bot.decorators import Decorator
|
||||
from signal_bot.http_client import ImageFetcher
|
||||
from signal_bot.registry import CommandRegistry, Handler
|
||||
|
||||
COMMAND_NAME: Final = "trap"
|
||||
DESCRIPTION: Final = "losowe zdjęcie pułapki"
|
||||
TRAPS_FILE: Final = Path(__file__).with_name("traps.json")
|
||||
|
||||
|
||||
def load_traps(path: Path = TRAPS_FILE) -> tuple[int, ...]:
|
||||
"""Read the trap pool.
|
||||
|
||||
The file holds Commons page ids -- bare integers -- and that is checked
|
||||
here rather than assumed. Filenames would be free text carrying
|
||||
photographers' and sitters' names, and a name committed here collides
|
||||
sooner or later with a label somebody minted from the chat, at which
|
||||
point scripts/check_personal_data.py refuses every commit for good over
|
||||
data nobody can edit by hand. An integer cannot do that, and this is the
|
||||
guard that keeps it an integer.
|
||||
|
||||
Args:
|
||||
path: Where to read the pool from. Defaults to the shipped file.
|
||||
|
||||
Returns:
|
||||
Every Commons page id in the pool.
|
||||
|
||||
Raises:
|
||||
ValueError: If the pool is empty or holds anything but integers.
|
||||
"""
|
||||
document = json.loads(path.read_text(encoding="utf-8"))
|
||||
traps = document["traps"]
|
||||
if not traps or not all(isinstance(page, int) for page in traps):
|
||||
msg = f"{path} must hold a non-empty list of Commons page ids"
|
||||
raise ValueError(msg)
|
||||
return tuple(traps)
|
||||
|
||||
|
||||
def random_trap(traps: Sequence[int]) -> int:
|
||||
"""Return one page id, uniformly at random.
|
||||
|
||||
secrets rather than random for the reason animals.paged_url gives: the
|
||||
choice is not security-sensitive, but ruff's ALL rule set flags `random`
|
||||
and this repository forbids the noqa that would silence it.
|
||||
"""
|
||||
return traps[secrets.randbelow(len(traps))]
|
||||
|
||||
|
||||
def trap_handler(
|
||||
fetcher: ImageFetcher,
|
||||
api_url: str,
|
||||
traps: Sequence[int] | None = None,
|
||||
) -> Handler:
|
||||
"""Build the handler that posts one random trap."""
|
||||
pool = load_traps() if traps is None else tuple(traps)
|
||||
|
||||
async def handler(ctx: CommandContext) -> None:
|
||||
# One request, because the caption and the picture have to describe
|
||||
# the same file; two would be a chance for them to disagree.
|
||||
document = await fetcher.fetch_json(
|
||||
api_url,
|
||||
imageinfo_params(THUMB_WIDTH, page_id=random_trap(pool), describe=True),
|
||||
)
|
||||
page = page_of(document)
|
||||
image = await fetcher.fetch_image(str(page["imageinfo"][0]["thumburl"]))
|
||||
await ctx.reply_image(b64encode(image).decode("ascii"), caption_for(page))
|
||||
|
||||
return handler
|
||||
|
||||
|
||||
def register_trap_command(
|
||||
registry: CommandRegistry,
|
||||
fetcher: ImageFetcher,
|
||||
api_url: str,
|
||||
*,
|
||||
decorate: Decorator | None = None,
|
||||
aliases: dict[str, list[str]] | None = None,
|
||||
traps: Sequence[int] | None = None,
|
||||
) -> None:
|
||||
"""Register !pułapka under every trigger aliases.json gives it."""
|
||||
table = load_aliases() if aliases is None else aliases
|
||||
handler = trap_handler(fetcher, api_url, traps)
|
||||
if decorate is not None:
|
||||
handler = decorate(handler)
|
||||
registry.register(COMMAND_NAME, table[COMMAND_NAME], handler, DESCRIPTION)
|
||||
123
tests/test_commons.py
Normal file
123
tests/test_commons.py
Normal file
@ -0,0 +1,123 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
||||
"""The imageinfo query both Commons commands send."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from signal_bot.commons import (
|
||||
CAPTION_LIMIT,
|
||||
SUFFIXES,
|
||||
THUMB_PATH,
|
||||
caption_for,
|
||||
imageinfo_params,
|
||||
page_of,
|
||||
)
|
||||
from signal_bot.http_client import ImageFetchError
|
||||
|
||||
|
||||
def test_a_file_can_be_named_by_title() -> None:
|
||||
params = imageinfo_params(640, title="File:1868 Mausefalle.jpg")
|
||||
assert params["titles"] == "File:1868 Mausefalle.jpg"
|
||||
assert "pageids" not in params
|
||||
assert params["iiurlwidth"] == "640"
|
||||
assert params["prop"] == "imageinfo"
|
||||
|
||||
|
||||
def test_a_file_can_be_named_by_page_id() -> None:
|
||||
# The page id arrives as an int and has to reach Commons as a string;
|
||||
# sending the int would raise inside httpx rather than here.
|
||||
params = imageinfo_params(480, page_id=111419226)
|
||||
assert params["pageids"] == "111419226"
|
||||
assert "titles" not in params
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("title", "page_id"),
|
||||
[(None, None), ("File:x.jpg", 1)],
|
||||
)
|
||||
def test_naming_a_file_neither_way_or_both_ways_is_refused(
|
||||
title: str | None, page_id: int | None
|
||||
) -> None:
|
||||
# Preferring one silently would make a caller's mistake look like a 404,
|
||||
# which is the one failure shape that already exists downstream.
|
||||
with pytest.raises(ValueError, match="exactly one"):
|
||||
imageinfo_params(640, title=title, page_id=page_id)
|
||||
|
||||
|
||||
def test_the_thumbnail_path_is_where_commons_actually_puts_it() -> None:
|
||||
# Pinned because both callers walk it and neither can see it fail until
|
||||
# a real fetch returns something unwalkable.
|
||||
assert THUMB_PATH == ("query", "pages", 0, "imageinfo", 0, "thumburl")
|
||||
|
||||
|
||||
def test_the_suffix_allowlist_is_what_signal_can_render() -> None:
|
||||
assert ".jpg" in SUFFIXES
|
||||
assert ".svg" not in SUFFIXES
|
||||
|
||||
|
||||
def _page(
|
||||
description: str | None = None, *, title: str = "File:Molchreuse.JPG", url: str = ""
|
||||
) -> dict[str, object]:
|
||||
"""One imageinfo page, shaped the way Commons shapes it."""
|
||||
info: dict[str, object] = {"thumburl": "https://cdn/x.jpg"}
|
||||
if url:
|
||||
info["descriptionurl"] = url
|
||||
if description is not None:
|
||||
info["extmetadata"] = {"ImageDescription": {"value": description}}
|
||||
return {"title": title, "imageinfo": [info]}
|
||||
|
||||
|
||||
def test_asking_for_a_description_widens_the_query() -> None:
|
||||
plain = imageinfo_params(640, page_id=1)
|
||||
described = imageinfo_params(640, page_id=1, describe=True)
|
||||
assert plain["iiprop"] == "url"
|
||||
assert "iiextmetadatafilter" not in plain
|
||||
assert described["iiprop"] == "url|extmetadata"
|
||||
assert described["iiextmetadatafilter"] == "ImageDescription"
|
||||
|
||||
|
||||
def test_the_caption_strips_markup_and_links_the_page() -> None:
|
||||
page = _page("Eine <b>Molchreuse</b> & ein Korb", url="https://commons/File:M.JPG")
|
||||
assert caption_for(page) == "Eine Molchreuse & ein Korb\nhttps://commons/File:M.JPG"
|
||||
|
||||
|
||||
def test_the_catalogue_list_is_cut_and_the_sentence_kept() -> None:
|
||||
# The Fisheries records append their filing fields as an HTML list.
|
||||
page = _page("Round Fyke (Bollreuse). Prussia\n<ul><li>Subject: Fishing nets</li></ul>")
|
||||
assert caption_for(page) == "Round Fyke (Bollreuse). Prussia"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("description", [None, "", "<ul><li>Subject: only filing</li></ul>"])
|
||||
def test_a_description_that_says_nothing_falls_back_to_the_title(description: str | None) -> None:
|
||||
assert caption_for(_page(description)) == "Molchreuse"
|
||||
|
||||
|
||||
def test_a_long_description_is_cut_rather_than_burying_the_photo() -> None:
|
||||
page = _page("word " * 200)
|
||||
caption = caption_for(page)
|
||||
assert caption.endswith("…")
|
||||
# rstrip() before the ellipsis means the exact length depends on where
|
||||
# the cut landed; the ceiling is what matters.
|
||||
assert len(caption) <= CAPTION_LIMIT + 1
|
||||
|
||||
|
||||
def test_a_page_commons_could_not_find_is_an_image_fetch_error() -> None:
|
||||
# A deleted or renumbered page id comes back as {"missing": true}.
|
||||
for document in ({"query": {"pages": [{"missing": True}]}}, {"query": {"pages": []}}, {}, []):
|
||||
with pytest.raises(ImageFetchError, match="named no image"):
|
||||
page_of(document)
|
||||
|
||||
|
||||
def test_the_page_comes_back_when_commons_did_find_it() -> None:
|
||||
assert page_of({"query": {"pages": [_page("x")]}})["title"] == "File:Molchreuse.JPG"
|
||||
|
||||
|
||||
def test_a_template_that_is_not_a_description_falls_back_to_the_title() -> None:
|
||||
# Measured on a Maltese fish trap: the filename said more than Commons'
|
||||
# own description field did.
|
||||
page = _page(
|
||||
"This photo has been taken in the country: Malta",
|
||||
title="File:Hand woven fish trap.jpg",
|
||||
)
|
||||
assert caption_for(page) == "Hand woven fish trap"
|
||||
@ -33,6 +33,7 @@ from signal_bot.portraits import PortraitCache
|
||||
from signal_bot.registry import CommandRegistry
|
||||
from signal_bot.stats.commands import register_stats_commands
|
||||
from signal_bot.suggest_command import register_suggest_command
|
||||
from signal_bot.traps_command import register_trap_command
|
||||
from tests import pack_fixture
|
||||
from tests.conftest import SECOND_GROUP, WATCHED_GROUP, envelope
|
||||
|
||||
@ -64,6 +65,7 @@ def full_registry() -> CommandRegistry:
|
||||
fetcher = _offline_fetcher()
|
||||
register_animal_commands(registry, fetcher)
|
||||
register_pope_command(registry, PortraitCache(fetcher, DEFAULT_COMMONS_API_URL))
|
||||
register_trap_command(registry, fetcher, DEFAULT_COMMONS_API_URL, traps=(1,))
|
||||
register_stats_commands(registry)
|
||||
register_suggest_command(registry)
|
||||
register_help_command(registry)
|
||||
|
||||
191
tests/test_traps_command.py
Normal file
191
tests/test_traps_command.py
Normal file
@ -0,0 +1,191 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
||||
"""The !pułapka command: a photograph of the device, with nothing attached."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from signal_bot.context import CommandContext
|
||||
from signal_bot.decorators import DISPLAY_MARGIN_SECONDS, cooldown
|
||||
from signal_bot.http_client import ImageFetcher
|
||||
from signal_bot.models import parse_envelope
|
||||
from signal_bot.registry import CommandRefusedError, CommandRegistry
|
||||
from signal_bot.traps_command import (
|
||||
COMMAND_NAME,
|
||||
TRAPS_FILE,
|
||||
load_traps,
|
||||
random_trap,
|
||||
register_trap_command,
|
||||
)
|
||||
from tests.conftest import WATCHED_GROUP, envelope
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from signal_bot.config import Config
|
||||
from signal_bot.store import Store
|
||||
from signal_bot.transport_fake import FakeTransport
|
||||
|
||||
IMAGE = b"trap-bytes"
|
||||
API = "https://commons.example/w/api.php"
|
||||
ALIASES = {COMMAND_NAME: ["!pułapka", "!pulapka", "!trap"]}
|
||||
POOL = (111419226, 124481655, 151918804)
|
||||
|
||||
|
||||
def _fetcher(seen: list[httpx.URL] | None = None) -> ImageFetcher:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
if "commons" in str(request.url):
|
||||
if seen is not None:
|
||||
seen.append(request.url)
|
||||
page = {
|
||||
"title": "File:Molchreuse.JPG",
|
||||
"imageinfo": [
|
||||
{
|
||||
"thumburl": "https://cdn/trap.jpg",
|
||||
"descriptionurl": "https://commons.wikimedia.org/wiki/File:Molchreuse.JPG",
|
||||
"extmetadata": {
|
||||
"ImageDescription": {"value": "Eine <b>Molchreuse</b>"},
|
||||
},
|
||||
}
|
||||
],
|
||||
}
|
||||
return httpx.Response(200, json={"query": {"pages": [page]}})
|
||||
return httpx.Response(200, content=IMAGE)
|
||||
|
||||
return ImageFetcher(httpx.AsyncClient(transport=httpx.MockTransport(handler)))
|
||||
|
||||
|
||||
def _ctx(transport: FakeTransport, store: Store, config: Config, text: str) -> CommandContext:
|
||||
message = parse_envelope(envelope(text=text))
|
||||
assert message is not None
|
||||
return CommandContext(
|
||||
message=message,
|
||||
group_id=WATCHED_GROUP,
|
||||
transport=transport,
|
||||
store=store,
|
||||
config=config,
|
||||
command_name=COMMAND_NAME,
|
||||
now=100.0,
|
||||
day="2026-08-31",
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("trigger", ["!pułapka", "!pulapka", "!trap"])
|
||||
def test_every_alias_resolves(trigger: str) -> None:
|
||||
registry = CommandRegistry()
|
||||
register_trap_command(registry, _fetcher(), API, aliases=ALIASES, traps=POOL)
|
||||
|
||||
command = registry.resolve(trigger)
|
||||
assert command is not None
|
||||
assert command.name == COMMAND_NAME
|
||||
|
||||
|
||||
def test_the_real_alias_file_carries_the_command() -> None:
|
||||
registry = CommandRegistry()
|
||||
register_trap_command(registry, _fetcher(), API, traps=POOL)
|
||||
# The Polish spelling is the primary one !pomoc prints; the English
|
||||
# !trap is the word that prompted the command in the first place.
|
||||
assert registry.resolve("!pułapka") is not None
|
||||
assert registry.resolve("!trap") is not None
|
||||
|
||||
|
||||
async def test_it_posts_an_image_captioned_with_what_the_thing_is(
|
||||
transport: FakeTransport, store: Store, config: Config
|
||||
) -> None:
|
||||
registry = CommandRegistry()
|
||||
register_trap_command(registry, _fetcher(), API, aliases=ALIASES, traps=POOL)
|
||||
command = registry.resolve("!trap")
|
||||
assert command is not None
|
||||
|
||||
await command.handler(_ctx(transport, store, config, "!trap"))
|
||||
|
||||
sent = transport.sent[0]
|
||||
assert sent.recipient == WATCHED_GROUP
|
||||
assert sent.image_base64 is not None
|
||||
assert base64.b64decode(sent.image_base64) == IMAGE
|
||||
assert len(transport.sent) == 1
|
||||
# Commons is full of files whose subject is unguessable, so the caption
|
||||
# says what it is and links the page that says more. Tags come out.
|
||||
assert sent.caption == (
|
||||
"Eine Molchreuse\nhttps://commons.wikimedia.org/wiki/File:Molchreuse.JPG"
|
||||
)
|
||||
|
||||
|
||||
async def test_it_names_the_file_by_page_id_and_never_by_title(
|
||||
transport: FakeTransport, store: Store, config: Config
|
||||
) -> None:
|
||||
# A title would be free text carrying people's names into the repository;
|
||||
# see load_traps. This is the assertion that keeps the query on page ids.
|
||||
seen: list[httpx.URL] = []
|
||||
registry = CommandRegistry()
|
||||
register_trap_command(registry, _fetcher(seen), API, aliases=ALIASES, traps=POOL)
|
||||
command = registry.resolve("!trap")
|
||||
assert command is not None
|
||||
|
||||
await command.handler(_ctx(transport, store, config, "!trap"))
|
||||
|
||||
assert len(seen) == 1
|
||||
assert seen[0].params["pageids"] in {str(page) for page in POOL}
|
||||
assert "titles" not in seen[0].params
|
||||
|
||||
|
||||
async def test_the_cooldown_decorator_applies(
|
||||
transport: FakeTransport, store: Store, config: Config
|
||||
) -> None:
|
||||
registry = CommandRegistry()
|
||||
register_trap_command(
|
||||
registry,
|
||||
_fetcher(),
|
||||
API,
|
||||
decorate=cooldown(10, "WAIT {seconds}"),
|
||||
aliases=ALIASES,
|
||||
traps=POOL,
|
||||
)
|
||||
store.record_use(WATCHED_GROUP, "uuid-alice", COMMAND_NAME, 100.0, "2026-08-31")
|
||||
command = registry.resolve("!trap")
|
||||
assert command is not None
|
||||
|
||||
with pytest.raises(CommandRefusedError):
|
||||
await command.handler(_ctx(transport, store, config, "!trap"))
|
||||
|
||||
assert transport.texts == [f"WAIT {10 + DISPLAY_MARGIN_SECONDS}"]
|
||||
|
||||
|
||||
def test_the_draw_actually_varies() -> None:
|
||||
"""0.0.3 posted one identical gecko forever; that is what this catches."""
|
||||
drawn = {random_trap(POOL) for _ in range(20)}
|
||||
assert len(drawn) > 1
|
||||
|
||||
|
||||
def test_the_shipped_pool_is_page_ids_and_nothing_else() -> None:
|
||||
"""The personal-data invariant, as a unit test.
|
||||
|
||||
A filename here would carry photographers' and sitters' names into a
|
||||
public repository, and one of them colliding with a label somebody minted
|
||||
from the chat wedges check_personal_data.py permanently. Integers cannot.
|
||||
"""
|
||||
raw = json.loads(TRAPS_FILE.read_text(encoding="utf-8"))["traps"]
|
||||
assert raw
|
||||
assert all(isinstance(page, int) for page in raw)
|
||||
assert len(set(raw)) == len(raw)
|
||||
assert raw == sorted(raw)
|
||||
|
||||
|
||||
def test_the_shipped_pool_is_large_enough_to_be_worth_drawing_from() -> None:
|
||||
# kuhy asked for a set "as big as possible", and a regeneration that
|
||||
# silently truncated would still pass every other test here.
|
||||
assert len(load_traps()) >= 500
|
||||
|
||||
|
||||
@pytest.mark.parametrize("traps", [[], ["111419226.jpg"], [None]])
|
||||
def test_a_pool_that_is_not_page_ids_is_refused(tmp_path: Path, traps: list[object]) -> None:
|
||||
path = tmp_path / "traps.json"
|
||||
path.write_text(json.dumps({"traps": traps}), encoding="utf-8")
|
||||
|
||||
with pytest.raises(ValueError, match="page ids"):
|
||||
load_traps(path)
|
||||
216
tool/gen_traps.py
Executable file
216
tool/gen_traps.py
Executable file
@ -0,0 +1,216 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT licence.
|
||||
"""Regenerate ``signal_bot/traps.json`` from Wikimedia Commons.
|
||||
|
||||
Run this by hand; the output is committed, so the bot never searches Commons
|
||||
at the moment somebody types ``!pułapka``.
|
||||
|
||||
**The file holds page ids, not filenames, and that is load-bearing.** Commons
|
||||
filenames are free text carrying people's names -- the harvest turns up things
|
||||
like ``"Fishing-Boat with Fishermen" - Vasileios Chatzis.jpg``. Committing two
|
||||
thousand of those would eventually collide with a label somebody minted from
|
||||
the chat with ``!add_photo``, and ``scripts/check_personal_data.py`` would then
|
||||
refuse every commit for good, over data nobody can edit out by hand. An
|
||||
integer cannot carry a name. ``prop=imageinfo`` accepts ``pageids=`` and puts
|
||||
``thumburl`` at the same place ``titles=`` does, so nothing downstream cares.
|
||||
|
||||
Depth is one, and that is also measured: recursing three levels from
|
||||
*Category:Fish traps* reaches Alfred Sisley's net paintings and Hawaiian
|
||||
fisherman watercolours. Immediate subcategories only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Final
|
||||
|
||||
import httpx
|
||||
|
||||
CATEGORIES_FILE: Final = Path(__file__).with_name("trap_categories.json")
|
||||
|
||||
API: Final = "https://commons.wikimedia.org/w/api.php"
|
||||
# Wikimedia's policy requires a descriptive agent with a contact URL; the
|
||||
# generic client default is rate-limited hard and may be blocked outright.
|
||||
USER_AGENT: Final = "signal-bot traps generator (https://github.com/kuhyx/signal-bot)"
|
||||
|
||||
# The extension is what tells Signal how to render the attachment, so an
|
||||
# unknown one is dropped rather than guessed. Same set as portraits.SUFFIXES.
|
||||
SUFFIXES: Final = frozenset({".jpg", ".jpeg", ".png", ".gif"})
|
||||
|
||||
# Commons hosts non-free content too. !gekon filters licences at its query and
|
||||
# endpoints.py calls that filter load-bearing; this is the same idea, on the
|
||||
# only field Commons exposes for it. Matched case-folded, by prefix.
|
||||
LICENCES: Final = ("public domain", "cc0", "cc by", "cc-by", "no restrictions")
|
||||
|
||||
# imageinfo takes fifty page ids per request, joined with "|". A comma is
|
||||
# rejected outright ("badinteger"), which is worth writing down because the
|
||||
# error names the parameter rather than the separator.
|
||||
BATCH: Final = 50
|
||||
# A harvest smaller than this is a truncated one, and committing it would
|
||||
# quietly shrink the command's whole world.
|
||||
MINIMUM: Final = 500
|
||||
|
||||
TOO_MANY_REQUESTS: Final = 429
|
||||
RETRIES: Final = 6
|
||||
PAUSE: Final = 0.4
|
||||
|
||||
|
||||
def categories() -> tuple[tuple[str, ...], frozenset[str], tuple[str, ...]]:
|
||||
"""Return the seeds, the pruned set and the pruned patterns."""
|
||||
document = json.loads(CATEGORIES_FILE.read_text(encoding="utf-8"))
|
||||
return (
|
||||
tuple(document["seeds"]),
|
||||
frozenset(document["excluded"]),
|
||||
tuple(document["prune_patterns"]),
|
||||
)
|
||||
|
||||
|
||||
def _get(params: dict[str, str]) -> dict[str, Any]:
|
||||
"""One API call, retried with a widening pause when Commons says slow down.
|
||||
|
||||
Commons answers 429 after a few hundred requests even at a steady 0.4s,
|
||||
and a harvest that silently loses a category is worse than a slow one.
|
||||
"""
|
||||
query = {"action": "query", "format": "json", "formatversion": "2", **params}
|
||||
for attempt in range(RETRIES):
|
||||
time.sleep(PAUSE)
|
||||
response = httpx.get(API, params=query, headers={"User-Agent": USER_AGENT}, timeout=60.0)
|
||||
if response.status_code == TOO_MANY_REQUESTS:
|
||||
time.sleep(2.0**attempt)
|
||||
continue
|
||||
response.raise_for_status()
|
||||
document: dict[str, Any] = response.json()
|
||||
return document
|
||||
msg = f"Commons kept answering {TOO_MANY_REQUESTS} for {params}"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
|
||||
def _members(category: str, kind: str) -> list[dict[str, Any]]:
|
||||
"""Every member of one category, following continuations."""
|
||||
found: list[dict[str, Any]] = []
|
||||
carry_on: str | None = None
|
||||
while True:
|
||||
params = {
|
||||
"list": "categorymembers",
|
||||
"cmtitle": category,
|
||||
"cmtype": kind,
|
||||
"cmlimit": "500",
|
||||
}
|
||||
if carry_on is not None:
|
||||
params["cmcontinue"] = carry_on
|
||||
document = _get(params)
|
||||
found += document.get("query", {}).get("categorymembers", [])
|
||||
carry_on = document.get("continue", {}).get("cmcontinue")
|
||||
if carry_on is None:
|
||||
return found
|
||||
|
||||
|
||||
def verify_seeds(seeds: tuple[str, ...]) -> None:
|
||||
"""Fail before harvesting if any seed category does not exist.
|
||||
|
||||
A category that is not there is not an error to Commons -- it is an empty
|
||||
answer -- so a plausible-looking typo removes a whole subject silently.
|
||||
Five of the first draft's names were wrong this way.
|
||||
|
||||
Raises:
|
||||
RuntimeError: If any seed is missing, naming all of them at once.
|
||||
"""
|
||||
missing: list[str] = []
|
||||
for start in range(0, len(seeds), BATCH):
|
||||
document = _get({"titles": "|".join(seeds[start : start + BATCH])})
|
||||
missing += [
|
||||
str(page["title"])
|
||||
for page in document.get("query", {}).get("pages", [])
|
||||
if page.get("missing")
|
||||
]
|
||||
if missing:
|
||||
msg = f"these seed categories do not exist on Commons: {sorted(missing)}"
|
||||
raise RuntimeError(msg)
|
||||
|
||||
|
||||
def harvest() -> dict[int, str]:
|
||||
"""Page id to title, for every usable file one step from the seeds."""
|
||||
seeds, excluded, patterns = categories()
|
||||
verify_seeds(seeds)
|
||||
files: dict[int, str] = {}
|
||||
seen: set[str] = set(excluded)
|
||||
for number, seed in enumerate(seeds, start=1):
|
||||
sys.stderr.write(f"[{number}/{len(seeds)}] {seed} ({len(files)} files so far)\n")
|
||||
sys.stderr.flush()
|
||||
branch = [seed, *(sub["title"] for sub in _members(seed, "subcat"))]
|
||||
for category in branch:
|
||||
if category in seen or any(pattern in category for pattern in patterns):
|
||||
continue
|
||||
seen.add(category)
|
||||
for entry in _members(category, "file"):
|
||||
title = str(entry["title"])
|
||||
if Path(title).suffix.lower() in SUFFIXES:
|
||||
files[int(entry["pageid"])] = title
|
||||
return files
|
||||
|
||||
|
||||
def licensed(files: dict[int, str]) -> dict[int, str]:
|
||||
"""Drop everything whose licence does not permit reposting it."""
|
||||
kept: dict[int, str] = {}
|
||||
ids = sorted(files)
|
||||
sys.stderr.write(f"checking licences on {len(ids)} files\n")
|
||||
sys.stderr.flush()
|
||||
for start in range(0, len(ids), BATCH):
|
||||
chunk = ids[start : start + BATCH]
|
||||
document = _get(
|
||||
{
|
||||
"prop": "imageinfo",
|
||||
"iiprop": "extmetadata",
|
||||
"iiextmetadatafilter": "LicenseShortName",
|
||||
"pageids": "|".join(str(page) for page in chunk),
|
||||
}
|
||||
)
|
||||
for page in document.get("query", {}).get("pages", []):
|
||||
info = page.get("imageinfo") or [{}]
|
||||
short = info[0].get("extmetadata", {}).get("LicenseShortName", {}).get("value", "")
|
||||
name = str(short).casefold()
|
||||
if name.startswith(LICENCES):
|
||||
kept[int(page["pageid"])] = files[int(page["pageid"])]
|
||||
return kept
|
||||
|
||||
|
||||
def _rejection(traps: dict[int, str]) -> str:
|
||||
"""Return why this harvest must not be written, or an empty string."""
|
||||
if len(traps) < MINIMUM:
|
||||
return f"got {len(traps)} traps, expected at least {MINIMUM}"
|
||||
return ""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
"""Write traps.json, refusing to write a harvest that failed its own checks."""
|
||||
parser = argparse.ArgumentParser(description="Regenerate signal_bot/traps.json.")
|
||||
parser.add_argument(
|
||||
"--titles",
|
||||
action="store_true",
|
||||
help="print every kept title; the committed file is integers, so this is the review",
|
||||
)
|
||||
arguments = parser.parse_args()
|
||||
|
||||
traps = licensed(harvest())
|
||||
rejection = _rejection(traps)
|
||||
if rejection:
|
||||
sys.stderr.write(f"refusing to write: {rejection}\n")
|
||||
return 1
|
||||
|
||||
if arguments.titles:
|
||||
for page in sorted(traps):
|
||||
sys.stdout.write(f"{page}\t{traps[page]}\n")
|
||||
|
||||
out = Path(__file__).resolve().parent.parent / "signal_bot" / "traps.json"
|
||||
document = {"source": str(CATEGORIES_FILE.name), "traps": sorted(traps)}
|
||||
out.write_text(json.dumps(document, ensure_ascii=False, indent=1) + "\n", encoding="utf-8")
|
||||
sys.stdout.write(f"wrote {len(traps)} traps to {out}\n")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
58
tool/trap_categories.json
Normal file
58
tool/trap_categories.json
Normal file
@ -0,0 +1,58 @@
|
||||
{
|
||||
"_comment": "Which Commons categories !pulapka draws from. Data, not source, for the reason aliases.json is: changing the policy should never mean touching Python. Every seed is checked against Commons before the harvest starts -- Snares, Man traps, Hunting traps, Wasp traps and Animal trapping all look like plausible names and none of them exists, so a typo would otherwise remove a whole subject in silence. by-function and by-animal are container categories whose immediate children hold the leg-hold, deadfall and per-species traps.",
|
||||
"_excluded_comment": "Pruned as whole subtrees, never enqueued and never read. The command posts an attachment with no caption into a group chat, so this list is the only place the content of that attachment can be decided at all. Camera traps are photographs taken *by* a trap and drown the pool; booby traps reach wartime propaganda including a poster whose title carries a racial slur; the Wolfsangel is a heraldic charge that is also a neo-Nazi symbol; the mice categories are animals dying; the rest are geology, botany, paintings, architecture and bait rather than devices.",
|
||||
"seeds": [
|
||||
"Category:Traps",
|
||||
"Category:Traps by function",
|
||||
"Category:Traps by animal",
|
||||
"Category:Mousetraps",
|
||||
"Category:Rat traps",
|
||||
"Category:Mole traps",
|
||||
"Category:Bear traps",
|
||||
"Category:Leghold traps",
|
||||
"Category:Snare traps",
|
||||
"Category:Cage traps",
|
||||
"Category:Live capture traps",
|
||||
"Category:Pitfall traps",
|
||||
"Category:Trapping pits",
|
||||
"Category:Deadfall traps",
|
||||
"Category:Mantraps",
|
||||
"Category:Fish traps",
|
||||
"Category:Lobster traps",
|
||||
"Category:Crab traps",
|
||||
"Category:Eel traps",
|
||||
"Category:Fyke nets",
|
||||
"Category:Fly traps",
|
||||
"Category:Flypaper",
|
||||
"Category:Insect traps"
|
||||
],
|
||||
"excluded": [
|
||||
"Category:Animals in traps",
|
||||
"Category:Baits",
|
||||
"Category:Booby traps",
|
||||
"Category:Booby traps in the Vietnam War",
|
||||
"Category:Camera traps",
|
||||
"Category:Carnivorous plants with prey",
|
||||
"Category:Fishing bait",
|
||||
"Category:Fixed gillnets",
|
||||
"Category:Glue traps",
|
||||
"Category:Mammals in traps",
|
||||
"Category:Mice in cage traps",
|
||||
"Category:Mice in glue mousetraps",
|
||||
"Category:Mice in spring-loaded bar traps",
|
||||
"Category:Petroleum traps",
|
||||
"Category:Rodent bait stations",
|
||||
"Category:Tar baby",
|
||||
"Category:Trapdoors",
|
||||
"Category:Traps in art",
|
||||
"Category:Traps in heraldry",
|
||||
"Category:Traps of carnivorous plants",
|
||||
"Category:Videos of mice in mousetraps",
|
||||
"Category:Wolfsangel"
|
||||
],
|
||||
"prune_patterns": [
|
||||
" in art",
|
||||
" in heraldry"
|
||||
],
|
||||
"_prune_comment": "Any subcategory whose title contains one of these is pruned wholesale. Measured: Category:Mousetraps in art put a Robert Campin altarpiece panel and an 18th-century oil of a street vendor into the pool, and both arrive with no caption to explain why a painting is the answer to !pulapka. A pattern rather than a list because Commons mints a new '<thing> in art' category whenever somebody needs one."
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user