kuhyx
71e4a6115e
Merges ~/ck3-knowledge and ~/kcd-knowledge, which were 90% duplicate code and
neither of which had ever been committed anywhere.
Works on any wiki, not just MediaWiki:
- sources.py splits the engine-specific parts (enumerate, fetch) behind a Source
protocol. MediaWikiSource is preferred whenever an api.php exists;
SitemapHtmlSource handles custom engines with no API (verified on Fextralife).
Everything downstream was already source-agnostic.
- scripts/detect_wiki.py probes /api.php, /w/api.php then /sitemap.xml and prints
the config to use. The API path and article URL vary independently — Fandom
serves /api.php but /wiki/{Title}.
- Profiles are plain env files, since config.py was already fully env-driven.
Per-wiki behaviour that must be measured rather than assumed:
- Subpage policy. On the CK3 wiki "A/B" is a mod; on the KCD wiki it is the
sequel, and rejecting subpages deletes 441 pages of wanted content.
- Noise sections. "Other languages" tables were 7% of KCD chunks and matched
English queries by repeating every title in eight languages.
- Variant tagging for wikis covering two editions, with scopes that exclude the
other edition but keep both/unknown pages.
Build performance: length bucketing in embed_efficiently is 2.7x (fastembed pads
each batch to its longest member; 184-char median vs 2000-char max). With noise
sections that is 61 min -> 20 min on a 6,140-page wiki at half the workers.
Fandom portable-infoboxes are transformed to tables before markdownify —
otherwise stat values survive but detach from their labels.
100% branch coverage, ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
60 lines
2.1 KiB
Python
60 lines
2.1 KiB
Python
"""Tag pages with which edition of a subject they describe.
|
|
|
|
Some wikis cover two editions in one namespace — a game and its sequel, two versions
|
|
of a product. The Kingdom Come: Deliverance wiki is the worked example: the original
|
|
game's `Alchemy` and the sequel's `Alchemy/KCD2` sit side by side, and a search that
|
|
cannot tell them apart will happily answer a question about one with the other's
|
|
numbers.
|
|
|
|
Configured by ``config.VARIANTS`` as ordered ``(name, prefix)`` pairs, most specific
|
|
first. A page matches a variant if its title ends ``/PREFIX`` or a category starts
|
|
with ``PREFIX``, ignoring categories already claimed by an earlier, more specific
|
|
prefix — that ordering is what lets ``KCD`` mean edition 1 without also matching
|
|
``KCD2``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from . import config
|
|
|
|
BOTH = "both"
|
|
UNKNOWN = "unknown"
|
|
|
|
|
|
def _matches(prefix: str, title: str, categories: list[str], more_specific: list[str]) -> bool:
|
|
if title.endswith(f"/{prefix}"):
|
|
return True
|
|
return any(
|
|
c.startswith(prefix) and not any(c.startswith(other) for other in more_specific)
|
|
for c in categories
|
|
)
|
|
|
|
|
|
def classify(title: str, categories: list[str]) -> str:
|
|
"""Return a variant name, "both", or "unknown".
|
|
|
|
``unknown`` is a real answer, not a failure: on the KCD wiki ~9% of pages carry no
|
|
edition-specific category yet remain relevant to both ("Moravia", "Codex").
|
|
Callers must treat it as in scope for every variant filter.
|
|
"""
|
|
hits = [
|
|
name
|
|
for index, (name, prefix) in enumerate(config.VARIANTS)
|
|
if _matches(prefix, title, categories, [p for _, p in config.VARIANTS[:index]])
|
|
]
|
|
if len(hits) > 1:
|
|
return BOTH
|
|
return hits[0] if hits else UNKNOWN
|
|
|
|
|
|
def matches(variant: str, scope: str) -> bool:
|
|
"""True if a page tagged ``variant`` is in scope for filter ``scope``.
|
|
|
|
``scope="any"`` returns everything. Otherwise the scope EXCLUDES the other
|
|
editions rather than filtering TO one — dropping ``both``/``unknown`` pages would
|
|
be a recall hole, not a precision win.
|
|
"""
|
|
if scope == "any":
|
|
return True
|
|
return variant in (scope, BOTH, UNKNOWN)
|