kuhyx
71e4a6115e
Merges ~/ck3-knowledge and ~/kcd-knowledge, which were 90% duplicate code and
neither of which had ever been committed anywhere.
Works on any wiki, not just MediaWiki:
- sources.py splits the engine-specific parts (enumerate, fetch) behind a Source
protocol. MediaWikiSource is preferred whenever an api.php exists;
SitemapHtmlSource handles custom engines with no API (verified on Fextralife).
Everything downstream was already source-agnostic.
- scripts/detect_wiki.py probes /api.php, /w/api.php then /sitemap.xml and prints
the config to use. The API path and article URL vary independently — Fandom
serves /api.php but /wiki/{Title}.
- Profiles are plain env files, since config.py was already fully env-driven.
Per-wiki behaviour that must be measured rather than assumed:
- Subpage policy. On the CK3 wiki "A/B" is a mod; on the KCD wiki it is the
sequel, and rejecting subpages deletes 441 pages of wanted content.
- Noise sections. "Other languages" tables were 7% of KCD chunks and matched
English queries by repeating every title in eight languages.
- Variant tagging for wikis covering two editions, with scopes that exclude the
other edition but keep both/unknown pages.
Build performance: length bucketing in embed_efficiently is 2.7x (fastembed pads
each batch to its longest member; 184-char median vs 2000-char max). With noise
sections that is 61 min -> 20 min on a 6,140-page wiki at half the workers.
Fandom portable-infoboxes are transformed to tables before markdownify —
otherwise stat values survive but detach from their labels.
100% branch coverage, ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
23 lines
1.1 KiB
Bash
23 lines
1.1 KiB
Bash
# Kingdom Come: Deliverance wiki (Fandom) — MediaWiki, articles at {host}/wiki/{Title}.
|
|
# Covers BOTH games in one namespace; the sequel's pages are "/KCD2" subpages.
|
|
WIKI_KB_HOST=https://kingdom-come-deliverance.fandom.com
|
|
WIKI_KB_API_URL=https://kingdom-come-deliverance.fandom.com/api.php
|
|
WIKI_KB_PAGE_URL=https://kingdom-come-deliverance.fandom.com/wiki/{title}
|
|
WIKI_KB_NAMESPACE=kcd
|
|
WIKI_KB_ROOT=$HOME/.local/share/wiki-kb/kcd
|
|
|
|
# Subpages are STRUCTURAL here ("Alchemy/KCD2" is the sequel's Alchemy page), so
|
|
# rejecting them outright would delete all 441 KCD2 pages. Drop only "/loc", which
|
|
# holds the same in-game book text repeated in ~8 languages (371 pages, ~32k chars each).
|
|
WIKI_KB_SUBPAGES=keep
|
|
WIKI_KB_NOISE_TAILS=loc
|
|
WIKI_KB_MOD_PREFIXES=
|
|
|
|
# Per-page tables of localized UI names: 7% of chunks, and they match English queries
|
|
# because they repeat every page's title in every language.
|
|
WIKI_KB_NOISE_SECTIONS=other languages
|
|
|
|
# Two-edition wiki: tag each page kcd1/kcd2/both/unknown and default searches to kcd2.
|
|
WIKI_KB_VARIANTS=kcd2:KCD2,kcd1:KCD
|
|
WIKI_KB_DEFAULT_VARIANT=kcd2
|