kuhyx/wiki-kb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
main
wiki-kb/NOTICE.md
kuhyx 71e4a6115e Consolidate CK3/KCD wiki knowledge bases into one generic wiki-kb 
Merges ~/ck3-knowledge and ~/kcd-knowledge, which were 90% duplicate code and
neither of which had ever been committed anywhere.

Works on any wiki, not just MediaWiki:
- sources.py splits the engine-specific parts (enumerate, fetch) behind a Source
  protocol. MediaWikiSource is preferred whenever an api.php exists;
  SitemapHtmlSource handles custom engines with no API (verified on Fextralife).
  Everything downstream was already source-agnostic.
- scripts/detect_wiki.py probes /api.php, /w/api.php then /sitemap.xml and prints
  the config to use. The API path and article URL vary independently — Fandom
  serves /api.php but /wiki/{Title}.
- Profiles are plain env files, since config.py was already fully env-driven.

Per-wiki behaviour that must be measured rather than assumed:
- Subpage policy. On the CK3 wiki "A/B" is a mod; on the KCD wiki it is the
  sequel, and rejecting subpages deletes 441 pages of wanted content.
- Noise sections. "Other languages" tables were 7% of KCD chunks and matched
  English queries by repeating every title in eight languages.
- Variant tagging for wikis covering two editions, with scopes that exclude the
  other edition but keep both/unknown pages.

Build performance: length bucketing in embed_efficiently is 2.7x (fastembed pads
each batch to its longest member; 184-char median vs 2000-char max). With noise
sections that is 61 min -> 20 min on a 6,140-page wiki at half the workers.

Fandom portable-infoboxes are transformed to tables before markdownify —
otherwise stat values survive but detach from their labels.

100% branch coverage, ruff clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-26 18:12:56 +02:00

1.0 KiB
Raw Permalink Blame History

NOTICE

This project creates a personal, offline searchable mirror of the Crusader Kings III Wiki for use as a local retrieval/RAG knowledge base.

  • Wiki text content is authored by the CK3 wiki community and is generally available under CC BY-SA 4.0. Original content and attribution belong to the wiki and its contributors.
  • Crusader Kings III and related names are trademarks of Paradox Interactive. This project is unofficial and not affiliated with or endorsed by Paradox.
  • The scraper is intended for personal, non-redistributive use. It fetches through the public MediaWiki API with a descriptive User-Agent, maxlag, single-flight ~1 req/s rate limiting, and exponential backoff — following MediaWiki API:Etiquette.
  • The generated corpus/ and index/ are not committed to git and are not meant to be republished. Each mirrored page retains a link back to its source URL.