wiki-kb/README.md

wiki-kb

Turn a wiki into a local, offline, searchable knowledge base your LLM can query as an MCP tool.

scrape (MediaWiki API, or sitemap + HTML extraction for wikis with no API) → corpus/ markdown → index (SQLite FTS5 + local bge-small embeddings) → hybrid search (BM25 + cosine, fused with Reciprocal Rank Fusion) → MCP stdio server.

Everything runs locally. No API keys, no cloud, no torch — fastembed uses ONNX on CPU.

Quick start

./install.sh                                  # venv (Python 3.12) + deps
export WIKI_KB_CONTACT="you@example.com"      # wiki etiquette requires a real contact

python scripts/detect_wiki.py https://some.wiki.example.com   # what kind of wiki is it?
./run.sh --profile ck3 scrape                 # mirror (resumable, ~1 req/s)
./run.sh --profile ck3 index                  # build the search index
./run.sh --profile ck3 search "domain limit by rank"
./run.sh check                                # ruff + pytest (100% branch coverage)

Register with Claude Code:

WIKI_KB_NAMESPACE=ck3 claude mcp add -s user ck3 -- "$PWD/.venv/bin/python" -m wikikb.server

Any wiki, not just MediaWiki

scripts/detect_wiki.py <host> probes /api.php, then /w/api.php, then /sitemap.xml, and prints the source kind, endpoint, article URL pattern, engine, page count, licence, and the env vars to configure with. Verified against three engines:

Host	Source	Endpoint	Article URL
ck3.paradoxwikis.com	mediawiki	/api.php	{host}/{title}
kingdom-come-deliverance.fandom.com	mediawiki	/api.php	{host}/wiki/{title}
kingdomcomedeliverance2.wiki.fextralife.com	html	/sitemap.xml	{host}/{Title+With+Plus}

Note the API path and the article URL vary independently — Fandom serves the API at /api.php but articles at /wiki/{Title}. Hardcoding one layout breaks on the next wiki.

An API is always preferred: it gives a real page list, server-rendered HTML with templates expanded, categories, revision ids, and maxlag so the server can push back. The HTML fallback genuinely gives up all of those — detect_wiki.py says so rather than pretending otherwise.

Profiles

config.py reads every setting from WIKI_KB_* environment variables, so a profile is just an env file in profiles/ — no plugin system. Each sets its own WIKI_KB_ROOT, which is what keeps several wikis' corpora and indexes apart.

See profiles/README.md. Shipped: ck3, kcd, kcd-fextralife.

Things that are per-wiki and must be measured, not assumed

• Are subpages noise? On the CK3 wiki A/B means a total-conversion mod (reject). On the KCD wiki it means the sequel — Alchemy/KCD2 — and rejecting subpages deletes 441 pages of exactly the content you want (keep). One API pass decides it: Counter(t.rsplit("/", 1)[-1] for t in titles if "/" in t).
• Which sections are boilerplate? WIKI_KB_NOISE_SECTIONS drops them from the index while *_get_page still serves the whole page. On the KCD wiki the "Other languages" tables were 7% of chunks and matched English queries because they repeat every page's title in eight languages.
• Does the wiki cover two editions at once? WIKI_KB_VARIANTS="kcd2:KCD2,kcd1:KCD" tags every page and lets searches scope to one. A scope excludes the other edition but always keeps pages tagged both/unknown — filtering to one tag is a recall hole, since ~9% of KCD pages carry no edition category yet matter to both.

Every search hit carries its source and variant, so a result can never be silently attributed to the wrong wiki or the wrong edition.

Build performance

The index build is CPU-heavy. Two optimisations matter, and the first is large:

• Length bucketing (indexer.embed_efficiently) — fastembed pads every batch to its longest member. Wiki chunks are wildly uneven (184-char median, 2000-char max, 53% under 200 chars), so most of the compute was padding. Sorting by length measured 2.7x faster. The sort is undone before the vectors are returned: embedding row i must stay aligned with chunk rowid i+1.
• Noise sections cut ~17% of chunks outright.

Together: 61 min → 20 min on a 6,140-page wiki, at half the worker count.

nice does not protect a running game. Measured with a GPU-bound game: an 8-worker build dropped GPU utilisation from 96.6% to ~50% while the game's total CPU barely moved — it was never denied cycles, its render thread was delayed getting them. Core reservation did not measurably help either. The real lever is finishing sooner.

WIKI_KB_RESERVED_CORES="0-3"   # cores to keep free (set "" to use all)
WIKI_KB_NICE=10
WIKI_KB_EMBED_PARALLEL=4       # 4 workers ~= 20 min on 6k pages; 8 is worse for games

build_index writes only at the end and is not resumable — make sure a build fits in the time you have.

Etiquette and licensing

• Descriptive User-Agent with a real contact (WIKI_KB_CONTACT), maxlag, ~1 req/s single-flight, exponential backoff, resumable. Per MediaWiki API:Etiquette.
• Cloudflare bot challenges arrive as HTTP 200 with an HTML body, so they pass raise_for_status(). They are treated as retryable. The User-Agent is never spoofed to evade one.
• corpus/, index/ and manifest.json are gitignored. Mirrored content belongs to the wiki and its contributors — most are CC BY-SA. Each mirrored page keeps a link back to its source URL. Check a site's robots.txt and terms before mirroring it; detect_wiki.py surfaces AI-crawler directives it finds.

Layout

wikikb/
  config.py     endpoints, paths, noise filters, variants, CPU settings — all env-driven
  sources.py    MediaWikiSource | SitemapHtmlSource + host detection
  scraper.py    politeness, noise filtering, resume, corpus + manifest
  html2md.py    parse-HTML -> markdown (incl. Fandom portable-infobox -> tables)
  chunker.py    heading-scoped, overlapping windows
  embed.py      fastembed bge-small, L2-normalised
  indexer.py    FTS5 table + row-aligned embeddings.npy
  search.py     BM25 (OR terms) + cosine -> RRF fusion
  variant.py    edition tagging for wikis covering two editions
  server.py     FastMCP stdio: <ns>_search / _get_page / _list_pages
profiles/       one env file per wiki
scripts/        detect_wiki.py + the three acceptance checks