# wiki-kb

Turn a wiki into a local, offline, searchable knowledge base your LLM can query as an
MCP tool.

**scrape** (MediaWiki API, or sitemap + HTML extraction for wikis with no API) →
**corpus/** markdown → **index** (SQLite FTS5 + local `bge-small` embeddings) →
**hybrid search** (BM25 + cosine, fused with Reciprocal Rank Fusion) → **MCP stdio
server**.

Everything runs locally. No API keys, no cloud, no torch — `fastembed` uses ONNX on
CPU.

## Quick start

```bash
./install.sh                                  # venv (Python 3.12) + deps
export WIKI_KB_CONTACT="you@example.com"      # wiki etiquette requires a real contact

python scripts/detect_wiki.py https://some.wiki.example.com   # what kind of wiki is it?
./run.sh --profile ck3 scrape                 # mirror (resumable, ~1 req/s)
./run.sh --profile ck3 index                  # build the search index
./run.sh --profile ck3 search "domain limit by rank"
./run.sh check                                # ruff + pytest (100% branch coverage)
```

Register with Claude Code:

```bash
WIKI_KB_NAMESPACE=ck3 claude mcp add -s user ck3 -- "$PWD/.venv/bin/python" -m wikikb.server
```

## Any wiki, not just MediaWiki

`scripts/detect_wiki.py <host>` probes `/api.php`, then `/w/api.php`, then
`/sitemap.xml`, and prints the source kind, endpoint, article URL pattern, engine,
page count, licence, and the env vars to configure with. Verified against three
engines:

| Host | Source | Endpoint | Article URL |
|---|---|---|---|
| ck3.paradoxwikis.com | mediawiki | `/api.php` | `{host}/{title}` |
| kingdom-come-deliverance.fandom.com | mediawiki | `/api.php` | `{host}/wiki/{title}` |
| kingdomcomedeliverance2.wiki.fextralife.com | **html** | `/sitemap.xml` | `{host}/{Title+With+Plus}` |

Note the API path and the article URL vary *independently* — Fandom serves the API at
`/api.php` but articles at `/wiki/{Title}`. Hardcoding one layout breaks on the next
wiki.

An API is always preferred: it gives a real page list, server-rendered HTML with
templates expanded, categories, revision ids, and `maxlag` so the server can push
back. The HTML fallback genuinely gives up all of those — `detect_wiki.py` says so
rather than pretending otherwise.

## Profiles

`config.py` reads every setting from `WIKI_KB_*` environment variables, so a profile
is just an env file in `profiles/` — no plugin system. Each sets its own
`WIKI_KB_ROOT`, which is what keeps several wikis' corpora and indexes apart.

See `profiles/README.md`. Shipped: `ck3`, `kcd`, `kcd-fextralife`.

## Things that are per-wiki and must be measured, not assumed

- **Are subpages noise?** On the CK3 wiki `A/B` means a total-conversion mod (reject).
  On the KCD wiki it means *the sequel* — `Alchemy/KCD2` — and rejecting subpages
  deletes 441 pages of exactly the content you want (keep). One API pass decides it:
  `Counter(t.rsplit("/", 1)[-1] for t in titles if "/" in t)`.
- **Which sections are boilerplate?** `WIKI_KB_NOISE_SECTIONS` drops them from the
  index while `*_get_page` still serves the whole page. On the KCD wiki the "Other
  languages" tables were 7% of chunks and matched English queries because they repeat
  every page's title in eight languages.
- **Does the wiki cover two editions at once?** `WIKI_KB_VARIANTS="kcd2:KCD2,kcd1:KCD"`
  tags every page and lets searches scope to one. A scope excludes the *other* edition
  but always keeps pages tagged `both`/`unknown` — filtering *to* one tag is a recall
  hole, since ~9% of KCD pages carry no edition category yet matter to both.

Every search hit carries its `source` and `variant`, so a result can never be silently
attributed to the wrong wiki or the wrong edition.

## Build performance

The index build is CPU-heavy. Two optimisations matter, and the first is large:

- **Length bucketing** (`indexer.embed_efficiently`) — fastembed pads every batch to
  its longest member. Wiki chunks are wildly uneven (184-char median, 2000-char max,
  53% under 200 chars), so most of the compute was padding. Sorting by length measured
  **2.7x faster**. The sort is undone before the vectors are returned: embedding row
  `i` must stay aligned with chunk rowid `i+1`.
- **Noise sections** cut ~17% of chunks outright.

Together: 61 min → 20 min on a 6,140-page wiki, at *half* the worker count.

**`nice` does not protect a running game.** Measured with a GPU-bound game: an
8-worker build dropped GPU utilisation from 96.6% to ~50% while the game's *total* CPU
barely moved — it was never denied cycles, its render thread was delayed getting them.
Core reservation did not measurably help either. The real lever is finishing sooner.

```bash
WIKI_KB_RESERVED_CORES="0-3"   # cores to keep free (set "" to use all)
WIKI_KB_NICE=10
WIKI_KB_EMBED_PARALLEL=4       # 4 workers ~= 20 min on 6k pages; 8 is worse for games
```

`build_index` writes only at the end and is not resumable — make sure a build fits in
the time you have.

## Etiquette and licensing

- Descriptive User-Agent with a real contact (`WIKI_KB_CONTACT`), `maxlag`, ~1 req/s
  single-flight, exponential backoff, resumable. Per
  [MediaWiki API:Etiquette](https://www.mediawiki.org/wiki/API:Etiquette).
- Cloudflare bot challenges arrive as **HTTP 200 with an HTML body**, so they pass
  `raise_for_status()`. They are treated as retryable. The User-Agent is never spoofed
  to evade one.
- `corpus/`, `index/` and `manifest.json` are gitignored. Mirrored content belongs to
  the wiki and its contributors — most are CC BY-SA. Each mirrored page keeps a link
  back to its source URL. Check a site's `robots.txt` and terms before mirroring it;
  `detect_wiki.py` surfaces AI-crawler directives it finds.

## Layout

```
wikikb/
  config.py     endpoints, paths, noise filters, variants, CPU settings — all env-driven
  sources.py    MediaWikiSource | SitemapHtmlSource + host detection
  scraper.py    politeness, noise filtering, resume, corpus + manifest
  html2md.py    parse-HTML -> markdown (incl. Fandom portable-infobox -> tables)
  chunker.py    heading-scoped, overlapping windows
  embed.py      fastembed bge-small, L2-normalised
  indexer.py    FTS5 table + row-aligned embeddings.npy
  search.py     BM25 (OR terms) + cosine -> RRF fusion
  variant.py    edition tagging for wikis covering two editions
  server.py     FastMCP stdio: <ns>_search / _get_page / _list_pages
profiles/       one env file per wiki
scripts/        detect_wiki.py + the three acceptance checks
```