kuhyx
71e4a6115e
Merges ~/ck3-knowledge and ~/kcd-knowledge, which were 90% duplicate code and
neither of which had ever been committed anywhere.
Works on any wiki, not just MediaWiki:
- sources.py splits the engine-specific parts (enumerate, fetch) behind a Source
protocol. MediaWikiSource is preferred whenever an api.php exists;
SitemapHtmlSource handles custom engines with no API (verified on Fextralife).
Everything downstream was already source-agnostic.
- scripts/detect_wiki.py probes /api.php, /w/api.php then /sitemap.xml and prints
the config to use. The API path and article URL vary independently — Fandom
serves /api.php but /wiki/{Title}.
- Profiles are plain env files, since config.py was already fully env-driven.
Per-wiki behaviour that must be measured rather than assumed:
- Subpage policy. On the CK3 wiki "A/B" is a mod; on the KCD wiki it is the
sequel, and rejecting subpages deletes 441 pages of wanted content.
- Noise sections. "Other languages" tables were 7% of KCD chunks and matched
English queries by repeating every title in eight languages.
- Variant tagging for wikis covering two editions, with scopes that exclude the
other edition but keep both/unknown pages.
Build performance: length bucketing in embed_efficiently is 2.7x (fastembed pads
each batch to its longest member; 184-char median vs 2000-char max). With noise
sections that is 61 min -> 20 min on a 6,140-page wiki at half the workers.
Fandom portable-infoboxes are transformed to tables before markdownify —
otherwise stat values survive but detach from their labels.
100% branch coverage, ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
138 lines
6.3 KiB
Markdown
138 lines
6.3 KiB
Markdown
# wiki-kb
|
|
|
|
Turn a wiki into a local, offline, searchable knowledge base your LLM can query as an
|
|
MCP tool.
|
|
|
|
**scrape** (MediaWiki API, or sitemap + HTML extraction for wikis with no API) →
|
|
**corpus/** markdown → **index** (SQLite FTS5 + local `bge-small` embeddings) →
|
|
**hybrid search** (BM25 + cosine, fused with Reciprocal Rank Fusion) → **MCP stdio
|
|
server**.
|
|
|
|
Everything runs locally. No API keys, no cloud, no torch — `fastembed` uses ONNX on
|
|
CPU.
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
./install.sh # venv (Python 3.12) + deps
|
|
export WIKI_KB_CONTACT="you@example.com" # wiki etiquette requires a real contact
|
|
|
|
python scripts/detect_wiki.py https://some.wiki.example.com # what kind of wiki is it?
|
|
./run.sh --profile ck3 scrape # mirror (resumable, ~1 req/s)
|
|
./run.sh --profile ck3 index # build the search index
|
|
./run.sh --profile ck3 search "domain limit by rank"
|
|
./run.sh check # ruff + pytest (100% branch coverage)
|
|
```
|
|
|
|
Register with Claude Code:
|
|
|
|
```bash
|
|
WIKI_KB_NAMESPACE=ck3 claude mcp add -s user ck3 -- "$PWD/.venv/bin/python" -m wikikb.server
|
|
```
|
|
|
|
## Any wiki, not just MediaWiki
|
|
|
|
`scripts/detect_wiki.py <host>` probes `/api.php`, then `/w/api.php`, then
|
|
`/sitemap.xml`, and prints the source kind, endpoint, article URL pattern, engine,
|
|
page count, licence, and the env vars to configure with. Verified against three
|
|
engines:
|
|
|
|
| Host | Source | Endpoint | Article URL |
|
|
|---|---|---|---|
|
|
| ck3.paradoxwikis.com | mediawiki | `/api.php` | `{host}/{title}` |
|
|
| kingdom-come-deliverance.fandom.com | mediawiki | `/api.php` | `{host}/wiki/{title}` |
|
|
| kingdomcomedeliverance2.wiki.fextralife.com | **html** | `/sitemap.xml` | `{host}/{Title+With+Plus}` |
|
|
|
|
Note the API path and the article URL vary *independently* — Fandom serves the API at
|
|
`/api.php` but articles at `/wiki/{Title}`. Hardcoding one layout breaks on the next
|
|
wiki.
|
|
|
|
An API is always preferred: it gives a real page list, server-rendered HTML with
|
|
templates expanded, categories, revision ids, and `maxlag` so the server can push
|
|
back. The HTML fallback genuinely gives up all of those — `detect_wiki.py` says so
|
|
rather than pretending otherwise.
|
|
|
|
## Profiles
|
|
|
|
`config.py` reads every setting from `WIKI_KB_*` environment variables, so a profile
|
|
is just an env file in `profiles/` — no plugin system. Each sets its own
|
|
`WIKI_KB_ROOT`, which is what keeps several wikis' corpora and indexes apart.
|
|
|
|
See `profiles/README.md`. Shipped: `ck3`, `kcd`, `kcd-fextralife`.
|
|
|
|
## Things that are per-wiki and must be measured, not assumed
|
|
|
|
- **Are subpages noise?** On the CK3 wiki `A/B` means a total-conversion mod (reject).
|
|
On the KCD wiki it means *the sequel* — `Alchemy/KCD2` — and rejecting subpages
|
|
deletes 441 pages of exactly the content you want (keep). One API pass decides it:
|
|
`Counter(t.rsplit("/", 1)[-1] for t in titles if "/" in t)`.
|
|
- **Which sections are boilerplate?** `WIKI_KB_NOISE_SECTIONS` drops them from the
|
|
index while `*_get_page` still serves the whole page. On the KCD wiki the "Other
|
|
languages" tables were 7% of chunks and matched English queries because they repeat
|
|
every page's title in eight languages.
|
|
- **Does the wiki cover two editions at once?** `WIKI_KB_VARIANTS="kcd2:KCD2,kcd1:KCD"`
|
|
tags every page and lets searches scope to one. A scope excludes the *other* edition
|
|
but always keeps pages tagged `both`/`unknown` — filtering *to* one tag is a recall
|
|
hole, since ~9% of KCD pages carry no edition category yet matter to both.
|
|
|
|
Every search hit carries its `source` and `variant`, so a result can never be silently
|
|
attributed to the wrong wiki or the wrong edition.
|
|
|
|
## Build performance
|
|
|
|
The index build is CPU-heavy. Two optimisations matter, and the first is large:
|
|
|
|
- **Length bucketing** (`indexer.embed_efficiently`) — fastembed pads every batch to
|
|
its longest member. Wiki chunks are wildly uneven (184-char median, 2000-char max,
|
|
53% under 200 chars), so most of the compute was padding. Sorting by length measured
|
|
**2.7x faster**. The sort is undone before the vectors are returned: embedding row
|
|
`i` must stay aligned with chunk rowid `i+1`.
|
|
- **Noise sections** cut ~17% of chunks outright.
|
|
|
|
Together: 61 min → 20 min on a 6,140-page wiki, at *half* the worker count.
|
|
|
|
**`nice` does not protect a running game.** Measured with a GPU-bound game: an
|
|
8-worker build dropped GPU utilisation from 96.6% to ~50% while the game's *total* CPU
|
|
barely moved — it was never denied cycles, its render thread was delayed getting them.
|
|
Core reservation did not measurably help either. The real lever is finishing sooner.
|
|
|
|
```bash
|
|
WIKI_KB_RESERVED_CORES="0-3" # cores to keep free (set "" to use all)
|
|
WIKI_KB_NICE=10
|
|
WIKI_KB_EMBED_PARALLEL=4 # 4 workers ~= 20 min on 6k pages; 8 is worse for games
|
|
```
|
|
|
|
`build_index` writes only at the end and is not resumable — make sure a build fits in
|
|
the time you have.
|
|
|
|
## Etiquette and licensing
|
|
|
|
- Descriptive User-Agent with a real contact (`WIKI_KB_CONTACT`), `maxlag`, ~1 req/s
|
|
single-flight, exponential backoff, resumable. Per
|
|
[MediaWiki API:Etiquette](https://www.mediawiki.org/wiki/API:Etiquette).
|
|
- Cloudflare bot challenges arrive as **HTTP 200 with an HTML body**, so they pass
|
|
`raise_for_status()`. They are treated as retryable. The User-Agent is never spoofed
|
|
to evade one.
|
|
- `corpus/`, `index/` and `manifest.json` are gitignored. Mirrored content belongs to
|
|
the wiki and its contributors — most are CC BY-SA. Each mirrored page keeps a link
|
|
back to its source URL. Check a site's `robots.txt` and terms before mirroring it;
|
|
`detect_wiki.py` surfaces AI-crawler directives it finds.
|
|
|
|
## Layout
|
|
|
|
```
|
|
wikikb/
|
|
config.py endpoints, paths, noise filters, variants, CPU settings — all env-driven
|
|
sources.py MediaWikiSource | SitemapHtmlSource + host detection
|
|
scraper.py politeness, noise filtering, resume, corpus + manifest
|
|
html2md.py parse-HTML -> markdown (incl. Fandom portable-infobox -> tables)
|
|
chunker.py heading-scoped, overlapping windows
|
|
embed.py fastembed bge-small, L2-normalised
|
|
indexer.py FTS5 table + row-aligned embeddings.npy
|
|
search.py BM25 (OR terms) + cosine -> RRF fusion
|
|
variant.py edition tagging for wikis covering two editions
|
|
server.py FastMCP stdio: <ns>_search / _get_page / _list_pages
|
|
profiles/ one env file per wiki
|
|
scripts/ detect_wiki.py + the three acceptance checks
|
|
```
|