testsAndMisc/python_pkg/archwiki_rag/README.md

archwiki_rag

Offline Arch Wiki corpus for the knowledge-rag MCP server. Gives Claude semantic search across the whole ArchWiki without a network round-trip, so questions can be asked by symptom rather than by page title:

why doesn't my systemd user unit see DISPLAY under i3

Why this exists

The only Arch-specific MCP server in the wild (nihalxkumar/arch-mcp) fetches wiki pages over the network and bundles pacman install/remove/update, PKGBUILD checks and system stats that duplicate the local yay-mcp. This package uses the arch-wiki-docs dump that is already on disk instead, and does nothing but search.

Usage

# One-time setup: dependencies, corpus, MCP config, pacman hook.
./install.sh

# Manual refresh.
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.archwiki_rag sync --reindex

# Query it.
claude-archwiki

Then, in the session, search_knowledge("...") against the archwiki server.

Design notes

Three non-obvious constraints shaped this package. All three come from reading the knowledge-rag source rather than its docs.

Markdown, not text. knowledge-rag chunks .md files with a heading-aware splitter and everything else with a blind fixed-width one. ArchWiki pages are heavily sectioned, so emitting real ##/### headings is what makes chunk boundaries land on section breaks instead of mid-sentence. HTML is not a supported suffix at all, so conversion is mandatory regardless.

Never rewrite an unchanged file. The incremental reindex decides a document is current by comparing stored mtime and size, not content. arch-wiki-docs regenerates all ~2500 files on every release, so writing unconditionally would bump every mtime and re-embed the entire corpus on each upgrade. sync.py compares rendered output against what is on disk and skips identical pages; this is the single most load-bearing behaviour in the package, and test_sync.py asserts it directly.

Subpages are nested. Systemd/User lives at en/Systemd/User.html, not en/Systemd_User.html. A non-recursive glob silently misses 164 pages, many of them the most useful ones. iter_pages uses rglob and the documents tree mirrors the dump's layout.

Store isolation

The corpus lives in its own store, ~/.local/share/knowledge-rag-archwiki, not the shared ~/.local/share/knowledge-rag. Roughly 27k Arch chunks would otherwise dominate both the BM25 index and the embedding space of any other corpus kept there. claude-rag and claude-archwiki are separate sessions against separate indexes.

Refresh guards

The pacman hook fires unattended, so reindex.py refuses to run when either:

• a claude-archwiki session already holds the store's instance lock (knowledge-rag's own lock is taken only by its main(), so importing the orchestrator directly would bypass it), or
• the machine is already under load (1-minute load average above 1.5 per core).

The load guard, not a GPU one: knowledge-rag ships the CPU-only onnxruntime build, whose execution providers are AzureExecutionProvider and CPUExecutionProvider. There is no CUDA provider, so despite the 3090 in this box the embedding pass runs on the CPU and saturates roughly every core for several minutes. A VRAM check would have looked reassuring and never fired.

Deferring is safe: the converted Markdown is already on disk, so the next run picks up exactly where this one stopped.