testsAndMisc/python_pkg/archwiki_rag
Krzysztof kuhy Rudnicki e9ba82ed66
Some checks are pending
md-naming / markdown naming convention (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (mypy, mypy) (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (pylint, pylint) (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (rest, ) (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (shellcheck, shellcheck) (push) Waiting to run
Python tests / test (push) Waiting to run
repo-contract / repo contract (push) Waiting to run
Shell tests / shell-tests (push) Waiting to run
Shell tests / lib-tests (push) Waiting to run
Shell tests / arch-tests (push) Waiting to run
Put the kuhy credit on every python_pkg CLI
Twelve interactive CLIs end their --help on credit_line() from
kuhy-credit v0.3.1 (android_ui, workout_firebase_flow, app_icons,
archwiki_rag, artgate, code_tutor, js_anki, morning_routine,
phone_lease, random_jpg, sql_drills, vscode_optimizer). The repo has
no GUI, web app or game, so --help is the only surface.

morning_routine's unit runs /usr/bin/python, so its install.sh now
installs the requirements.txt pin there. Stdio/daemon entry points
stay without it; the evidence file lists each with the reason.

Also bumps Flutter to 3.50.1 and the stale Python pins.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 10:17:38 +01:00
..
tests Fix a test that only passed in a directory named testsAndMisc 2026-08-23 14:53:29 +02:00
__init__.py archwiki_rag: offline semantic search over the Arch Wiki 2026-08-10 15:33:18 +02:00
__main__.py Re-point paths for the ~ reorganisation (home-tidy) 2026-09-11 20:50:19 +02:00
_reindex_entry.py archwiki_rag: offline semantic search over the Arch Wiki 2026-08-10 15:33:18 +02:00
cli.py Put the kuhy credit on every python_pkg CLI 2026-11-21 10:17:38 +01:00
constants.py archwiki_rag: offline semantic search over the Arch Wiki 2026-08-10 15:33:18 +02:00
convert.py archwiki_rag: offline semantic search over the Arch Wiki 2026-08-10 15:33:18 +02:00
install.sh archwiki_rag: offline semantic search over the Arch Wiki 2026-08-10 15:33:18 +02:00
README.md Re-point paths for the ~ reorganisation (home-tidy) 2026-09-11 20:50:19 +02:00
reindex.py Grade against 3.14 too, not just run on it 2026-08-29 11:50:39 +02:00
sync.py archwiki_rag: offline semantic search over the Arch Wiki 2026-08-10 15:33:18 +02:00

archwiki_rag

Offline Arch Wiki corpus for the knowledge-rag MCP server. Gives Claude semantic search across the whole ArchWiki without a network round-trip, so questions can be asked by symptom rather than by page title:

why doesn't my systemd user unit see DISPLAY under i3

Why this exists

The only Arch-specific MCP server in the wild (nihalxkumar/arch-mcp) fetches wiki pages over the network and bundles pacman install/remove/update, PKGBUILD checks and system stats that duplicate the local yay-mcp. This package uses the arch-wiki-docs dump that is already on disk instead, and does nothing but search.

Usage

# One-time setup: dependencies, corpus, MCP config, pacman hook.
./install.sh

# Manual refresh.
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.archwiki_rag sync --reindex

# Query it.
claude-archwiki

Then, in the session, search_knowledge("...") against the archwiki server.

Design notes

Three non-obvious constraints shaped this package. All three come from reading the knowledge-rag source rather than its docs.

Markdown, not text. knowledge-rag chunks .md files with a heading-aware splitter and everything else with a blind fixed-width one. ArchWiki pages are heavily sectioned, so emitting real ##/### headings is what makes chunk boundaries land on section breaks instead of mid-sentence. HTML is not a supported suffix at all, so conversion is mandatory regardless.

Never rewrite an unchanged file. The incremental reindex decides a document is current by comparing stored mtime and size, not content. arch-wiki-docs regenerates all ~2500 files on every release, so writing unconditionally would bump every mtime and re-embed the entire corpus on each upgrade. sync.py compares rendered output against what is on disk and skips identical pages; this is the single most load-bearing behaviour in the package, and test_sync.py asserts it directly.

Subpages are nested. Systemd/User lives at en/Systemd/User.html, not en/Systemd_User.html. A non-recursive glob silently misses 164 pages, many of them the most useful ones. iter_pages uses rglob and the documents tree mirrors the dump's layout.

Store isolation

The corpus lives in its own store, ~/.local/share/knowledge-rag-archwiki, not the shared ~/.local/share/knowledge-rag. Roughly 27k Arch chunks would otherwise dominate both the BM25 index and the embedding space of any other corpus kept there. claude-rag and claude-archwiki are separate sessions against separate indexes.

Refresh guards

The pacman hook fires unattended, so reindex.py refuses to run when either:

  • a claude-archwiki session already holds the store's instance lock (knowledge-rag's own lock is taken only by its main(), so importing the orchestrator directly would bypass it), or
  • the machine is already under load (1-minute load average above 1.5 per core).

The load guard, not a GPU one: knowledge-rag ships the CPU-only onnxruntime build, whose execution providers are AzureExecutionProvider and CPUExecutionProvider. There is no CUDA provider, so despite the 3090 in this box the embedding pass runs on the CPU and saturates roughly every core for several minutes. A VRAM check would have looked reassuring and never fired.

Deferring is safe: the converted Markdown is already on disk, so the next run picks up exactly where this one stopped.