testsAndMisc/docs/superpowers/evidence/archwiki-rag-corpus-2026-07-19.json

{
  "intent": "Give Claude offline semantic search over the whole Arch Wiki, so questions can be asked by symptom (\"systemd user unit can't see DISPLAY under i3\") rather than by page title, with no network round-trip.",
  "scope": [
    "python_pkg/archwiki_rag/ — new package: HTML→Markdown converter, sync, reindex, CLI",
    "meta/requirements.txt — add markdownify",
    "meta/scripts/check_requirements_sync.py — bs4→beautifulsoup4 map, foreign-runtime import exclusion",
    "linux_configuration/zsh/claude-tty-fix.zsh — claude-archwiki wrapper",
    "Non-goal: any pacman read/write tooling — yay-mcp already covers that",
    "Non-goal: non-English wiki trees (~4x corpus, no expected queries)"
  ],
  "changes": [
    "convert.py renders each page to Markdown with ATX headings, because knowledge-rag chunks .md with a heading-aware splitter and everything else with a blind fixed-width one; HTML is not a supported suffix at all.",
    "sync.py writes a page only when the rendered Markdown differs from what is on disk. knowledge-rag's incremental reindex compares mtime+size, not content, and arch-wiki-docs regenerates every file per release — writing unconditionally would re-embed the whole corpus on each upgrade.",
    "iter_pages uses rglob: subpages live in nested directories (en/Systemd/User.html). A non-recursive glob silently missed 164 pages, caught by comparing counts against the dump.",
    "Corpus lives in its own store (~/.local/share/knowledge-rag-archwiki), not the shared knowledge-rag one; ~27k Arch chunks would dominate BM25 and the embedding space of any other corpus.",
    "reindex.py guards on the knowledge-rag instance lock (only taken by its main(), so a direct orchestrator import would bypass it) and on /proc/loadavg load-per-core.",
    "Load guard replaced an initial VRAM guard after discovering knowledge-rag ships CPU-only onnxruntime (providers: Azure, CPU — no CUDA), so embedding is CPU-bound and a VRAM check could never fire."
  ],
  "verification": [
    {
      "command": "python -m python_pkg.archwiki_rag sync",
      "result": "pass",
      "evidence": "converted 2501, changed 164, skipped 8 — the 164 were exactly the newly-discovered subpages; the 8 skips are the md5-named stray non-English pages."
    },
    {
      "command": "python -m python_pkg.archwiki_rag sync  (immediate re-run)",
      "result": "pass",
      "evidence": "converted 2501, changed 0. Youngest .md mtime age 29.7s, i.e. no file rewritten — the incremental-reindex precondition holds."
    },
    {
      "command": "python -c 'reindex.blocking_reason(store)'  (live, machine under real load)",
      "result": "pass",
      "evidence": "/proc/loadavg 36.31 over 24 cores = 1.51/core -> 'machine is busy (load 1.51/core); deferring to avoid contention'; ignore_load=True returns None."
    },
    {
      "command": "ruff check python_pkg/archwiki_rag/ && mypy python_pkg/archwiki_rag/",
      "result": "pass",
      "evidence": "ruff: All checks passed. mypy: Success, no issues in 15 source files. Zero suppressions added."
    },
    {
      "command": "pytest python_pkg/archwiki_rag/ --cov --cov-branch",
      "result": "pass",
      "evidence": "61 passed; 100% statement and branch coverage on all 8 modules."
    },
    {
      "command": "python meta/scripts/check_requirements_sync.py --package python_pkg",
      "result": "pass",
      "evidence": "EXIT=0 after adding bs4->beautifulsoup4 and excluding mcp_server as a foreign-runtime import."
    },
    {
      "command": "end-to-end search_knowledge() against the built index",
      "result": "NOT YET RUN",
      "evidence": "The initial embedding pass was stopped at ~40% on user instruction (it held ~19 of 24 cores while they were at the machine) and rescheduled to 22:00 via `systemd-run --user --unit=archwiki-rag-index --on-calendar=22:00`. The retrieval-quality acceptance test is therefore still outstanding; the conversion half is verified above."
    }
  ],
  "risks": [
    "Initial embed is CPU-bound and long (~2501 files); the pacman hook detaches it via systemd-run so a pacman transaction is never blocked.",
    "The load guard can defer indefinitely on a permanently busy machine; the converted Markdown persists on disk, so the next run resumes with no lost work, and --ignore-load forces it.",
    "The pacman hook aborts every transaction if malformed; only Install/Upgrade/Remove Operation values are used, and it must be validated with a real pacman -S before being trusted.",
    "claude-extras globs mcp-optional/*.json and so now also loads the archwiki server.",
    "knowledge-rag calls _save_metadata() only after the full reindex loop, so a reindex killed mid-run leaves vectors in Chroma with no metadata record; the next run then re-embeds everything AND the orphan cleanup (iterating the empty _indexed_docs) never removes the old chunks, duplicating them. Observed here: 24,225 orphaned embeddings after the interrupted run. Recovery is to delete the store's data/ directory; documents/ is unaffected."
  ],
  "rollback": [
    "rm /etc/pacman.d/hooks/95-archwiki-rag.hook /usr/local/bin/archwiki-rag-refresh",
    "rm -rf ~/.local/share/knowledge-rag-archwiki ~/.claude/mcp-optional/archwiki.json",
    "git revert the commit; the shared ~/.local/share/knowledge-rag store is untouched by this change and needs no validation.",
    "After rollback, confirm `pacman -S <any small package>` still completes — that proves the hook is gone and transactions are unaffected."
  ]
}