mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 00:27:01 +01:00
Krzysztof kuhy Rudnicki
86565bdef6
Indexes the arch-wiki-docs dump that is already on disk and exposes it
to the knowledge-rag MCP server, so Arch questions can be asked by
symptom ("why doesn't my systemd user unit see DISPLAY under i3")
rather than by page title, with no network round-trip at query time.
The one Arch-specific MCP server in the wild fetches pages over the
network and bundles pacman operations that duplicate the local yay-mcp.
This does neither: it only searches, against a local corpus.
install.sh installs its own dependencies, wires the MCP config and
registers a pacman hook so the index follows package updates. The
generated index is machine-specific and regenerable, so it is
gitignored rather than committed.
_reindex_entry runs under the knowledge-rag interpreter, where
mcp_server is importable and this repo's packages are not. Resolving
that import with importlib.import_module states the deferral outright
instead of a function-level import statement that reads as an oversight,
and the backend failure path now re-raises as SystemExit, so the
exit-code translation is explicit rather than a swallowed exception.
Both replace lint suppressions; the package carries none.
pylint runs in its own venv, so beautifulsoup4 and markdownify are
declared there too — being in meta/requirements.txt does not make them
resolvable to the hook.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQLwdquDCNmWtvgi3Lk72J
22 lines
1.1 KiB
Python
22 lines
1.1 KiB
Python
"""Offline Arch Wiki corpus for the ``knowledge-rag`` MCP server.
|
|
|
|
The ``arch-wiki-docs`` package ships the whole ArchWiki as HTML under
|
|
``/usr/share/doc/arch-wiki/html/en``. ``knowledge-rag`` cannot index HTML --
|
|
its parser dispatch table only understands ``.md``, ``.txt``, ``.pdf`` and a
|
|
handful of code suffixes -- so this package converts each page to Markdown and
|
|
drops it into a dedicated knowledge-rag store.
|
|
|
|
Markdown specifically, not plain text: knowledge-rag chunks ``.md`` files with
|
|
a heading-aware splitter and everything else with a blind fixed-width one.
|
|
ArchWiki pages are heavily sectioned, so emitting real ``##``/``###`` headings
|
|
is what makes chunk boundaries land on section breaks instead of mid-sentence.
|
|
|
|
The other load-bearing detail is in :mod:`python_pkg.archwiki_rag.sync`: the
|
|
knowledge-rag incremental reindex decides a file is unchanged by comparing
|
|
mtime and size, not content. The wiki dump regenerates every file on each
|
|
release, so the converter must leave byte-identical pages untouched on disk or
|
|
every ``arch-wiki-docs`` upgrade would re-embed the entire corpus.
|
|
"""
|
|
|
|
from __future__ import annotations
|