kuhyx
71e4a6115e
Merges ~/ck3-knowledge and ~/kcd-knowledge, which were 90% duplicate code and
neither of which had ever been committed anywhere.
Works on any wiki, not just MediaWiki:
- sources.py splits the engine-specific parts (enumerate, fetch) behind a Source
protocol. MediaWikiSource is preferred whenever an api.php exists;
SitemapHtmlSource handles custom engines with no API (verified on Fextralife).
Everything downstream was already source-agnostic.
- scripts/detect_wiki.py probes /api.php, /w/api.php then /sitemap.xml and prints
the config to use. The API path and article URL vary independently — Fandom
serves /api.php but /wiki/{Title}.
- Profiles are plain env files, since config.py was already fully env-driven.
Per-wiki behaviour that must be measured rather than assumed:
- Subpage policy. On the CK3 wiki "A/B" is a mod; on the KCD wiki it is the
sequel, and rejecting subpages deletes 441 pages of wanted content.
- Noise sections. "Other languages" tables were 7% of KCD chunks and matched
English queries by repeating every title in eight languages.
- Variant tagging for wikis covering two editions, with scopes that exclude the
other edition but keep both/unknown pages.
Build performance: length bucketing in embed_efficiently is 2.7x (fastembed pads
each batch to its longest member; 184-char median vs 2000-char max). With noise
sections that is 61 min -> 20 min on a 6,140-page wiki at half the workers.
Fandom portable-infoboxes are transformed to tables before markdownify —
otherwise stat values survive but detach from their labels.
100% branch coverage, ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
51 lines
1.5 KiB
Bash
Executable File
51 lines
1.5 KiB
Bash
Executable File
#!/bin/bash
|
|
|
|
# ============================================================================
|
|
# install.sh — create an isolated venv (Python 3.12) and install wiki-kb
|
|
# ============================================================================
|
|
|
|
set -euo pipefail
|
|
|
|
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
readonly VENV_DIR="${SCRIPT_DIR}/.venv"
|
|
|
|
# Python 3.12 is pinned because fastembed/onnxruntime wheels lag on newer
|
|
# interpreters. Prefer pyenv's 3.12, fall back to any python3.12 on PATH.
|
|
find_python() {
|
|
local candidates=(
|
|
"${HOME}/.pyenv/versions/3.12.8/bin/python3.12"
|
|
"$(command -v python3.12 || true)"
|
|
)
|
|
local p
|
|
for p in "${candidates[@]}"; do
|
|
if [[ -n "$p" && -x "$p" ]]; then
|
|
printf '%s\n' "$p"
|
|
return 0
|
|
fi
|
|
done
|
|
echo "Error: python3.12 not found (need it for fastembed/onnxruntime wheels)." >&2
|
|
echo "Install it, e.g.: pyenv install 3.12.8" >&2
|
|
return 1
|
|
}
|
|
|
|
main() {
|
|
local py
|
|
py="$(find_python)"
|
|
echo "Using interpreter: $py ($("$py" --version))"
|
|
|
|
if [[ ! -d "$VENV_DIR" ]]; then
|
|
echo "Creating venv at $VENV_DIR"
|
|
"$py" -m venv "$VENV_DIR"
|
|
fi
|
|
|
|
# shellcheck disable=SC1091
|
|
source "${VENV_DIR}/bin/activate"
|
|
python -m pip install --quiet --upgrade pip
|
|
echo "Installing wiki-kb (+dev extras)..."
|
|
python -m pip install --quiet -e "${SCRIPT_DIR}[dev]"
|
|
|
|
echo "Done. Activate with: source ${VENV_DIR}/bin/activate"
|
|
}
|
|
|
|
main "$@"
|