kuhyx
71e4a6115e
Merges ~/ck3-knowledge and ~/kcd-knowledge, which were 90% duplicate code and
neither of which had ever been committed anywhere.
Works on any wiki, not just MediaWiki:
- sources.py splits the engine-specific parts (enumerate, fetch) behind a Source
protocol. MediaWikiSource is preferred whenever an api.php exists;
SitemapHtmlSource handles custom engines with no API (verified on Fextralife).
Everything downstream was already source-agnostic.
- scripts/detect_wiki.py probes /api.php, /w/api.php then /sitemap.xml and prints
the config to use. The API path and article URL vary independently — Fandom
serves /api.php but /wiki/{Title}.
- Profiles are plain env files, since config.py was already fully env-driven.
Per-wiki behaviour that must be measured rather than assumed:
- Subpage policy. On the CK3 wiki "A/B" is a mod; on the KCD wiki it is the
sequel, and rejecting subpages deletes 441 pages of wanted content.
- Noise sections. "Other languages" tables were 7% of KCD chunks and matched
English queries by repeating every title in eight languages.
- Variant tagging for wikis covering two editions, with scopes that exclude the
other edition but keep both/unknown pages.
Build performance: length bucketing in embed_efficiently is 2.7x (fastembed pads
each batch to its longest member; 184-char median vs 2000-char max). With noise
sections that is 61 min -> 20 min on a 6,140-page wiki at half the workers.
Fandom portable-infoboxes are transformed to tables before markdownify —
otherwise stat values survive but detach from their labels.
100% branch coverage, ruff clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
34 lines
1.0 KiB
Python
34 lines
1.0 KiB
Python
"""Keep heavy jobs (embedding) off reserved cores and at low priority.
|
|
|
|
The foreground game keeps its cores; this process is pinned to the rest and niced
|
|
down so it yields on any shared core. Linux-only affinity is applied when present.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
from collections.abc import Sequence
|
|
|
|
from . import config
|
|
|
|
|
|
def isolate(
|
|
*,
|
|
allowed: Sequence[int] | None = None,
|
|
nice_increment: int | None = None,
|
|
) -> list[int]:
|
|
"""Pin this process to `allowed` cores and lower its priority.
|
|
|
|
Returns the cores it was pinned to. No-ops the affinity call on platforms
|
|
without ``os.sched_setaffinity`` (non-Linux).
|
|
"""
|
|
cores = list(config.ALLOWED_CORES if allowed is None else allowed)
|
|
increment = config.NICE_INCREMENT if nice_increment is None else nice_increment
|
|
|
|
set_affinity = getattr(os, "sched_setaffinity", None)
|
|
if set_affinity is not None:
|
|
set_affinity(0, set(cores))
|
|
if increment and hasattr(os, "nice"):
|
|
os.nice(increment)
|
|
return cores
|