requirements-sync: map bs4 and exempt foreign-runtime imports
Some checks are pending
Pre-commit checks / pre-commit (push) Waiting to run
Python tests / test (push) Waiting to run

Two fixes the checker needs for archwiki_rag, left uncommitted in the
working tree and therefore stashed away by pre-commit on every push:

- bs4 resolves to the beautifulsoup4 distribution; without the mapping
  the checker reports it as undeclared.
- mcp_server is resolved by the knowledge-rag uv tool venv at runtime,
  not by CI's `pip install -r requirements.txt`. Declaring it would be
  wrong, so it is exempted via _FOREIGN_RUNTIME_IMPORTS with a comment
  naming the owning venv.

Not authored in this session; committed because the pre-push CI mirror
cannot pass while these live only in the working tree.
This commit is contained in:
Krzysztof Rudnicki 2026-07-19 18:58:35 +02:00
parent 90305e03c2
commit 1b7a364e64
2 changed files with 56 additions and 4 deletions

View File

@ -0,0 +1,34 @@
{
"intent": "Let the requirements-sync checker recognise bs4 and ignore imports resolved by a foreign runtime, so the pre-push CI mirror stops failing on the archwiki_rag package.",
"scope": [
"meta/scripts/check_requirements_sync.py — import->distribution map and a new foreign-runtime exemption set",
"Non-goal: committing python_pkg/archwiki_rag/ itself, which remains untracked and owned by another workstream",
"Non-goal: any behaviour change to packages other than how their imports are classified"
],
"changes": [
"Map bs4 -> beautifulsoup4 in _IMPORT_TO_DIST; without it the checker reports bs4 as an undeclared dependency because the import name differs from the distribution name.",
"Add _FOREIGN_RUNTIME_IMPORTS containing mcp_server, and filter it out in _third_party_imports. That module is provided by the knowledge-rag uv tool venv at runtime, not by CI's `pip install -r requirements.txt`, so declaring it in requirements.txt would be incorrect rather than merely redundant.",
"Docstring updated to state that foreign-runtime modules are filtered alongside stdlib and first-party names."
],
"verification": [
{
"command": "python3 meta/scripts/check_requirements_sync.py --package python_pkg",
"result": "pass",
"evidence": "Exits clean with the change applied. With the change stashed (which is what pre-commit does on push) it fails with 'these third-party imports are not declared in requirements.txt', which is the exact failure this commit removes."
},
{
"command": "pre-commit run --all-files",
"result": "pass",
"evidence": "Green once the repo .venv was synced to meta/requirements.txt; ruff, mypy, pylint, bandit and pytest-coverage all pass on the modified checker."
}
],
"risks": [
"_FOREIGN_RUNTIME_IMPORTS is an allowlist: any module added to it stops being checked, so a genuinely missing dependency could be masked. Mitigated by requiring each entry to carry a comment naming the venv that owns it.",
"The bs4 mapping is correct for beautifulsoup4 but the map remains manual — other import/distribution name mismatches will still need adding case by case.",
"This change was authored in a previous session, not this one. It is committed here only because the pre-push CI mirror stashes unstaged changes and therefore cannot pass while it lives solely in the working tree. It was reviewed before committing but not written as part of this task."
],
"rollback": [
"git revert this commit; the checker returns to flagging bs4 and mcp_server as undeclared.",
"Validate after rollback: python3 meta/scripts/check_requirements_sync.py --package python_pkg reproduces the original failure, confirming nothing else depended on the change."
]
}

View File

@ -40,8 +40,22 @@ _IMPORT_TO_DIST: dict[str, str] = {
"yaml": "pyyaml",
"kasa": "python-kasa",
"PIL": "pillow",
"bs4": "beautifulsoup4",
}
# Modules that are deliberately resolved by a DIFFERENT interpreter at runtime,
# not by the CI `pip install -r requirements.txt`. Adding them to
# requirements.txt would be wrong: they are not this repo's dependencies, and
# the code that imports them only ever executes under the foreign venv that
# owns them. Each entry needs a comment naming that venv.
_FOREIGN_RUNTIME_IMPORTS: frozenset[str] = frozenset(
{
# archwiki_rag._reindex_entry runs under the knowledge-rag uv tool venv
# (~/.local/share/uv/tools/knowledge-rag), which owns mcp_server.
"mcp_server",
},
)
def _normalise(name: str) -> str:
"""Return the PEP 503 canonical form of a distribution/import name."""
@ -114,16 +128,20 @@ def _first_party_names(package_dir: Path, package_name: str) -> set[str]:
def _third_party_imports(package_dir: Path, package_name: str) -> set[str]:
"""Return third-party top-level imports across the whole package.
Standard-library modules (via ``sys.stdlib_module_names``) and first-party
names (the package and its subpackages) are filtered out, leaving only
external dependencies.
Standard-library modules (via ``sys.stdlib_module_names``), first-party
names (the package and its subpackages) and foreign-runtime modules are
filtered out, leaving only genuine external dependencies.
"""
stdlib = sys.stdlib_module_names
internal = _first_party_names(package_dir, package_name)
external: set[str] = set()
for module in _iter_package_modules(package_dir):
for name in _top_level_imports(module):
if name not in internal and name not in stdlib:
if (
name not in internal
and name not in stdlib
and name not in _FOREIGN_RUNTIME_IMPORTS
):
external.add(name)
return external