Split ci-mirror gate into <60 s steps
Some checks are pending
dependency freshness / dependencies on newest stable (push) Waiting to run
md-naming / markdown naming convention (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (mypy, mypy, ) (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (pylint, pylint, ) (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (rest, , mypy,pylint) (push) Waiting to run
Tests / test (push) Waiting to run
Tests / web-coverage (push) Waiting to run
repo-contract / repo contract (push) Waiting to run

The single ci-mirror manual hook (clean venv + pre-commit --all-files +
serial pytest) ran past the 60 s per-command cap, so the tree could not be
stamped for a push. It is now nine gate steps, each under 55 s at load ~40:

- ci-mirror-venv / -precommit / -pytest: scripts/ci_mirror.sh parts;
  pytest runs on 4 xdist workers, pytest-cov still enforces 100 %.
- ci-mirror-pylint-{pkg,tests-1,tests-2,tests-3}: per-file checks over a
  partition of the pylint hook's targets; -dup and -cycles: the two
  cross-file checks on the whole tree. pylint is now a local hook so all of
  them share one env (&pylint_env, pylint==4.0.9).
- scripts/pylint_symilar_cache.py: pylint plugin that hashes each file once
  for duplicate-code instead of once per file pair (same R0801 output).

Tests: an autouse guard refuses non-loopback DNS (several tests made live
HLTB requests), two web fixtures poll at 0.01 s, and the rate-limit test
uses a fake clock. CLAUDE.md and AGENTS.md get the push line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzxdWWjUkRCFGNMnnXZrSP
This commit is contained in:
Krzysztof Rudnicki 2026-11-11 16:28:29 +01:00
parent 87c6f200e3
commit db44ca5b0b
11 changed files with 358 additions and 43 deletions

View File

@ -120,10 +120,19 @@ repos:
# ===========================================================================
# PYLINT
# ===========================================================================
- repo: https://github.com/pylint-dev/pylint
rev: v4.0.9
# A local hook (was the pylint-dev/pylint repo at v4.0.9) so the manual
# ci-mirror-pylint-* shards below can share its env: pre-commit gives local
# `language: python` hooks with the same additional_dependencies ONE env,
# and the &pylint_env anchor is the single place the pylint version and its
# import deps are pinned.
- repo: local
hooks:
- id: pylint
name: pylint
entry: pylint
language: python
types: [python]
require_serial: true
# Whole tree on every run, never the staged subset: `--fail-under`
# grades a score, and a score over whatever happened to be staged let a
# 9.99 tree pass at commit time and fail in CI. scripts/file_length/
@ -137,11 +146,13 @@ repos:
- --jobs=1
- steam_backlog_enforcer
- scripts/check_requirements_sync.py
- scripts/pylint_symilar_cache.py
# pylint resolves imports in its OWN isolated env, so anything the
# package imports has to be listed here or it reports import-error and
# drags the score under --fail-under. `mcp` is a real runtime import
# (steam_backlog_enforcer._mcp_server).
additional_dependencies:
additional_dependencies: &pylint_env
- pylint==4.0.9
- aiohttp==3.14.5
- earned-time @ git+https://github.com/kuhyx/utils@earned-time-v0.21.0#subdirectory=earned_time
- leisure-watch @ git+https://github.com/kuhyx/leisure-watch@leisure-watch-v0.2.7
@ -168,13 +179,97 @@ repos:
exclude: ^(tests/|.*test.*\.py$)
# ===========================================================================
# PYTEST + COVERAGE (push stage)
# CI MIRROR (manual stage: the gate steps before a push)
# ===========================================================================
# One gate step per hook, each under the 60 s per-command cap even on a
# loaded box: `scripts/check_gate_stamp.sh step <id>` per id of
# `scripts/check_gate_stamp.sh full --list`; the last one stamps the tree.
# The pylint hook above is split into the ci-mirror-pylint-* shards and
# ci-mirror-precommit skips it, so the union checks exactly what CI does:
# - per-file checks run per shard; the shards' `files` partition the hook's
# targets (tests by name: rest|test_[a-h], test_[i-r], test_[s-z]). With
# --fail-under=10.0 a shard passes only with zero scored messages, so
# every shard passing == the whole tree at 10.00;
# - the two cross-file checks (duplicate-code, cyclic-import) see the whole
# tree, as in the full hook (ci-mirror-pylint-dup, ci-mirror-pylint-cycles).
- repo: local
hooks:
- id: ci-mirror
name: CI mirror (clean-venv pytest + pre-commit --all-files)
entry: bash scripts/ci_mirror.sh steam_backlog_enforcer/tests/
- id: ci-mirror-venv
name: CI mirror (clean requirements.txt venv, rebuilt on change)
entry: bash scripts/ci_mirror.sh venv
language: system
pass_filenames: false
always_run: true
require_serial: true
stages: [manual]
- id: ci-mirror-precommit
name: CI mirror (pre-commit --all-files, pylint via the shards)
entry: bash scripts/ci_mirror.sh precommit
language: system
pass_filenames: false
always_run: true
require_serial: true
stages: [manual]
- &pylint_shard
id: ci-mirror-pylint-pkg
name: CI mirror pylint (package + scripts, per-file checks)
entry: pylint
language: python
additional_dependencies: *pylint_env
types: [python]
require_serial: true
stages: [manual]
files: ^(steam_backlog_enforcer/(?!tests/).*|scripts/(check_requirements_sync|pylint_symilar_cache))\.py$
args: &pylint_per_file
- --rcfile=pyproject.toml
- --fail-under=10.0
- --jobs=1
- --disable=duplicate-code,cyclic-import
- <<: *pylint_shard
id: ci-mirror-pylint-tests-1
name: CI mirror pylint (tests helpers + test_[a-h], per-file checks)
files: ^steam_backlog_enforcer/tests/(?!test_[i-z]).*\.py$
- <<: *pylint_shard
id: ci-mirror-pylint-tests-2
name: CI mirror pylint (test_[i-r], per-file checks)
files: ^steam_backlog_enforcer/tests/test_[i-r][^/]*\.py$
- <<: *pylint_shard
id: ci-mirror-pylint-tests-3
name: CI mirror pylint (test_[s-z], per-file checks)
files: ^steam_backlog_enforcer/tests/test_[s-z][^/]*\.py$
- &pylint_whole_tree
<<: *pylint_shard
id: ci-mirror-pylint-dup
name: CI mirror pylint (whole tree, duplicate-code)
files: ''
pass_filenames: false
always_run: true
args:
- --rcfile=pyproject.toml
- --fail-under=10.0
- --jobs=1
- --disable=all
# bad-plugin-value keeps a plugin that failed to load (the
# duplicate-code speed-up) loud instead of silently slow.
- --enable=duplicate-code,bad-plugin-value
- steam_backlog_enforcer
- scripts/check_requirements_sync.py
- scripts/pylint_symilar_cache.py
- <<: *pylint_whole_tree
id: ci-mirror-pylint-cycles
name: CI mirror pylint (whole tree, cyclic-import)
args:
- --rcfile=pyproject.toml
- --fail-under=10.0
- --jobs=1
- --disable=all
- --enable=cyclic-import
- steam_backlog_enforcer
- scripts/check_requirements_sync.py
- scripts/pylint_symilar_cache.py
- id: ci-mirror-pytest
name: CI mirror (clean-venv pytest, xdist, 100% coverage)
entry: bash scripts/ci_mirror.sh pytest steam_backlog_enforcer/tests/
language: system
pass_filenames: false
always_run: true

View File

@ -23,3 +23,4 @@ After running tests fix all coverage gaps and issues, do not ignore unless speci
- lint: `pre-commit run ruff --all-files`
- coverage: `python3 -m pytest -q`
- coverage-gaps: `coverage-gaps coverage.lcov`
- push: `scripts/check_gate_stamp.sh full && git push` (every manual-stage gate hook, ~3 min, then push). Agents (60 s cap): one call per id of `scripts/check_gate_stamp.sh full --list`: `CAP_MEM=4G CAP_CPU_PCT=20 ~/.claude/scripts/capped.sh timeout 55 scripts/check_gate_stamp.sh step <id>` (each < 55 s; the last stamps the tree), then `git push`.

View File

@ -24,3 +24,4 @@ After running tests fix all coverage gaps and issues, do not ignore unless speci
- coverage: `python3 -m pytest -q`
- coverage-gaps: `coverage-gaps coverage.lcov`
- deploy: `scripts/deploy.sh`
- push: `scripts/check_gate_stamp.sh full && git push` (every manual-stage gate hook, ~3 min, then push). Agents (60 s cap): one call per id of `scripts/check_gate_stamp.sh full --list`: `CAP_MEM=4G CAP_CPU_PCT=20 ~/.claude/scripts/capped.sh timeout 55 scripts/check_gate_stamp.sh step <id>` (each < 55 s; the last stamps the tree), then `git push`.

View File

@ -49,6 +49,10 @@ include = ["*.py", "**/*.py"]
# byte-identical to upstream; ruff --unsafe-fixes otherwise strips its print()
# calls (T201) and silently guts the gate's reporting.
exclude = [".git", ".venv", "__pycache__", "build", "dist", ".eggs", "scripts/file_length"]
# scripts/ has no __init__.py on purpose (one would make it a second top-level
# package for requirements-sync), yet pylint imports scripts.pylint_symilar_cache
# from it: an implicit namespace package, declared so INP001 knows.
namespace-packages = ["scripts"]
[tool.ruff.lint]
select = ["ALL"]
@ -130,6 +134,10 @@ exclude = [".venv/"]
# The repo root on the path, so pylint resolves the package's own imports the
# same way pytest does instead of reporting import-error for them.
init-hook = "import sys; sys.path.insert(0, '.')"
# Hashes each file once for duplicate-code instead of once per file PAIR; same
# R0801 output, whole-tree pylint fits the 60 s cap. Needs the init-hook above
# (pylint runs it before loading config-file plugins, never CLI ones).
load-plugins = ["scripts.pylint_symilar_cache"]
analyse-fallback-blocks = true
persistent = true
# Serial on purpose: with jobs != 1 pylint ships the linter to its workers via

View File

@ -8,16 +8,26 @@
# requirements.txt. That gap is why green-locally repos went red in CI
# (missing runtime deps, --all-files lint debt, order-dependent test flakes).
#
# This script reproduces both CI workflows on the developer machine:
# 1. a venv built ONLY from requirements.txt (mirrors the Tests workflow's
# `pip install -r requirements.txt`), rebuilt only when requirements.txt
# changes (hash-gated so day-to-day pushes stay fast);
# 2. `pre-commit run --all-files` (mirrors the pre-commit workflow);
# 3. `python -m pytest` inside that clean venv (mirrors Tests).
# This script reproduces both CI workflows on the developer machine, split
# into parts that each fit the 60 s per-command cap (one gate step each, see
# the ci-mirror-* manual hooks in .pre-commit-config.yaml):
# venv a venv built ONLY from requirements.txt (mirrors the Tests
# workflow's `uv pip install -r requirements.txt`), rebuilt only
# when requirements.txt changes (hash-gated);
# precommit `pre-commit run --all-files` minus pylint (mirrors the
# pre-commit workflow; pylint runs as the ci-mirror-pylint-*
# shards, which together check exactly what the pylint hook does);
# pytest `python -m pytest` in that clean venv (mirrors Tests), spread
# over CI_MIRROR_WORKERS xdist workers (default 4); pytest-cov
# merges the workers and enforces the 100 % gate itself.
# `all` (or no part, or a first argument that is not a part name, the old
# `ci_mirror.sh <pytest args>` form) runs the three in order with the full
# pylint hook: kuhy's terminal, over 60 s, so agents run the gate steps.
# Arguments after the part go to pytest.
#
# Runs on demand (`scripts/check_gate_stamp.sh full`, pre-commit's manual
# stage) and in CI. It left the pre-push path on 2026-09-20: 10+ minutes
# per push under the resource cap, for work finish_auto.sh had just done.
# Runs on demand (`scripts/check_gate_stamp.sh step <hook-id>`, pre-commit's
# manual stage). It left the pre-push path on 2026-09-20: 10+ minutes per push
# under the resource cap, for work finish_auto.sh had just done.
# ============================================================================
set -euo pipefail
@ -25,6 +35,7 @@ set -euo pipefail
# Requirements file may be overridden (e.g. testsAndMisc uses meta/…).
REQUIREMENTS_FILE="${REQUIREMENTS_FILE:-requirements.txt}"
readonly REQUIREMENTS_FILE
readonly WORKERS="${CI_MIRROR_WORKERS:-4}"
ROOT="$(git rev-parse --show-toplevel)"
readonly ROOT
@ -48,47 +59,87 @@ require_file() {
fi
}
# Rebuild the venv only when requirements.txt changed since the last build.
ensure_venv() {
local current stored
current="$(sha256sum "$REQ_PATH" | cut -d' ' -f1)"
stored=""
if [[ -f "$HASH_FILE" ]]; then
stored="$(cat "$HASH_FILE")"
fi
requirements_hash() {
sha256sum "$REQ_PATH" | cut -d' ' -f1
}
if [[ -x "$VENV_DIR/bin/python" && "$current" == "$stored" ]]; then
log "venv up to date (requirements.txt unchanged)"
venv_is_fresh() {
[[ -x "$VENV_DIR/bin/python" && -f "$HASH_FILE" \
&& "$(requirements_hash)" == "$(cat "$HASH_FILE")" ]]
}
# uv when present: the installer CI uses, and fast enough for one gate step.
rebuild_venv() {
rm -rf "$VENV_DIR"
if command -v uv >/dev/null 2>&1; then
uv venv --quiet --python python3 "$VENV_DIR" \
|| fail "uv venv for the clean venv"
uv pip install --quiet --python "$VENV_DIR/bin/python" -r "$REQ_PATH" \
|| fail "uv pip install -r $REQUIREMENTS_FILE (a dep may be undeclared)"
return
fi
log "requirements.txt changed — rebuilding clean venv (mirrors CI install)"
rm -rf "$VENV_DIR"
python3 -m venv "$VENV_DIR"
"$VENV_DIR/bin/python" -m pip install --quiet --upgrade pip \
|| fail "pip self-upgrade in the clean venv"
"$VENV_DIR/bin/python" -m pip install --quiet -r "$REQ_PATH" \
|| fail "pip install -r $REQUIREMENTS_FILE (a dep may be undeclared)"
printf '%s' "$current" > "$HASH_FILE"
}
# Rebuild the venv only when requirements.txt changed since the last build.
ensure_venv() {
if venv_is_fresh; then
log "venv up to date (requirements.txt unchanged)"
return
fi
log "requirements.txt changed — rebuilding clean venv (mirrors CI install)"
rebuild_venv
requirements_hash > "$HASH_FILE"
log "clean venv ready"
}
# The pytest part never rebuilds: a rebuild would not fit its budget, and a
# stale venv would test the wrong dependencies. Fail closed instead.
require_fresh_venv() {
venv_is_fresh \
|| fail "clean venv missing or stale; run the ci-mirror-venv step first"
}
run_precommit_all_files() {
log "pre-commit run --all-files (mirrors the pre-commit workflow)"
pre-commit run --all-files || fail "pre-commit --all-files"
local skip="${1:-}"
log "pre-commit run --all-files${skip:+ minus $skip} (mirrors the pre-commit workflow)"
SKIP="${SKIP:+$SKIP,}$skip" pre-commit run --all-files \
|| fail "pre-commit --all-files"
}
run_pytest_clean_venv() {
log "pytest in the clean venv (mirrors the Tests workflow)"
"$VENV_DIR/bin/python" -m pytest "$@" || fail "pytest (clean requirements.txt venv)"
log "pytest in the clean venv, $WORKERS workers (mirrors the Tests workflow)"
"$VENV_DIR/bin/python" -m pytest -n "$WORKERS" "$@" \
|| fail "pytest (clean requirements.txt venv)"
}
run_all() {
ensure_venv
run_precommit_all_files
run_pytest_clean_venv "$@"
}
main() {
require_file
ensure_venv
run_precommit_all_files
run_pytest_clean_venv "$@"
log "all CI gates passed locally — safe to push"
case "${1:-all}" in
venv) ensure_venv ;;
precommit) run_precommit_all_files pylint ;;
pytest)
shift
require_fresh_venv
run_pytest_clean_venv "$@"
;;
all)
[[ $# -gt 0 ]] && shift
run_all "$@"
;;
*) run_all "$@" ;;
esac
log "CI gate passed locally"
}
main "$@"

View File

@ -0,0 +1,77 @@
"""Pylint plugin: hash each file once for the duplicate-code check (R0801).
pylint's similarity checker compares every pair of linted files, and
``Symilar._find_common`` calls ``symilar.hash_lineset`` on BOTH files of every
pair. With ~550 files that is ~150 000 hashings of the same ~550 files, and it
was the slowest part of a whole-tree pylint run (well over 28 s of ~55 s under
the resource cap).
``hash_lineset`` is a pure function of a file's stripped lines and the window
size, and ``_find_common`` only reads the two dicts it returns (the line limits
it mutates are ``copy.copy``-ed first). Caching the result per (file, window)
therefore yields the same R0801 messages, computed once per file.
Fails closed: if a pylint upgrade changes ``hash_lineset``'s signature, this
raises instead of caching a function whose contract it no longer knows.
Loaded through ``load-plugins`` in pyproject.toml; the ``init-hook`` there puts
the repo root on ``sys.path`` first, which makes ``scripts.`` importable.
"""
from __future__ import annotations
import functools
import inspect
from typing import TYPE_CHECKING
from pylint.checkers import symilar
if TYPE_CHECKING:
from collections.abc import Callable
from pylint.lint import PyLinter
HashResult = tuple[symilar.HashToIndex_T, symilar.IndexToLines_T]
HashFunc = Callable[[symilar.LineSet, int], HashResult]
EXPECTED_PARAMS = ("lineset", "min_common_lines")
def memoise(func: HashFunc) -> HashFunc:
"""Wrap ``hash_lineset`` with a per-(file, window) cache.
The key is ``id(lineset)``; the cache keeps the lineset itself alive and
checks identity on a hit, so a recycled id can never return another
file's hashes.
"""
cache: dict[tuple[int, int], tuple[symilar.LineSet, HashResult]] = {}
@functools.wraps(func)
def cached(
lineset: symilar.LineSet,
min_common_lines: int = symilar.DEFAULT_MIN_SIMILARITY_LINE,
) -> HashResult:
key = (id(lineset), min_common_lines)
hit = cache.get(key)
if hit is None or hit[0] is not lineset:
hit = (lineset, func(lineset, min_common_lines))
cache[key] = hit
return hit[1]
return cached
def register(_linter: PyLinter) -> None:
"""Patch ``symilar.hash_lineset`` once per process (pylint plugin hook)."""
current = symilar.hash_lineset
if hasattr(current, "__wrapped__"):
return
params = tuple(inspect.signature(current).parameters)
if params != EXPECTED_PARAMS:
msg = (
f"pylint_symilar_cache: symilar.hash_lineset{params} no longer "
f"matches {EXPECTED_PARAMS}; re-check that it is still pure and "
"update scripts/pylint_symilar_cache.py"
)
raise RuntimeError(msg)
symilar.hash_lineset = memoise(current)

View File

@ -0,0 +1,67 @@
"""Autouse guard: tests resolve loopback names only, never the internet.
Without it a test that walked into an unpatched HLTB lookup (the poll-count
backfill after a pick, for example) made real HTTPS requests to
howlongtobeat.com: ~1.5 s per test, and a result that depended on the network
and on HLTB's live data. Every outbound call starts with ``getaddrinfo``
(requests/urllib3 directly, aiohttp through asyncio's thread resolver), so
refusing non-loopback hosts there turns each leak into the same fast failure
the package already handles when the machine is offline.
Tests that need a resolver patch ``socket.getaddrinfo`` themselves; their patch
sits on top of this one for the test's duration.
Split out of ``conftest.py`` to keep every file inside the 250-line cap;
``conftest`` imports the fixture by name, which is what registers it.
"""
from __future__ import annotations
import ipaddress
import socket
from typing import TYPE_CHECKING
from unittest.mock import patch
import pytest
if TYPE_CHECKING:
from collections.abc import Iterator
_REAL_GETADDRINFO = socket.getaddrinfo
_LOCAL_NAMES = frozenset({"", "localhost"})
def is_local_host(host: bytes | str | None) -> bool:
"""Whether *host* names this machine (loopback, unspecified, or None)."""
if host is None:
return True
if isinstance(host, bytes):
host = host.decode("ascii", "replace")
name = str(host)
if name.lower() in _LOCAL_NAMES:
return True
try:
address = ipaddress.ip_address(name)
except ValueError:
return False
return address.is_loopback or address.is_unspecified
def local_only_getaddrinfo(
host: bytes | str | None,
port: bytes | str | int | None,
*args: int,
**kwargs: int,
) -> list[tuple[object, ...]]:
"""``socket.getaddrinfo`` for loopback hosts; offline for anything else."""
if not is_local_host(host):
msg = f"tests may not resolve {host!r}: patch the network call instead"
raise socket.gaierror(socket.EAI_NONAME, msg)
return list(_REAL_GETADDRINFO(host, port, *args, **kwargs))
@pytest.fixture(autouse=True)
def _no_internet() -> Iterator[None]:
"""Refuse DNS for non-loopback hosts for the whole test."""
with patch("socket.getaddrinfo", local_only_getaddrinfo):
yield

View File

@ -32,6 +32,7 @@ from steam_backlog_enforcer.tests._isolate_reading import _isolate_reading
from steam_backlog_enforcer.tests._isolate_research import _isolate_research
from steam_backlog_enforcer.tests._isolate_shadow import _isolate_shadow
from steam_backlog_enforcer.tests._no_fast_install import _no_fast_install
from steam_backlog_enforcer.tests._no_internet import _no_internet
from steam_backlog_enforcer.tests._no_live_focus_project import _no_live_focus_project
from steam_backlog_enforcer.tests._no_real_effects import (
_no_live_games,
@ -56,6 +57,7 @@ __all__ = [
"_isolate_research",
"_isolate_shadow",
"_no_fast_install",
"_no_internet",
"_no_live_focus_project",
"_no_live_games",
"_no_real_effects",

View File

@ -45,9 +45,18 @@ class TestSteamAPIClient:
"""A burst past the per-second budget makes the client sleep."""
client = SteamAPIClient("key", "id")
client.session.get = MagicMock(return_value=_owned_response())
with patch(
"steam_backlog_enforcer._steam_api_client.time.sleep",
) as mock_sleep:
# Only sleeping advances this clock: no busy-wait for a real second.
clock = [1000.0]
with (
patch(
"steam_backlog_enforcer._steam_api_client.time.time",
side_effect=lambda: clock[-1],
),
patch(
"steam_backlog_enforcer._steam_api_client.time.sleep",
side_effect=lambda seconds: clock.append(clock[-1] + seconds),
) as mock_sleep,
):
# Keep requesting until the window fills and the throttle kicks in.
for _ in range(_BURST_LIMIT):
client.get_owned_games()

View File

@ -26,7 +26,9 @@ _DATASET_PKG = "steam_backlog_enforcer._web_dataset"
def _running() -> Iterator[int]:
"""Start the server on an ephemeral port in a thread; yield the port."""
server = create_server("127.0.0.1", 0)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread = threading.Thread(
target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True
)
thread.start()
try:
yield server.server_address[1]

View File

@ -188,7 +188,9 @@ def _serve(status: int, body: bytes) -> Iterator[str]:
self.wfile.write(body)
server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread = threading.Thread(
target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True
)
thread.start()
try:
yield f"http://127.0.0.1:{server.server_address[1]}/api/status"