mirror of
https://github.com/kuhyx/steam-backlog-enforcer.git
synced 2026-11-11 17:53:48 +01:00
Split ci-mirror gate into <60 s steps
Some checks are pending
dependency freshness / dependencies on newest stable (push) Waiting to run
md-naming / markdown naming convention (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (mypy, mypy, ) (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (pylint, pylint, ) (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (rest, , mypy,pylint) (push) Waiting to run
Tests / test (push) Waiting to run
Tests / web-coverage (push) Waiting to run
repo-contract / repo contract (push) Waiting to run
Some checks are pending
dependency freshness / dependencies on newest stable (push) Waiting to run
md-naming / markdown naming convention (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (mypy, mypy, ) (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (pylint, pylint, ) (push) Waiting to run
pre-commit / pre-commit (${{ matrix.group }}) (rest, , mypy,pylint) (push) Waiting to run
Tests / test (push) Waiting to run
Tests / web-coverage (push) Waiting to run
repo-contract / repo contract (push) Waiting to run
The single ci-mirror manual hook (clean venv + pre-commit --all-files +
serial pytest) ran past the 60 s per-command cap, so the tree could not be
stamped for a push. It is now nine gate steps, each under 55 s at load ~40:
- ci-mirror-venv / -precommit / -pytest: scripts/ci_mirror.sh parts;
pytest runs on 4 xdist workers, pytest-cov still enforces 100 %.
- ci-mirror-pylint-{pkg,tests-1,tests-2,tests-3}: per-file checks over a
partition of the pylint hook's targets; -dup and -cycles: the two
cross-file checks on the whole tree. pylint is now a local hook so all of
them share one env (&pylint_env, pylint==4.0.9).
- scripts/pylint_symilar_cache.py: pylint plugin that hashes each file once
for duplicate-code instead of once per file pair (same R0801 output).
Tests: an autouse guard refuses non-loopback DNS (several tests made live
HLTB requests), two web fixtures poll at 0.01 s, and the rate-limit test
uses a fake clock. CLAUDE.md and AGENTS.md get the push line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WzxdWWjUkRCFGNMnnXZrSP
This commit is contained in:
parent
87c6f200e3
commit
db44ca5b0b
@ -120,10 +120,19 @@ repos:
|
||||
# ===========================================================================
|
||||
# PYLINT
|
||||
# ===========================================================================
|
||||
- repo: https://github.com/pylint-dev/pylint
|
||||
rev: v4.0.9
|
||||
# A local hook (was the pylint-dev/pylint repo at v4.0.9) so the manual
|
||||
# ci-mirror-pylint-* shards below can share its env: pre-commit gives local
|
||||
# `language: python` hooks with the same additional_dependencies ONE env,
|
||||
# and the &pylint_env anchor is the single place the pylint version and its
|
||||
# import deps are pinned.
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: pylint
|
||||
name: pylint
|
||||
entry: pylint
|
||||
language: python
|
||||
types: [python]
|
||||
require_serial: true
|
||||
# Whole tree on every run, never the staged subset: `--fail-under`
|
||||
# grades a score, and a score over whatever happened to be staged let a
|
||||
# 9.99 tree pass at commit time and fail in CI. scripts/file_length/
|
||||
@ -137,11 +146,13 @@ repos:
|
||||
- --jobs=1
|
||||
- steam_backlog_enforcer
|
||||
- scripts/check_requirements_sync.py
|
||||
- scripts/pylint_symilar_cache.py
|
||||
# pylint resolves imports in its OWN isolated env, so anything the
|
||||
# package imports has to be listed here or it reports import-error and
|
||||
# drags the score under --fail-under. `mcp` is a real runtime import
|
||||
# (steam_backlog_enforcer._mcp_server).
|
||||
additional_dependencies:
|
||||
additional_dependencies: &pylint_env
|
||||
- pylint==4.0.9
|
||||
- aiohttp==3.14.5
|
||||
- earned-time @ git+https://github.com/kuhyx/utils@earned-time-v0.21.0#subdirectory=earned_time
|
||||
- leisure-watch @ git+https://github.com/kuhyx/leisure-watch@leisure-watch-v0.2.7
|
||||
@ -168,13 +179,97 @@ repos:
|
||||
exclude: ^(tests/|.*test.*\.py$)
|
||||
|
||||
# ===========================================================================
|
||||
# PYTEST + COVERAGE (push stage)
|
||||
# CI MIRROR (manual stage: the gate steps before a push)
|
||||
# ===========================================================================
|
||||
# One gate step per hook, each under the 60 s per-command cap even on a
|
||||
# loaded box: `scripts/check_gate_stamp.sh step <id>` per id of
|
||||
# `scripts/check_gate_stamp.sh full --list`; the last one stamps the tree.
|
||||
# The pylint hook above is split into the ci-mirror-pylint-* shards and
|
||||
# ci-mirror-precommit skips it, so the union checks exactly what CI does:
|
||||
# - per-file checks run per shard; the shards' `files` partition the hook's
|
||||
# targets (tests by name: rest|test_[a-h], test_[i-r], test_[s-z]). With
|
||||
# --fail-under=10.0 a shard passes only with zero scored messages, so
|
||||
# every shard passing == the whole tree at 10.00;
|
||||
# - the two cross-file checks (duplicate-code, cyclic-import) see the whole
|
||||
# tree, as in the full hook (ci-mirror-pylint-dup, ci-mirror-pylint-cycles).
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: ci-mirror
|
||||
name: CI mirror (clean-venv pytest + pre-commit --all-files)
|
||||
entry: bash scripts/ci_mirror.sh steam_backlog_enforcer/tests/
|
||||
- id: ci-mirror-venv
|
||||
name: CI mirror (clean requirements.txt venv, rebuilt on change)
|
||||
entry: bash scripts/ci_mirror.sh venv
|
||||
language: system
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
require_serial: true
|
||||
stages: [manual]
|
||||
- id: ci-mirror-precommit
|
||||
name: CI mirror (pre-commit --all-files, pylint via the shards)
|
||||
entry: bash scripts/ci_mirror.sh precommit
|
||||
language: system
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
require_serial: true
|
||||
stages: [manual]
|
||||
- &pylint_shard
|
||||
id: ci-mirror-pylint-pkg
|
||||
name: CI mirror pylint (package + scripts, per-file checks)
|
||||
entry: pylint
|
||||
language: python
|
||||
additional_dependencies: *pylint_env
|
||||
types: [python]
|
||||
require_serial: true
|
||||
stages: [manual]
|
||||
files: ^(steam_backlog_enforcer/(?!tests/).*|scripts/(check_requirements_sync|pylint_symilar_cache))\.py$
|
||||
args: &pylint_per_file
|
||||
- --rcfile=pyproject.toml
|
||||
- --fail-under=10.0
|
||||
- --jobs=1
|
||||
- --disable=duplicate-code,cyclic-import
|
||||
- <<: *pylint_shard
|
||||
id: ci-mirror-pylint-tests-1
|
||||
name: CI mirror pylint (tests helpers + test_[a-h], per-file checks)
|
||||
files: ^steam_backlog_enforcer/tests/(?!test_[i-z]).*\.py$
|
||||
- <<: *pylint_shard
|
||||
id: ci-mirror-pylint-tests-2
|
||||
name: CI mirror pylint (test_[i-r], per-file checks)
|
||||
files: ^steam_backlog_enforcer/tests/test_[i-r][^/]*\.py$
|
||||
- <<: *pylint_shard
|
||||
id: ci-mirror-pylint-tests-3
|
||||
name: CI mirror pylint (test_[s-z], per-file checks)
|
||||
files: ^steam_backlog_enforcer/tests/test_[s-z][^/]*\.py$
|
||||
- &pylint_whole_tree
|
||||
<<: *pylint_shard
|
||||
id: ci-mirror-pylint-dup
|
||||
name: CI mirror pylint (whole tree, duplicate-code)
|
||||
files: ''
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
args:
|
||||
- --rcfile=pyproject.toml
|
||||
- --fail-under=10.0
|
||||
- --jobs=1
|
||||
- --disable=all
|
||||
# bad-plugin-value keeps a plugin that failed to load (the
|
||||
# duplicate-code speed-up) loud instead of silently slow.
|
||||
- --enable=duplicate-code,bad-plugin-value
|
||||
- steam_backlog_enforcer
|
||||
- scripts/check_requirements_sync.py
|
||||
- scripts/pylint_symilar_cache.py
|
||||
- <<: *pylint_whole_tree
|
||||
id: ci-mirror-pylint-cycles
|
||||
name: CI mirror pylint (whole tree, cyclic-import)
|
||||
args:
|
||||
- --rcfile=pyproject.toml
|
||||
- --fail-under=10.0
|
||||
- --jobs=1
|
||||
- --disable=all
|
||||
- --enable=cyclic-import
|
||||
- steam_backlog_enforcer
|
||||
- scripts/check_requirements_sync.py
|
||||
- scripts/pylint_symilar_cache.py
|
||||
- id: ci-mirror-pytest
|
||||
name: CI mirror (clean-venv pytest, xdist, 100% coverage)
|
||||
entry: bash scripts/ci_mirror.sh pytest steam_backlog_enforcer/tests/
|
||||
language: system
|
||||
pass_filenames: false
|
||||
always_run: true
|
||||
|
||||
@ -23,3 +23,4 @@ After running tests fix all coverage gaps and issues, do not ignore unless speci
|
||||
- lint: `pre-commit run ruff --all-files`
|
||||
- coverage: `python3 -m pytest -q`
|
||||
- coverage-gaps: `coverage-gaps coverage.lcov`
|
||||
- push: `scripts/check_gate_stamp.sh full && git push` (every manual-stage gate hook, ~3 min, then push). Agents (60 s cap): one call per id of `scripts/check_gate_stamp.sh full --list`: `CAP_MEM=4G CAP_CPU_PCT=20 ~/.claude/scripts/capped.sh timeout 55 scripts/check_gate_stamp.sh step <id>` (each < 55 s; the last stamps the tree), then `git push`.
|
||||
|
||||
@ -24,3 +24,4 @@ After running tests fix all coverage gaps and issues, do not ignore unless speci
|
||||
- coverage: `python3 -m pytest -q`
|
||||
- coverage-gaps: `coverage-gaps coverage.lcov`
|
||||
- deploy: `scripts/deploy.sh`
|
||||
- push: `scripts/check_gate_stamp.sh full && git push` (every manual-stage gate hook, ~3 min, then push). Agents (60 s cap): one call per id of `scripts/check_gate_stamp.sh full --list`: `CAP_MEM=4G CAP_CPU_PCT=20 ~/.claude/scripts/capped.sh timeout 55 scripts/check_gate_stamp.sh step <id>` (each < 55 s; the last stamps the tree), then `git push`.
|
||||
|
||||
@ -49,6 +49,10 @@ include = ["*.py", "**/*.py"]
|
||||
# byte-identical to upstream; ruff --unsafe-fixes otherwise strips its print()
|
||||
# calls (T201) and silently guts the gate's reporting.
|
||||
exclude = [".git", ".venv", "__pycache__", "build", "dist", ".eggs", "scripts/file_length"]
|
||||
# scripts/ has no __init__.py on purpose (one would make it a second top-level
|
||||
# package for requirements-sync), yet pylint imports scripts.pylint_symilar_cache
|
||||
# from it: an implicit namespace package, declared so INP001 knows.
|
||||
namespace-packages = ["scripts"]
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["ALL"]
|
||||
@ -130,6 +134,10 @@ exclude = [".venv/"]
|
||||
# The repo root on the path, so pylint resolves the package's own imports the
|
||||
# same way pytest does instead of reporting import-error for them.
|
||||
init-hook = "import sys; sys.path.insert(0, '.')"
|
||||
# Hashes each file once for duplicate-code instead of once per file PAIR; same
|
||||
# R0801 output, whole-tree pylint fits the 60 s cap. Needs the init-hook above
|
||||
# (pylint runs it before loading config-file plugins, never CLI ones).
|
||||
load-plugins = ["scripts.pylint_symilar_cache"]
|
||||
analyse-fallback-blocks = true
|
||||
persistent = true
|
||||
# Serial on purpose: with jobs != 1 pylint ships the linter to its workers via
|
||||
|
||||
@ -8,16 +8,26 @@
|
||||
# requirements.txt. That gap is why green-locally repos went red in CI
|
||||
# (missing runtime deps, --all-files lint debt, order-dependent test flakes).
|
||||
#
|
||||
# This script reproduces both CI workflows on the developer machine:
|
||||
# 1. a venv built ONLY from requirements.txt (mirrors the Tests workflow's
|
||||
# `pip install -r requirements.txt`), rebuilt only when requirements.txt
|
||||
# changes (hash-gated so day-to-day pushes stay fast);
|
||||
# 2. `pre-commit run --all-files` (mirrors the pre-commit workflow);
|
||||
# 3. `python -m pytest` inside that clean venv (mirrors Tests).
|
||||
# This script reproduces both CI workflows on the developer machine, split
|
||||
# into parts that each fit the 60 s per-command cap (one gate step each, see
|
||||
# the ci-mirror-* manual hooks in .pre-commit-config.yaml):
|
||||
# venv a venv built ONLY from requirements.txt (mirrors the Tests
|
||||
# workflow's `uv pip install -r requirements.txt`), rebuilt only
|
||||
# when requirements.txt changes (hash-gated);
|
||||
# precommit `pre-commit run --all-files` minus pylint (mirrors the
|
||||
# pre-commit workflow; pylint runs as the ci-mirror-pylint-*
|
||||
# shards, which together check exactly what the pylint hook does);
|
||||
# pytest `python -m pytest` in that clean venv (mirrors Tests), spread
|
||||
# over CI_MIRROR_WORKERS xdist workers (default 4); pytest-cov
|
||||
# merges the workers and enforces the 100 % gate itself.
|
||||
# `all` (or no part, or a first argument that is not a part name, the old
|
||||
# `ci_mirror.sh <pytest args>` form) runs the three in order with the full
|
||||
# pylint hook: kuhy's terminal, over 60 s, so agents run the gate steps.
|
||||
# Arguments after the part go to pytest.
|
||||
#
|
||||
# Runs on demand (`scripts/check_gate_stamp.sh full`, pre-commit's manual
|
||||
# stage) and in CI. It left the pre-push path on 2026-09-20: 10+ minutes
|
||||
# per push under the resource cap, for work finish_auto.sh had just done.
|
||||
# Runs on demand (`scripts/check_gate_stamp.sh step <hook-id>`, pre-commit's
|
||||
# manual stage). It left the pre-push path on 2026-09-20: 10+ minutes per push
|
||||
# under the resource cap, for work finish_auto.sh had just done.
|
||||
# ============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
@ -25,6 +35,7 @@ set -euo pipefail
|
||||
# Requirements file may be overridden (e.g. testsAndMisc uses meta/…).
|
||||
REQUIREMENTS_FILE="${REQUIREMENTS_FILE:-requirements.txt}"
|
||||
readonly REQUIREMENTS_FILE
|
||||
readonly WORKERS="${CI_MIRROR_WORKERS:-4}"
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel)"
|
||||
readonly ROOT
|
||||
@ -48,47 +59,87 @@ require_file() {
|
||||
fi
|
||||
}
|
||||
|
||||
# Rebuild the venv only when requirements.txt changed since the last build.
|
||||
ensure_venv() {
|
||||
local current stored
|
||||
current="$(sha256sum "$REQ_PATH" | cut -d' ' -f1)"
|
||||
stored=""
|
||||
if [[ -f "$HASH_FILE" ]]; then
|
||||
stored="$(cat "$HASH_FILE")"
|
||||
fi
|
||||
requirements_hash() {
|
||||
sha256sum "$REQ_PATH" | cut -d' ' -f1
|
||||
}
|
||||
|
||||
if [[ -x "$VENV_DIR/bin/python" && "$current" == "$stored" ]]; then
|
||||
log "venv up to date (requirements.txt unchanged)"
|
||||
venv_is_fresh() {
|
||||
[[ -x "$VENV_DIR/bin/python" && -f "$HASH_FILE" \
|
||||
&& "$(requirements_hash)" == "$(cat "$HASH_FILE")" ]]
|
||||
}
|
||||
|
||||
# uv when present: the installer CI uses, and fast enough for one gate step.
|
||||
rebuild_venv() {
|
||||
rm -rf "$VENV_DIR"
|
||||
if command -v uv >/dev/null 2>&1; then
|
||||
uv venv --quiet --python python3 "$VENV_DIR" \
|
||||
|| fail "uv venv for the clean venv"
|
||||
uv pip install --quiet --python "$VENV_DIR/bin/python" -r "$REQ_PATH" \
|
||||
|| fail "uv pip install -r $REQUIREMENTS_FILE (a dep may be undeclared)"
|
||||
return
|
||||
fi
|
||||
|
||||
log "requirements.txt changed — rebuilding clean venv (mirrors CI install)"
|
||||
rm -rf "$VENV_DIR"
|
||||
python3 -m venv "$VENV_DIR"
|
||||
"$VENV_DIR/bin/python" -m pip install --quiet --upgrade pip \
|
||||
|| fail "pip self-upgrade in the clean venv"
|
||||
"$VENV_DIR/bin/python" -m pip install --quiet -r "$REQ_PATH" \
|
||||
|| fail "pip install -r $REQUIREMENTS_FILE (a dep may be undeclared)"
|
||||
printf '%s' "$current" > "$HASH_FILE"
|
||||
}
|
||||
|
||||
# Rebuild the venv only when requirements.txt changed since the last build.
|
||||
ensure_venv() {
|
||||
if venv_is_fresh; then
|
||||
log "venv up to date (requirements.txt unchanged)"
|
||||
return
|
||||
fi
|
||||
log "requirements.txt changed — rebuilding clean venv (mirrors CI install)"
|
||||
rebuild_venv
|
||||
requirements_hash > "$HASH_FILE"
|
||||
log "clean venv ready"
|
||||
}
|
||||
|
||||
# The pytest part never rebuilds: a rebuild would not fit its budget, and a
|
||||
# stale venv would test the wrong dependencies. Fail closed instead.
|
||||
require_fresh_venv() {
|
||||
venv_is_fresh \
|
||||
|| fail "clean venv missing or stale; run the ci-mirror-venv step first"
|
||||
}
|
||||
|
||||
run_precommit_all_files() {
|
||||
log "pre-commit run --all-files (mirrors the pre-commit workflow)"
|
||||
pre-commit run --all-files || fail "pre-commit --all-files"
|
||||
local skip="${1:-}"
|
||||
log "pre-commit run --all-files${skip:+ minus $skip} (mirrors the pre-commit workflow)"
|
||||
SKIP="${SKIP:+$SKIP,}$skip" pre-commit run --all-files \
|
||||
|| fail "pre-commit --all-files"
|
||||
}
|
||||
|
||||
run_pytest_clean_venv() {
|
||||
log "pytest in the clean venv (mirrors the Tests workflow)"
|
||||
"$VENV_DIR/bin/python" -m pytest "$@" || fail "pytest (clean requirements.txt venv)"
|
||||
log "pytest in the clean venv, $WORKERS workers (mirrors the Tests workflow)"
|
||||
"$VENV_DIR/bin/python" -m pytest -n "$WORKERS" "$@" \
|
||||
|| fail "pytest (clean requirements.txt venv)"
|
||||
}
|
||||
|
||||
run_all() {
|
||||
ensure_venv
|
||||
run_precommit_all_files
|
||||
run_pytest_clean_venv "$@"
|
||||
}
|
||||
|
||||
main() {
|
||||
require_file
|
||||
ensure_venv
|
||||
run_precommit_all_files
|
||||
run_pytest_clean_venv "$@"
|
||||
log "all CI gates passed locally — safe to push"
|
||||
case "${1:-all}" in
|
||||
venv) ensure_venv ;;
|
||||
precommit) run_precommit_all_files pylint ;;
|
||||
pytest)
|
||||
shift
|
||||
require_fresh_venv
|
||||
run_pytest_clean_venv "$@"
|
||||
;;
|
||||
all)
|
||||
[[ $# -gt 0 ]] && shift
|
||||
run_all "$@"
|
||||
;;
|
||||
*) run_all "$@" ;;
|
||||
esac
|
||||
log "CI gate passed locally"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
||||
77
scripts/pylint_symilar_cache.py
Normal file
77
scripts/pylint_symilar_cache.py
Normal file
@ -0,0 +1,77 @@
|
||||
"""Pylint plugin: hash each file once for the duplicate-code check (R0801).
|
||||
|
||||
pylint's similarity checker compares every pair of linted files, and
|
||||
``Symilar._find_common`` calls ``symilar.hash_lineset`` on BOTH files of every
|
||||
pair. With ~550 files that is ~150 000 hashings of the same ~550 files, and it
|
||||
was the slowest part of a whole-tree pylint run (well over 28 s of ~55 s under
|
||||
the resource cap).
|
||||
|
||||
``hash_lineset`` is a pure function of a file's stripped lines and the window
|
||||
size, and ``_find_common`` only reads the two dicts it returns (the line limits
|
||||
it mutates are ``copy.copy``-ed first). Caching the result per (file, window)
|
||||
therefore yields the same R0801 messages, computed once per file.
|
||||
|
||||
Fails closed: if a pylint upgrade changes ``hash_lineset``'s signature, this
|
||||
raises instead of caching a function whose contract it no longer knows.
|
||||
|
||||
Loaded through ``load-plugins`` in pyproject.toml; the ``init-hook`` there puts
|
||||
the repo root on ``sys.path`` first, which makes ``scripts.`` importable.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import functools
|
||||
import inspect
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pylint.checkers import symilar
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from pylint.lint import PyLinter
|
||||
|
||||
HashResult = tuple[symilar.HashToIndex_T, symilar.IndexToLines_T]
|
||||
HashFunc = Callable[[symilar.LineSet, int], HashResult]
|
||||
|
||||
EXPECTED_PARAMS = ("lineset", "min_common_lines")
|
||||
|
||||
|
||||
def memoise(func: HashFunc) -> HashFunc:
|
||||
"""Wrap ``hash_lineset`` with a per-(file, window) cache.
|
||||
|
||||
The key is ``id(lineset)``; the cache keeps the lineset itself alive and
|
||||
checks identity on a hit, so a recycled id can never return another
|
||||
file's hashes.
|
||||
"""
|
||||
cache: dict[tuple[int, int], tuple[symilar.LineSet, HashResult]] = {}
|
||||
|
||||
@functools.wraps(func)
|
||||
def cached(
|
||||
lineset: symilar.LineSet,
|
||||
min_common_lines: int = symilar.DEFAULT_MIN_SIMILARITY_LINE,
|
||||
) -> HashResult:
|
||||
key = (id(lineset), min_common_lines)
|
||||
hit = cache.get(key)
|
||||
if hit is None or hit[0] is not lineset:
|
||||
hit = (lineset, func(lineset, min_common_lines))
|
||||
cache[key] = hit
|
||||
return hit[1]
|
||||
|
||||
return cached
|
||||
|
||||
|
||||
def register(_linter: PyLinter) -> None:
|
||||
"""Patch ``symilar.hash_lineset`` once per process (pylint plugin hook)."""
|
||||
current = symilar.hash_lineset
|
||||
if hasattr(current, "__wrapped__"):
|
||||
return
|
||||
params = tuple(inspect.signature(current).parameters)
|
||||
if params != EXPECTED_PARAMS:
|
||||
msg = (
|
||||
f"pylint_symilar_cache: symilar.hash_lineset{params} no longer "
|
||||
f"matches {EXPECTED_PARAMS}; re-check that it is still pure and "
|
||||
"update scripts/pylint_symilar_cache.py"
|
||||
)
|
||||
raise RuntimeError(msg)
|
||||
symilar.hash_lineset = memoise(current)
|
||||
67
steam_backlog_enforcer/tests/_no_internet.py
Normal file
67
steam_backlog_enforcer/tests/_no_internet.py
Normal file
@ -0,0 +1,67 @@
|
||||
"""Autouse guard: tests resolve loopback names only, never the internet.
|
||||
|
||||
Without it a test that walked into an unpatched HLTB lookup (the poll-count
|
||||
backfill after a pick, for example) made real HTTPS requests to
|
||||
howlongtobeat.com: ~1.5 s per test, and a result that depended on the network
|
||||
and on HLTB's live data. Every outbound call starts with ``getaddrinfo``
|
||||
(requests/urllib3 directly, aiohttp through asyncio's thread resolver), so
|
||||
refusing non-loopback hosts there turns each leak into the same fast failure
|
||||
the package already handles when the machine is offline.
|
||||
|
||||
Tests that need a resolver patch ``socket.getaddrinfo`` themselves; their patch
|
||||
sits on top of this one for the test's duration.
|
||||
|
||||
Split out of ``conftest.py`` to keep every file inside the 250-line cap;
|
||||
``conftest`` imports the fixture by name, which is what registers it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
from typing import TYPE_CHECKING
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
_REAL_GETADDRINFO = socket.getaddrinfo
|
||||
_LOCAL_NAMES = frozenset({"", "localhost"})
|
||||
|
||||
|
||||
def is_local_host(host: bytes | str | None) -> bool:
|
||||
"""Whether *host* names this machine (loopback, unspecified, or None)."""
|
||||
if host is None:
|
||||
return True
|
||||
if isinstance(host, bytes):
|
||||
host = host.decode("ascii", "replace")
|
||||
name = str(host)
|
||||
if name.lower() in _LOCAL_NAMES:
|
||||
return True
|
||||
try:
|
||||
address = ipaddress.ip_address(name)
|
||||
except ValueError:
|
||||
return False
|
||||
return address.is_loopback or address.is_unspecified
|
||||
|
||||
|
||||
def local_only_getaddrinfo(
|
||||
host: bytes | str | None,
|
||||
port: bytes | str | int | None,
|
||||
*args: int,
|
||||
**kwargs: int,
|
||||
) -> list[tuple[object, ...]]:
|
||||
"""``socket.getaddrinfo`` for loopback hosts; offline for anything else."""
|
||||
if not is_local_host(host):
|
||||
msg = f"tests may not resolve {host!r}: patch the network call instead"
|
||||
raise socket.gaierror(socket.EAI_NONAME, msg)
|
||||
return list(_REAL_GETADDRINFO(host, port, *args, **kwargs))
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_internet() -> Iterator[None]:
|
||||
"""Refuse DNS for non-loopback hosts for the whole test."""
|
||||
with patch("socket.getaddrinfo", local_only_getaddrinfo):
|
||||
yield
|
||||
@ -32,6 +32,7 @@ from steam_backlog_enforcer.tests._isolate_reading import _isolate_reading
|
||||
from steam_backlog_enforcer.tests._isolate_research import _isolate_research
|
||||
from steam_backlog_enforcer.tests._isolate_shadow import _isolate_shadow
|
||||
from steam_backlog_enforcer.tests._no_fast_install import _no_fast_install
|
||||
from steam_backlog_enforcer.tests._no_internet import _no_internet
|
||||
from steam_backlog_enforcer.tests._no_live_focus_project import _no_live_focus_project
|
||||
from steam_backlog_enforcer.tests._no_real_effects import (
|
||||
_no_live_games,
|
||||
@ -56,6 +57,7 @@ __all__ = [
|
||||
"_isolate_research",
|
||||
"_isolate_shadow",
|
||||
"_no_fast_install",
|
||||
"_no_internet",
|
||||
"_no_live_focus_project",
|
||||
"_no_live_games",
|
||||
"_no_real_effects",
|
||||
|
||||
@ -45,9 +45,18 @@ class TestSteamAPIClient:
|
||||
"""A burst past the per-second budget makes the client sleep."""
|
||||
client = SteamAPIClient("key", "id")
|
||||
client.session.get = MagicMock(return_value=_owned_response())
|
||||
with patch(
|
||||
"steam_backlog_enforcer._steam_api_client.time.sleep",
|
||||
) as mock_sleep:
|
||||
# Only sleeping advances this clock: no busy-wait for a real second.
|
||||
clock = [1000.0]
|
||||
with (
|
||||
patch(
|
||||
"steam_backlog_enforcer._steam_api_client.time.time",
|
||||
side_effect=lambda: clock[-1],
|
||||
),
|
||||
patch(
|
||||
"steam_backlog_enforcer._steam_api_client.time.sleep",
|
||||
side_effect=lambda seconds: clock.append(clock[-1] + seconds),
|
||||
) as mock_sleep,
|
||||
):
|
||||
# Keep requesting until the window fills and the throttle kicks in.
|
||||
for _ in range(_BURST_LIMIT):
|
||||
client.get_owned_games()
|
||||
|
||||
@ -26,7 +26,9 @@ _DATASET_PKG = "steam_backlog_enforcer._web_dataset"
|
||||
def _running() -> Iterator[int]:
|
||||
"""Start the server on an ephemeral port in a thread; yield the port."""
|
||||
server = create_server("127.0.0.1", 0)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread = threading.Thread(
|
||||
target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True
|
||||
)
|
||||
thread.start()
|
||||
try:
|
||||
yield server.server_address[1]
|
||||
|
||||
@ -188,7 +188,9 @@ def _serve(status: int, body: bytes) -> Iterator[str]:
|
||||
self.wfile.write(body)
|
||||
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler)
|
||||
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
||||
thread = threading.Thread(
|
||||
target=server.serve_forever, kwargs={"poll_interval": 0.01}, daemon=True
|
||||
)
|
||||
thread.start()
|
||||
try:
|
||||
yield f"http://127.0.0.1:{server.server_address[1]}/api/status"
|
||||
|
||||
Loading…
Reference in New Issue
Block a user