kuhyx/testsAndMisc
1
0
Fork 0
mirror of https://github.com/kuhyx/testsAndMisc.git synced 2026-11-10 18:47:04 +01:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

Split the code_tutor pytest runner and verdict streaming out

Browse Source 
_challenge_support.py 402 -> 237. _pytest_clean and _patch_and_test go
to _pytest_runner.py with the subprocess/os/sys imports they solely used;
_stream_verdict and _parse_verdict go to _verdict.py with rich's Live.

code_tutor patches by absolute string rather than a MOD constant, so the
seam cost 15 per-name rewrites instead of one sed. Two intermediate
failures caught by running it: the test modules import the moved names
directly, and five patch strings still named the old module.

Pass count is 216 before and after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011w2Pj4Ek3ZNMQVYcFHgXDY
This commit is contained in:
Krzysztof Rudnicki 2026-08-16 17:33:58 +02:00
parent 2c01a0267e
commit 40a4eaf5b5
9 changed files with 287 additions and 187 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

19
docs/superpowers/contracts/code-tutor-challenge-support-split-2026-08-16.json Normal file
View File

@ -0,0 +1,19 @@
{
"title": "Split code_tutor/_challenge_support.py under the 250-line cap",
"objective": "Bring python_pkg/code_tutor/_challenge_support.py (402 lines) under the repo-wide 250-line cap by moving the isolated pytest runner and the LLM verdict streaming into their own modules.",
"acceptance_criteria": [
"_challenge_support.py, _pytest_runner.py and _verdict.py are each at most 250 lines.",
"The code_tutor suite still reports exactly 216 passing tests -- the same count as before the split, proving no test was dropped or silently skipped.",
"The package keeps 100% branch coverage of the code_tutor source modules.",
"Every patch string naming a moved function or one of its plain-import collaborators is repointed at the module the code actually lives in now. code_tutor uses absolute patch strings rather than a MOD constant, so this is a per-string edit and cannot be done with one substitution.",
"_challenge.py imports _patch_and_test, _pytest_clean, _parse_verdict and _stream_verdict back, because test_challenge_part3.py patches the first two on _challenge.",
"No assertion is changed and no test is deleted, skipped or weakened.",
"No suppression is added and no coverage exclusion is introduced."
],
"out_of_scope": [
"Changing what any test asserts, or any pytest invocation flags.",
"Splitting _challenge.py, cli.py, _analyzer.py or the code_tutor test modules, which are separate commits.",
"Adding a re-export shim in _challenge_support; importers point at the new modules directly to keep the dependency direction one-way."
],
"verifier": "PYTHONPATH=. python3 -m pytest python_pkg/code_tutor/tests -q (expects exactly 216 passed), plus an import-time attribute probe over the four modules, plus pre-commit run --files on the touched files."
}

52
docs/superpowers/evidence/code-tutor-challenge-support-split-2026-08-16.json Normal file
View File

@ -0,0 +1,52 @@
{
"intent": "Bring code_tutor/_challenge_support.py under the 250-line cap by moving the isolated pytest runner and the LLM verdict streaming into their own modules, with no behaviour change.",
"scope": [
"python_pkg/code_tutor/_pytest_runner.py, _verdict.py (new)",
"python_pkg/code_tutor/_challenge_support.py (402 -> 237)",
"python_pkg/code_tutor/_challenge.py (import paths only)",
"python_pkg/code_tutor/tests/test_challenge.py, test_challenge_part2.py, test_challenge_part3.py (import and patch paths only)",
"Non-goal: no assertion changed, no test deleted, skipped or weakened"
],
"changes": [
"Moved _pytest_clean and _patch_and_test into _pytest_runner.py, which now owns the subprocess, os and sys imports they were the only users of, and imports _project_root back from _challenge_support.",
"Moved _stream_verdict and _parse_verdict into _verdict.py, which owns rich.live.Live, Text, time, json and re. Live was used at exactly one call site, so nothing else in _challenge_support needed it.",
"_challenge_support.py dropped seven now-dead imports (Live, Text, time, json, os, subprocess, sys, re) and keeps ast, Path, Panel and Syntax, which its remaining source-inspection helpers use.",
"_challenge.py imports the four moved functions back from their new homes, preserving the patch targets test_challenge_part3.py names on _challenge.",
"Repointed the test patch strings per name: ten occurrences of _challenge_support.Live to _verdict.Live, three of _challenge_support.subprocess.run and two of _challenge_support._pytest_clean to _pytest_runner. Also repointed the direct imports in test_challenge.py and test_challenge_part2.py."
],
"verification": [
{
"command": "PYTHONPATH=. python3 -m pytest python_pkg/code_tutor/tests -q",
"result": "pass",
"evidence": "216 passed before the split and 216 passed after it - identical pass count, not merely a green run."
},
{
"command": "PYTHONPATH=. python3 -m pytest python_pkg/code_tutor/tests -q (intermediate runs)",
"result": "fail then pass",
"evidence": "Two intermediate failures, both caught by running rather than by reading: first 2 collection ImportErrors because test_challenge.py and test_challenge_part2.py import the moved functions directly, then 5 failures with 'does not have the attribute _pytest_clean' and 'no attribute subprocess' from patch strings still naming the old module. Both fixed per name."
},
{
"command": "PYTHONPATH=. python3 -c \"from python_pkg.code_tutor import _challenge as c, _challenge_support as s, _pytest_runner as r, _verdict as v; c._pytest_clean; c._patch_and_test; c._stream_verdict; s.Syntax; s.ast; r.subprocess; r.os; r.sys; v.Live; v.time\"",
"result": "pass",
"evidence": "runtime probe OK - every patched name is a real attribute of the module its test names."
},
{
"command": "wc -l python_pkg/code_tutor/_challenge_support.py python_pkg/code_tutor/_pytest_runner.py python_pkg/code_tutor/_verdict.py",
"result": "pass",
"evidence": "237, 111 and 89 lines, all under the 250-line cap. The first cut alone left 313, so a second seam was needed - the budgeting warning in docs/python-split-recipes.md again."
},
{
"command": "pre-commit run --files <the touched files and artifacts>",
"result": "pass",
"evidence": "ruff, ruff format, mypy, pylint, bandit and pytest-with-coverage all pass on the touched files."
}
],
"risks": [
"A sed with escaped alternation silently matched nothing and left the patch strings unchanged while reporting success; the failing test run caught it and the replacement was redone in Python. Worth remembering that a no-op rewrite looks identical to a successful one at the shell.",
"_pytest_runner runs the user's tests in a subprocess. Its subprocess import is patched by test_challenge.py, so a future move of that import would let the suite really invoke pytest - the same silent-failure shape as the CUPS restart in brother_printer."
],
"rollback": [
"git revert the commit; _challenge_support.py returns to 402 lines and the two new modules disappear.",
"After rollback, re-run the package suite and confirm it still reports 216 passed."
]
}

6
python_pkg/code_tutor/_challenge.py
View File

@ -31,13 +31,11 @@ from python_pkg.code_tutor._challenge_support import (
_extract_signature_block,
_find_tests,
_import_hint,
_parse_verdict,
_patch_and_test,
_project_root,
_pytest_clean,
_show_test_panels,
_stream_verdict,
)
from python_pkg.code_tutor._pytest_runner import _patch_and_test, _pytest_clean
from python_pkg.code_tutor._verdict import _parse_verdict, _stream_verdict
if TYPE_CHECKING:
from collections.abc import Callable

165
python_pkg/code_tutor/_challenge_support.py
View File

@ -11,19 +11,11 @@ challenge flows that consume it.
from __future__ import annotations
import ast
import json
import os
from pathlib import Path
import re
import subprocess
import sys
import time
from typing import TYPE_CHECKING
from rich.live import Live
from rich.panel import Panel
from rich.syntax import Syntax
from rich.text import Text
if TYPE_CHECKING:
from collections.abc import Callable
@ -31,7 +23,6 @@ if TYPE_CHECKING:
from rich.console import Console
from python_pkg.code_tutor._analyzer import CodeItem
from python_pkg.code_tutor._llm import Backend
def _scan_test_file(source: str, item_name: str) -> list[str]:
@ -186,162 +177,6 @@ def _project_root(start: Path) -> Path:
return start.resolve()
def _pytest_clean(
test_ids_or_file: list[str],
project_root: Path,
console: Console,
extra_env: dict[str, str] | None = None,
) -> bool:
"""Run pytest on *test_ids_or_file* and return True if all pass.
Coverage is disabled so the project's ``fail_under`` threshold does not
interfere with isolated challenge runs.
Args:
test_ids_or_file: Pytest node IDs or a single temp-file path.
project_root: Working directory for the pytest subprocess.
console: Rich console for output.
extra_env: Extra environment variables (e.g. PYTHONPATH override).
Returns:
``True`` when all collected tests pass.
"""
env = {**os.environ, **(extra_env or {})}
result = subprocess.run(
[
sys.executable,
"-m",
"pytest",
*test_ids_or_file,
"-v",
"--tb=short",
"--no-header",
"-p",
"no:cov",
"--override-ini=addopts=",
],
capture_output=True,
text=True,
cwd=str(project_root),
env=env,
check=False,
)
output = (result.stdout + result.stderr).strip()
if output:
console.print(output)
return result.returncode == 0
def _patch_and_test(
item: CodeItem,
codebase_path: str,
user_code: str,
test_entries: list[tuple[Path, list[str]]],
console: Console,
) -> bool:
"""Replace *item* in source with *user_code*, run tests, then restore.
Args:
item: Code item to patch.
codebase_path: Absolute codebase root path.
user_code: User's implementation text.
test_entries: ``(test_file, node_ids)`` pairs to run.
console: Rich console for output.
Returns:
``True`` if all tests passed.
"""
source_file = Path(codebase_path) / item.file
original = source_file.read_text(encoding="utf-8")
orig_lines = original.splitlines()
before = orig_lines[: item.start_line - 1]
after = orig_lines[item.end_line :]
new_source = "\n".join(before + user_code.splitlines() + after) + "\n"
try:
ast.parse(new_source)
except SyntaxError as exc:
console.print(f"[red]Syntax error in your implementation: {exc}[/red]")
return False
test_ids = [f"{tf}::{nid}" for tf, nids in test_entries for nid in nids]
project_root = _project_root(Path(codebase_path))
try:
source_file.write_text(new_source, encoding="utf-8")
return _pytest_clean(test_ids, project_root, console)
finally:
source_file.write_text(original, encoding="utf-8")
def _stream_verdict(
system: str,
user_msg: str,
backend: Backend,
console: Console,
label: str = "Rating",
) -> str:
"""Stream an LLM call silently while showing an elapsed timer.
Args:
system: System prompt for the judge.
user_msg: User message to judge.
backend: LLM backend with a ``stream(system, user, callback)`` method.
console: Rich console for the live timer.
label: Label prefix shown in the timer (e.g. ``"Rating"``).
Returns:
Accumulated response text.
"""
parts: list[str] = []
start = time.monotonic()
def _on_token(token: str) -> None:
parts.append(token)
elapsed = int(time.monotonic() - start)
live.update(Text(f"{label}... {elapsed}s", style="yellow"))
with Live(
Text(f"{label}... 0s", style="yellow"),
console=console,
refresh_per_second=4,
transient=True,
) as live:
backend.stream(system, user_msg, _on_token)
return "".join(parts)
def _parse_verdict(raw: str) -> tuple[str, str]:
"""Parse ``{"verdict": ..., "gap": ...}`` JSON from *raw*, tolerating fences.
Args:
raw: Raw LLM response text.
Returns:
``(verdict, gap)`` where verdict is ``"PASS"`` or ``"FAIL"``.
"""
clean = re.sub(r"```(?:json)?\s*", "", raw, flags=re.DOTALL).strip()
start = clean.find("{")
end = clean.rfind("}") + 1
if start == -1 or end == 0:
return "FAIL", "Could not parse judge response."
try:
data = json.loads(clean[start:end])
except json.JSONDecodeError:
return "FAIL", "Could not parse judge response."
verdict = str(data.get("verdict", "FAIL")).upper()
if verdict not in {"PASS", "FAIL"}:
verdict = "FAIL"
return verdict, str(data.get("gap", ""))
# ---------------------------------------------------------------------------
# "Write tests first" flow -- helpers
# ---------------------------------------------------------------------------
def _extract_signature_block(item: CodeItem, codebase_path: str) -> str:
"""Return the function signature plus docstring, without the body.

111
python_pkg/code_tutor/_pytest_runner.py Normal file
View File

@ -0,0 +1,111 @@
"""Running the user's tests in an isolated pytest subprocess.
Split out of :mod:`python_pkg.code_tutor._challenge_support` to keep it under
the 250-line cap. ``_challenge`` imports both entry points back, because
``tests/test_challenge_part3.py`` patches them on that module.
"""
from __future__ import annotations
import ast
import os
from pathlib import Path
import subprocess
import sys
from typing import TYPE_CHECKING
from python_pkg.code_tutor._challenge_support import _project_root
if TYPE_CHECKING:
from rich.console import Console
from python_pkg.code_tutor._analyzer import CodeItem
def _pytest_clean(
test_ids_or_file: list[str],
project_root: Path,
console: Console,
extra_env: dict[str, str] | None = None,
) -> bool:
"""Run pytest on *test_ids_or_file* and return True if all pass.
Coverage is disabled so the project's ``fail_under`` threshold does not
interfere with isolated challenge runs.
Args:
test_ids_or_file: Pytest node IDs or a single temp-file path.
project_root: Working directory for the pytest subprocess.
console: Rich console for output.
extra_env: Extra environment variables (e.g. PYTHONPATH override).
Returns:
``True`` when all collected tests pass.
"""
env = {**os.environ, **(extra_env or {})}
result = subprocess.run(
[
sys.executable,
"-m",
"pytest",
*test_ids_or_file,
"-v",
"--tb=short",
"--no-header",
"-p",
"no:cov",
"--override-ini=addopts=",
],
capture_output=True,
text=True,
cwd=str(project_root),
env=env,
check=False,
)
output = (result.stdout + result.stderr).strip()
if output:
console.print(output)
return result.returncode == 0
def _patch_and_test(
item: CodeItem,
codebase_path: str,
user_code: str,
test_entries: list[tuple[Path, list[str]]],
console: Console,
) -> bool:
"""Replace *item* in source with *user_code*, run tests, then restore.
Args:
item: Code item to patch.
codebase_path: Absolute codebase root path.
user_code: User's implementation text.
test_entries: ``(test_file, node_ids)`` pairs to run.
console: Rich console for output.
Returns:
``True`` if all tests passed.
"""
source_file = Path(codebase_path) / item.file
original = source_file.read_text(encoding="utf-8")
orig_lines = original.splitlines()
before = orig_lines[: item.start_line - 1]
after = orig_lines[item.end_line :]
new_source = "\n".join(before + user_code.splitlines() + after) + "\n"
try:
ast.parse(new_source)
except SyntaxError as exc:
console.print(f"[red]Syntax error in your implementation: {exc}[/red]")
return False
test_ids = [f"{tf}::{nid}" for tf, nids in test_entries for nid in nids]
project_root = _project_root(Path(codebase_path))
try:
source_file.write_text(new_source, encoding="utf-8")
return _pytest_clean(test_ids, project_root, console)
finally:
source_file.write_text(original, encoding="utf-8")

89
python_pkg/code_tutor/_verdict.py Normal file
View File

@ -0,0 +1,89 @@
"""Streaming an LLM verdict behind an elapsed-time display, and parsing it.
Split out of :mod:`python_pkg.code_tutor._challenge_support` to keep it under
the 250-line cap. ``Live`` is imported here rather than there because this is
the only code that used it, and the tests patch it on whichever module the
call resolves through.
"""
from __future__ import annotations
import json
import re
import time
from typing import TYPE_CHECKING
from rich.live import Live
from rich.text import Text
if TYPE_CHECKING:
from rich.console import Console
from python_pkg.code_tutor._llm import Backend
def _stream_verdict(
system: str,
user_msg: str,
backend: Backend,
console: Console,
label: str = "Rating",
) -> str:
"""Stream an LLM call silently while showing an elapsed timer.
Args:
system: System prompt for the judge.
user_msg: User message to judge.
backend: LLM backend with a ``stream(system, user, callback)`` method.
console: Rich console for the live timer.
label: Label prefix shown in the timer (e.g. ``"Rating"``).
Returns:
Accumulated response text.
"""
parts: list[str] = []
start = time.monotonic()
def _on_token(token: str) -> None:
parts.append(token)
elapsed = int(time.monotonic() - start)
live.update(Text(f"{label}... {elapsed}s", style="yellow"))
with Live(
Text(f"{label}... 0s", style="yellow"),
console=console,
refresh_per_second=4,
transient=True,
) as live:
backend.stream(system, user_msg, _on_token)
return "".join(parts)
def _parse_verdict(raw: str) -> tuple[str, str]:
"""Parse ``{"verdict": ..., "gap": ...}`` JSON from *raw*, tolerating fences.
Args:
raw: Raw LLM response text.
Returns:
``(verdict, gap)`` where verdict is ``"PASS"`` or ``"FAIL"``.
"""
clean = re.sub(r"```(?:json)?\s*", "", raw, flags=re.DOTALL).strip()
start = clean.find("{")
end = clean.rfind("}") + 1
if start == -1 or end == 0:
return "FAIL", "Could not parse judge response."
try:
data = json.loads(clean[start:end])
except json.JSONDecodeError:
return "FAIL", "Could not parse judge response."
verdict = str(data.get("verdict", "FAIL")).upper()
if verdict not in {"PASS", "FAIL"}:
verdict = "FAIL"
return verdict, str(data.get("gap", ""))
# ---------------------------------------------------------------------------
# "Write tests first" flow -- helpers
# ---------------------------------------------------------------------------

15
python_pkg/code_tutor/tests/test_challenge.py
View File

@ -14,12 +14,11 @@ from unittest.mock import MagicMock, patch
from python_pkg.code_tutor._challenge_support import (
_collect_lines,
_find_tests,
_patch_and_test,
_project_root,
_pytest_clean,
_scan_test_file,
_show_test_panels,
)
from python_pkg.code_tutor._pytest_runner import _patch_and_test, _pytest_clean
from python_pkg.code_tutor.tests.conftest import _item
# ---------------------------------------------------------------------------
@ -205,7 +204,7 @@ def test_pytest_clean_pass() -> None:
mock_result.stderr = ""
mock_console = MagicMock()
with patch(
"python_pkg.code_tutor._challenge_support.subprocess.run",
"python_pkg.code_tutor._pytest_runner.subprocess.run",
return_value=mock_result,
):
assert (
@ -220,7 +219,7 @@ def test_pytest_clean_fail() -> None:
mock_result.stderr = "FAILED"
mock_console = MagicMock()
with patch(
"python_pkg.code_tutor._challenge_support.subprocess.run",
"python_pkg.code_tutor._pytest_runner.subprocess.run",
return_value=mock_result,
):
assert (
@ -236,7 +235,7 @@ def test_pytest_clean_no_output() -> None:
mock_result.stderr = ""
mock_console = MagicMock()
with patch(
"python_pkg.code_tutor._challenge_support.subprocess.run",
"python_pkg.code_tutor._pytest_runner.subprocess.run",
return_value=mock_result,
):
_pytest_clean([], Path("/proj"), mock_console)
@ -270,9 +269,7 @@ def test_patch_and_test_passes(tmp_path: Path) -> None:
test_file = tmp_path / "test_mod.py"
test_file.write_text("def test_fn(): pass\n", encoding="utf-8")
mock_console = MagicMock()
with patch(
"python_pkg.code_tutor._challenge_support._pytest_clean", return_value=True
):
with patch("python_pkg.code_tutor._pytest_runner._pytest_clean", return_value=True):
result = _patch_and_test(
_item(file="mod.py", start=1, end=2),
str(tmp_path),
@ -292,7 +289,7 @@ def test_patch_and_test_fails(tmp_path: Path) -> None:
test_file.write_text("def test_fn(): pass\n", encoding="utf-8")
mock_console = MagicMock()
with patch(
"python_pkg.code_tutor._challenge_support._pytest_clean", return_value=False
"python_pkg.code_tutor._pytest_runner._pytest_clean", return_value=False
):
result = _patch_and_test(
_item(file="mod.py", start=1, end=2),

15
python_pkg/code_tutor/tests/test_challenge_part2.py
View File

@ -18,9 +18,8 @@ from python_pkg.code_tutor._challenge import (
from python_pkg.code_tutor._challenge_support import (
_extract_signature_block,
_import_hint,
_parse_verdict,
_stream_verdict,
)
from python_pkg.code_tutor._verdict import _parse_verdict, _stream_verdict
from python_pkg.code_tutor.tests.conftest import _item, _make_live_mock
# ---------------------------------------------------------------------------
@ -39,7 +38,7 @@ def test_stream_verdict_accumulates_tokens() -> None:
mock_backend.stream.side_effect = fake_stream
with patch("python_pkg.code_tutor._challenge_support.Live", return_value=live_mock):
with patch("python_pkg.code_tutor._verdict.Live", return_value=live_mock):
result = _stream_verdict("sys", "user", mock_backend, mock_console)
assert result == "" # parts is empty since on_token was never called by stream
@ -59,7 +58,7 @@ def test_stream_verdict_on_token_called() -> None:
mock_backend.stream.side_effect = fake_stream
with patch("python_pkg.code_tutor._challenge_support.Live", return_value=live_mock):
with patch("python_pkg.code_tutor._verdict.Live", return_value=live_mock):
result = _stream_verdict(
"sys", "user", mock_backend, mock_console, label="Test"
)
@ -212,7 +211,7 @@ def test_collect_and_rate_tests_skip() -> None:
mock_console = MagicMock()
live_mock = _make_live_mock()
with patch("python_pkg.code_tutor._challenge_support.Live", return_value=live_mock):
with patch("python_pkg.code_tutor._verdict.Live", return_value=live_mock):
result = _collect_and_rate_tests(
"def fn():",
"explanation",
@ -236,7 +235,7 @@ def test_collect_and_rate_tests_pass_first() -> None:
mock_backend.stream.side_effect = fake_stream
lines_iter = iter(["test code line", "END"])
with patch("python_pkg.code_tutor._challenge_support.Live", return_value=live_mock):
with patch("python_pkg.code_tutor._verdict.Live", return_value=live_mock):
result = _collect_and_rate_tests(
"def fn():",
"explanation",
@ -264,7 +263,7 @@ def test_collect_and_rate_tests_fail_twice() -> None:
]
inputs_iter = iter(attempt_inputs)
with patch("python_pkg.code_tutor._challenge_support.Live", return_value=live_mock):
with patch("python_pkg.code_tutor._verdict.Live", return_value=live_mock):
result = _collect_and_rate_tests(
"def fn():",
"explanation",
@ -297,7 +296,7 @@ def test_collect_and_rate_tests_fail_then_pass() -> None:
inputs = iter(["bad test", "END", "good test", "END"])
with patch("python_pkg.code_tutor._challenge_support.Live", return_value=live_mock):
with patch("python_pkg.code_tutor._verdict.Live", return_value=live_mock):
result = _collect_and_rate_tests(
"def fn():",
"explanation",

2
python_pkg/code_tutor/tests/test_challenge_part3.py
View File

@ -33,7 +33,7 @@ def test_run_user_impl_skip(tmp_path: Path) -> None:
live_mock = _make_live_mock()
with (
patch("python_pkg.code_tutor._challenge_support.Live", return_value=live_mock),
patch("python_pkg.code_tutor._verdict.Live", return_value=live_mock),
patch("python_pkg.code_tutor._challenge.Syntax"),
):
result = _run_user_impl(