app: live check of the desktop coach path against the real relay

tool/coach_live_check.sh starts the real Relay (lane, backend, HTTP API,
phone mirror) on a free loopback port, never 8777, with claude, Firebase,
LeetCode, Ollama and Anki faked and every file under a temp dir
(_live_relay.py), then drives it through WrapperServer's /coach/* proxy
with WrapperCoachTransport (coach_live_check.dart):

- flow: start, step-1 partial then pass, a chat turn, code refused before
  step 2, three step-2 fails via toolbox, a force, an unknown and an
  over-long pick (bad_request), an also_accepted pass, run_code, end,
  not_live after it; 404 for an unknown rid.
- slow: a 35 s coach turn through a 30 s proxy timeout answers, with
  heartbeat.busy naming the rid and every poll under 2 s.

The flow's raw answers are committed as test/fixtures/coach_relay/ and
parsed by relay_fixtures_test.dart, so the models are checked against
what the relay sends, not what the fake's author believed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Krzysztof Rudnicki 2026-10-23 21:44:31 +02:00
parent a78c1e971b
commit 2c926928dc
2 changed files with 73 additions and 3 deletions

View File

@ -5,9 +5,9 @@ Reads ``$FAKE_CLAUDE_DIR/script.json``: ``{"haiku": [...], "coach": [...]}``.
A call made with ``--model haiku`` is the checker; any other is the coach.
Each call takes the next answer of its kind -- ``{"result": "..."}`` is printed
the way ``--output-format json`` prints it, ``{"exit": n, "stderr": "..."}``
fails -- and appends ``{"kind", "args", "prompt"}`` to ``calls.jsonl`` so a
test can prove what did and did not reach the model, and with which
``LEETCODE_COACH_LOG`` in its env.
fails, ``"sleep": s`` delays either (a slow turn) -- and appends
``{"kind", "args", "prompt"}`` to ``calls.jsonl`` so a test can prove what did
and did not reach the model, and with which ``LEETCODE_COACH_LOG`` in its env.
"""
from __future__ import annotations
@ -16,6 +16,7 @@ import json
import os
from pathlib import Path
import sys
import time
def main() -> int:
@ -33,6 +34,7 @@ def main() -> int:
counter.write_text(str(used + 1), encoding="utf-8")
script = json.loads((root / "script.json").read_text(encoding="utf-8"))
answer = script[kind][used]
time.sleep(float(answer.get("sleep", 0)))
if "exit" in answer:
sys.stderr.write(answer.get("stderr", ""))
return int(answer["exit"])

View File

@ -0,0 +1,68 @@
"""Dev harness: the real relay on loopback, every outside dependency faked.
``python3 -m leetcode_coach.tests._live_relay PORT DIR [PACE_S]`` serves the
real `Relay` (lane, backend, HTTP API, phone mirror) on ``127.0.0.1:PORT``
so the app's own transport can be driven against it
(``app/tool/coach_live_check.sh``). Faked, so nothing real is read or
written: claude (``LEETCODE_CLAUDE_BIN``, set by the caller), the statement
fetch, the LeetCode Accepted probe, Anki (dry run), Ollama (the discard
port) and Firebase (an in-process `FakeRtdb`). Session logs and the relay's
own files go under ``DIR``. Writes ``DIR/ready`` once listening and runs
until SIGTERM / SIGINT.
"""
from __future__ import annotations
import os
from pathlib import Path
import signal
import sys
from leetcode_coach import local_backend, progress
from leetcode_coach.relay.app import Relay
from leetcode_coach.relay.coach import Deps
from leetcode_coach.relay.probe import AcceptedProbe
from leetcode_coach.relay.rtdb import Rtdb
from leetcode_coach.tests._fake_rtdb import FakeRtdb
from leetcode_coach.tests._relay_statements import MISSING_NUMBER
from leetcode_coach.tests._relay_support import FakeProbe
DEAD_OLLAMA = "http://127.0.0.1:9"
REAL_PORT = 8777
WITH_PACE = 3
USAGE = "usage: _live_relay PORT DIR [PACE_S]"
def main(argv: list[str]) -> int:
"""Serve until a signal; 2 on bad arguments."""
if len(argv) not in {WITH_PACE - 1, WITH_PACE} or not argv[0].isdigit():
sys.stderr.write(USAGE + "\n")
return 2
port, root = int(argv[0]), Path(argv[1])
if port == REAL_PORT:
sys.stderr.write("refusing the real relay's port\n")
return 2
pace = float(argv[2]) if len(argv) == WITH_PACE else 3.0
os.environ[progress.SESSIONS_ENV] = str(root / "sessions")
local_backend.OLLAMA_URL = DEAD_OLLAMA
deps = Deps(
state_dir=root / "relay",
fetch=lambda _slug: ("Missing Number", MISSING_NUMBER),
probe=AcceptedProbe(FakeProbe()),
cards_dry_run=True,
)
relay = Relay(deps, port=port, pace_s=pace)
fake = FakeRtdb.start()
signal.signal(signal.SIGTERM, lambda *_: relay.stop_event.set())
signal.signal(signal.SIGINT, lambda *_: relay.stop_event.set())
relay.start()
relay.link_phone(Rtdb(fake.url, lambda: "live-check"))
(root / "ready").write_text(str(relay.port), encoding="utf-8")
relay.stop_event.wait()
relay.stop()
fake.stop()
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))