mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 20:07:04 +01:00
Stop the endurain importer duplicating RunnerUp's own uploads
RunnerUp can now upload to Endurain directly as well as through the WebDAV inbox, and the two paths cannot see each other: the same run landed twice (activities 14 and 15, identical start time, distances differing only by tcx/gpx rounding). The ledger only knows what this importer handled, so it could not catch it. Add a third dedupe check that asks Endurain what it already holds, queried once per run. Endurain exposes no query-by-start-time endpoint, so a page of recent activities is fetched and compared client-side, with a +/-120s tolerance: the file for activity 15 embeds 21:51:04Z while the server stored 21:51:05Z, so exact equality misses the very duplicate that motivated this. A failed or unauthorised query returns None, never an empty list, so it can never be read as "nothing matches" and upload on the strength of it. Export filenames are parsed as LOCAL wall-clock time. RunnerUp_2026-08-14- 00-58-45_Running is 2026-08-13 22:58:45 UTC; reading it as UTC would move a late-evening run onto the wrong day and defeat the comparison. Also fix the adb fallback, which broke the moment a second device attached: phone_attached() returned true if ANY device was in the 'device' state, then every bare adb call failed with "more than one device/emulator" as a warning on a path nobody watches. The serial is now resolved once and threaded into every call as -s. Ambiguity is settled by configuration, never by guessing -- several devices and no ENDURAIN_ADB_SERIAL means the fallback is skipped. The remote check stays inert until the API key is granted activities:read; until then it falls back to the ledger, which is the previous behaviour. 85 tests, 100% branch coverage on all four modules, no suppressions. The oversized test modules are split by concern to stay under the 250-line cap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GbYEnfGnf1988cdFdGebot
This commit is contained in:
parent
2ab9effa4f
commit
325d5a8baa
@ -0,0 +1,22 @@
|
||||
{
|
||||
"title": "Teach the endurain importer to skip runs already in Endurain, and pin the adb fallback to one device",
|
||||
"objective": "RunnerUp can now upload to Endurain directly as well as via the WebDAV inbox, and the two paths cannot see each other: the same run lands twice (activities 14 and 15, identical start time, distances differing only by tcx/gpx rounding). Add a third dedupe check that asks Endurain what it already holds, and fix the adb fallback, which fails on every call the moment a second device is attached.",
|
||||
"acceptance_criteria": [
|
||||
"A run already present in Endurain is moved to processed/ without being uploaded again, even when the local ledger has never seen it.",
|
||||
"A failed or unauthorised remote query returns None and falls back to the existing ledger dedupe. None is never read as 'no matching activity exists', because that would upload a duplicate on every transport hiccup.",
|
||||
"Start times are compared with a tolerance: the file for activity 15 embeds 21:51:04Z while Endurain stored 21:51:05Z, so exact equality misses the very duplicate that motivated this work.",
|
||||
"Export filenames are parsed as LOCAL wall-clock time; RunnerUp_2026-08-14-00-58-45_Running resolves to 2026-08-13 22:58:45 UTC, not 2026-08-14.",
|
||||
"With several adb devices attached and no serial pinned, the phone fallback is skipped with a warning naming the attached serials and the env var, instead of failing every call with 'more than one device/emulator'.",
|
||||
"ENDURAIN_ADB_SERIAL selects the device when set; a pinned serial that is not attached is refused rather than silently ignored.",
|
||||
"Every adb invocation carries -s <serial>; no serial is hardcoded, since this repo is public.",
|
||||
"The five package modules keep 100% branch coverage with no lint suppressions and no coverage exclusions.",
|
||||
"The phone-side files are never written, moved or deleted: screen-locker reads the same directory as its unlock gate."
|
||||
],
|
||||
"out_of_scope": [
|
||||
"Deleting duplicates that already exist. This work prevents new ones; clearing the 14/15 collision needs session credentials and is the user's action.",
|
||||
"Granting the API key the activities:read scope. Both existing keys hold activities:upload only, so the remote check is inert until that changes; the fail-closed path is what ships working today.",
|
||||
"Changing the WebDAV inbox, the dufs config, or RunnerUp's own uploader.",
|
||||
"Recording the returned activity_id in the ledger. It is null on every existing entry, which is cosmetic and unrelated."
|
||||
],
|
||||
"verifier": ".venv/bin/python -m pytest python_pkg/endurain_import/tests/ -q (expects 85 passed), plus --cov=python_pkg.endurain_import showing 100% with zero missed branches on all five modules, plus .venv/bin/python -m ruff check python_pkg/endurain_import/, plus systemctl --user start endurain-import.service with two devices attached showing the named-serials warning rather than 'more than one device/emulator'."
|
||||
}
|
||||
@ -0,0 +1,71 @@
|
||||
{
|
||||
"intent": "Stop the endurain importer duplicating runs that RunnerUp uploaded to Endurain directly, and stop its adb fallback silently failing whenever more than one device is attached.",
|
||||
"scope": [
|
||||
"python_pkg/endurain_import/sources.py — adb device selection",
|
||||
"python_pkg/endurain_import/ledger.py — start time parsed from the export filename",
|
||||
"python_pkg/endurain_import/upload.py — recent-activity query and start-time comparison",
|
||||
"python_pkg/endurain_import/__main__.py — orchestration wiring",
|
||||
"Non-goal: deleting duplicates that already exist. This prevents new ones; it never deletes.",
|
||||
"Non-goal: changing the WebDAV inbox or the phone-side files (screen-locker reads the same directory)."
|
||||
],
|
||||
"changes": [
|
||||
"sources.py: replaced phone_attached() with attached_serials() + resolve_serial(). The old check returned true when ANY device was in the 'device' state, after which every bare adb call failed with 'more than one device/emulator' as a warning on a path nobody watches. The serial is now threaded into every _adb() call as -s. Ambiguity is resolved by configuration (ENDURAIN_ADB_SERIAL), never by guessing: several devices and no pin means the fallback is skipped outright.",
|
||||
"ledger.py: added activity_start(), which parses the run's start time out of the RunnerUp export filename. RunnerUp names exports in LOCAL wall-clock time, so the value is read in the local zone and returned as UTC; reading it as UTC directly shifts a late-evening run onto the wrong day (RunnerUp_2026-08-14-00-58-45 is 2026-08-13 22:58 UTC).",
|
||||
"upload.py: added EndurainClient.recent_start_times(), which fetches one page of recent activities and returns their start times, plus already_present() which compares with a +/-120s tolerance. Endurain exposes no query-by-start-time endpoint, so existence is decided client-side. The tolerance exists because the recorded value is not the one in the file: a run whose export embeds 21:51:04Z was stored by Endurain as 21:51:05Z, which exact equality would miss.",
|
||||
"upload.py: recent_start_times() returns None (not an empty list) for a transport failure, an auth failure, or an unparseable body, so a failed lookup can never be read as 'no matching activity exists'.",
|
||||
"__main__.py: _process() gained a third dedupe check against remote_starts, queried once per run rather than per file. A None result falls back to the existing digest/activity-key ledger checks, which is the previous behaviour. A run matched remotely is recorded in the ledger so the next run skips it without needing the query."
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"command": "systemctl --user start endurain-import.service && journalctl --user -u endurain-import.service -n 5",
|
||||
"result": "pass",
|
||||
"evidence": "Ran against the live service with two devices genuinely attached (Pixel 6a 23181JEGR08034 over USB, BL9000 over 192.168.1.131:5555). Logged: '2 devices attached (23181JEGR08034, 192.168.1.131:5555); set ENDURAIN_ADB_SERIAL to choose one. Skipping the phone fallback rather than pulling from an arbitrary device.' Before this change the same situation produced 'could not list /sdcard/Documents/RunnerUp: adb: more than one device/emulator'."
|
||||
},
|
||||
{
|
||||
"command": "python -c 'from python_pkg.endurain_import import sources; sources.resolve_serial()' under three real device configurations",
|
||||
"result": "pass",
|
||||
"evidence": "Two devices attached, no env: None + warning naming both serials. ENDURAIN_ADB_SERIAL=23181JEGR08034: returned '23181JEGR08034'. ENDURAIN_ADB_SERIAL=NOPE123: None + warning that the pinned device is not attached."
|
||||
},
|
||||
{
|
||||
"command": "python -c 'from python_pkg.endurain_import.ledger import activity_start' on real export filenames",
|
||||
"result": "pass",
|
||||
"evidence": "RunnerUp_2026-08-22-23-51-04_Running.tcx -> 2026-08-22 21:51:04+00:00 (DB holds 21:51:05, inside tolerance). RunnerUp_Pixel_6a_2026-08-22-23-51-04_Running.gpx -> identical instant, confirming the device model is stripped. RunnerUp_2026-08-14-00-58-45_Running.tcx -> 2026-08-13 22:58:45+00:00, the day-boundary case. garbage.tcx and RunnerUp_2026-13-45-99-99-99_Running.tcx -> None."
|
||||
},
|
||||
{
|
||||
"command": "python -c 'from python_pkg.endurain_import.upload import already_present' at the tolerance boundary",
|
||||
"result": "pass",
|
||||
"evidence": "1s skew (the real 14/15 collision) -> True. 120s -> True. 121s -> False. Unrelated run -> False. Empty known list -> False."
|
||||
},
|
||||
{
|
||||
"command": "python -c 'EndurainClient(...).recent_start_times(1)' against the running server with both API keys",
|
||||
"result": "pass",
|
||||
"evidence": "Both keys returned None and logged 'could not list recent activities: 401 (the API key needs the activities:read scope for duplicate detection)'. Confirmed against the DB: both keys hold scopes [\"activities:upload\"] only. The fail-closed path is therefore exercised end to end; the positive path is covered by unit tests until the scope is granted."
|
||||
},
|
||||
{
|
||||
"command": ".venv/bin/python -m pytest python_pkg/endurain_import/tests/ -q",
|
||||
"result": "pass",
|
||||
"evidence": "85 passed."
|
||||
},
|
||||
{
|
||||
"command": ".venv/bin/python -m pytest python_pkg/endurain_import/tests/ --cov=python_pkg.endurain_import --cov-report=term-missing",
|
||||
"result": "pass",
|
||||
"evidence": "__init__.py, __main__.py (92 stmts, 20 branches), ledger.py (74/12), sources.py (77/30), upload.py (93/20) all at 100% with zero missed branches. No lint suppressions added."
|
||||
},
|
||||
{
|
||||
"command": ".venv/bin/python -m ruff check python_pkg/endurain_import/",
|
||||
"result": "pass",
|
||||
"evidence": "All checks passed."
|
||||
}
|
||||
],
|
||||
"risks": [
|
||||
"The remote check is inert until the Endurain API key is granted the activities:read scope. Until then recent_start_times() returns None every run and dedupe falls back to the ledger, which is exactly the previous behaviour — no regression, but no new protection either.",
|
||||
"A pre-existing duplicate (activities 14 and 15 share a start time) makes Endurain's own get_activity_by_start_time raise MultipleResultsFound, which 500s an upload of THAT run. Other runs are unaffected. This change does not delete duplicates; the collision has to be cleared in the UI.",
|
||||
"The +/-120s tolerance would collapse two genuinely distinct runs starting within two minutes of each other into one. Judged not a real scenario for a running log; the alternative (exact match) demonstrably misses the real 1s recording skew.",
|
||||
"resolve_serial() now refuses to pull when several devices are attached and no serial is pinned. That is a deliberate behaviour change: the fallback stops silently rather than pulling from an arbitrary device. Set ENDURAIN_ADB_SERIAL to restore pulling."
|
||||
],
|
||||
"rollback": [
|
||||
"git revert the commit; the importer returns to phone_attached() and ledger-only dedupe.",
|
||||
"After rollback confirm: systemctl --user start endurain-import.service exits 0, and the ledger at ~/.local/state/endurain-import/ledger.json is unchanged (this change never rewrites existing entries).",
|
||||
"No data migration is involved — the ledger format is unchanged and no server state is written by the new code path (it only issues GETs)."
|
||||
]
|
||||
}
|
||||
@ -3,11 +3,15 @@
|
||||
Run from a systemd user timer. Configuration comes from the environment:
|
||||
|
||||
ENDURAIN_URL base URL (default http://127.0.0.1:8085)
|
||||
ENDURAIN_API_KEY API key with the activities:upload scope (required)
|
||||
ENDURAIN_API_KEY API key with activities:upload (required) and
|
||||
activities:read (optional; enables detection of
|
||||
runs RunnerUp uploaded to Endurain directly)
|
||||
ENDURAIN_INBOX WebDAV inbox (default ~/cloud/RunnerUp)
|
||||
ENDURAIN_STATE ledger directory (default ~/.local/state/endurain-import)
|
||||
ENDURAIN_BULK_DIR bulk_import dir for files the server rejected
|
||||
ENDURAIN_NO_ADB set to 1 to disable the phone fallback
|
||||
ENDURAIN_ADB_SERIAL which phone to pull from when several are attached
|
||||
ENDURAIN_USER_ID Endurain user whose activities are checked (default 1)
|
||||
|
||||
Exit status is 0 only when every file was accounted for; anything left in an
|
||||
ambiguous state exits non-zero so the timer surfaces it rather than logging
|
||||
@ -21,6 +25,7 @@ import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import sys
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import requests
|
||||
|
||||
@ -28,17 +33,27 @@ from python_pkg.endurain_import.ledger import (
|
||||
Entry,
|
||||
Ledger,
|
||||
activity_key,
|
||||
activity_start,
|
||||
file_digest,
|
||||
now_iso,
|
||||
)
|
||||
from python_pkg.endurain_import.sources import inbox_files, pull_from_phone
|
||||
from python_pkg.endurain_import.upload import EndurainClient, Outcome, SupportsUpload
|
||||
from python_pkg.endurain_import.upload import (
|
||||
EndurainClient,
|
||||
Outcome,
|
||||
SupportsUpload,
|
||||
already_present,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from datetime import datetime
|
||||
|
||||
_logger = logging.getLogger("endurain_import")
|
||||
|
||||
_DEFAULT_URL = "http://127.0.0.1:8085"
|
||||
_DEFAULT_INBOX = Path.home() / "cloud" / "RunnerUp"
|
||||
_DEFAULT_STATE = Path.home() / ".local" / "state" / "endurain-import"
|
||||
_DEFAULT_USER_ID = 1
|
||||
|
||||
|
||||
def _configure_logging() -> None:
|
||||
@ -83,18 +98,40 @@ def _process(
|
||||
ledger: Ledger,
|
||||
processed_dir: Path,
|
||||
bulk_dir: Path | None,
|
||||
remote_starts: list[datetime] | None = None,
|
||||
) -> str:
|
||||
"""Import one file. Returns one of: skipped, imported, rejected, ambiguous."""
|
||||
digest = file_digest(path)
|
||||
key = activity_key(path)
|
||||
# Two independent dedupe checks. The digest catches the identical file
|
||||
# arriving twice; the activity key catches the SAME RUN arriving as both
|
||||
# .gpx and .tcx, whose bytes differ and which Endurain would otherwise
|
||||
# import as two separate activities.
|
||||
if digest in ledger or ledger.has_activity(key):
|
||||
# Three dedupe checks, cheapest first. The digest catches the identical
|
||||
# file arriving twice; the activity key catches the SAME RUN arriving as
|
||||
# both .gpx and .tcx, whose bytes differ and which Endurain would
|
||||
# otherwise import as two separate activities.
|
||||
known_locally = digest in ledger or ledger.has_activity(key)
|
||||
# The third check asks Endurain itself, which is the only way to see a run
|
||||
# this importer never handled -- RunnerUp's own uploader writes straight to
|
||||
# the server and leaves no trace in the ledger. ``remote_starts`` is None
|
||||
# when that question could not be answered, and None must never be read as
|
||||
# "nothing matches": the local checks then decide alone.
|
||||
start = activity_start(path)
|
||||
known_remotely = (
|
||||
remote_starts is not None
|
||||
and start is not None
|
||||
and already_present(start, remote_starts)
|
||||
)
|
||||
if known_locally or known_remotely:
|
||||
processed_dir.mkdir(parents=True, exist_ok=True)
|
||||
shutil.move(str(path), str(processed_dir / path.name))
|
||||
_logger.info("%s already imported; moved to processed/", path.name)
|
||||
if known_remotely and not known_locally:
|
||||
# Record it so the next run skips without needing the query again.
|
||||
ledger.record(Entry(digest, key, path.name, None, now_iso()))
|
||||
_logger.info(
|
||||
"%s is already in Endurain (uploaded outside this importer); "
|
||||
"moved to processed/",
|
||||
path.name,
|
||||
)
|
||||
else:
|
||||
_logger.info("%s already imported; moved to processed/", path.name)
|
||||
return "skipped"
|
||||
|
||||
result = client.upload(path)
|
||||
@ -153,9 +190,21 @@ def main() -> int:
|
||||
_logger.info("nothing to import (ledger holds %d file(s))", len(ledger))
|
||||
return 0
|
||||
|
||||
# One query per run, not per file: the page is a snapshot of what Endurain
|
||||
# already holds, and every file in this pass is compared against it.
|
||||
user_id = int(os.environ.get("ENDURAIN_USER_ID", _DEFAULT_USER_ID))
|
||||
remote_starts = client.recent_start_times(user_id)
|
||||
if remote_starts is None:
|
||||
_logger.warning(
|
||||
"duplicate detection against Endurain is unavailable; falling "
|
||||
"back to the local ledger only"
|
||||
)
|
||||
|
||||
counts = {"skipped": 0, "imported": 0, "rejected": 0, "ambiguous": 0}
|
||||
for path in pending:
|
||||
counts[_process(path, client, ledger, processed_dir, bulk_dir)] += 1
|
||||
counts[
|
||||
_process(path, client, ledger, processed_dir, bulk_dir, remote_starts)
|
||||
] += 1
|
||||
|
||||
_logger.info(
|
||||
"done: %(imported)d imported, %(skipped)d already known, "
|
||||
|
||||
@ -21,6 +21,8 @@ if TYPE_CHECKING:
|
||||
_CHUNK = 1 << 20
|
||||
# "RunnerUp", <timestamp>, <sport> -- the minimum for a parseable export name.
|
||||
_MIN_NAME_PARTS = 3
|
||||
# <yyyy>-<mm>-<dd>-<HH>-<MM>-<SS> once split on "-".
|
||||
_STAMP_FIELDS = 6
|
||||
|
||||
|
||||
class LedgerCorruptError(RuntimeError):
|
||||
@ -48,6 +50,37 @@ def activity_key(path: Path) -> str:
|
||||
return stem
|
||||
|
||||
|
||||
def activity_start(path: Path) -> datetime | None:
|
||||
"""Parse the run's start time out of a RunnerUp export filename.
|
||||
|
||||
RunnerUp names exports with LOCAL wall-clock time
|
||||
(RunnerUp_2026-08-14-00-58-45_Running is 00:58 local, 22:58 UTC the day
|
||||
before), so the value is interpreted in the local zone and returned as
|
||||
UTC. Reading it as UTC directly would shift a late-evening run onto the
|
||||
wrong day and defeat the comparison it feeds.
|
||||
"""
|
||||
key = activity_key(path)
|
||||
parts = key.split("_")
|
||||
if len(parts) < _MIN_NAME_PARTS:
|
||||
return None
|
||||
fields = parts[-2].split("-")
|
||||
if len(fields) != _STAMP_FIELDS:
|
||||
return None
|
||||
try:
|
||||
naive = datetime(
|
||||
int(fields[0]),
|
||||
int(fields[1]),
|
||||
int(fields[2]),
|
||||
int(fields[3]),
|
||||
int(fields[4]),
|
||||
int(fields[5]),
|
||||
tzinfo=datetime.now().astimezone().tzinfo,
|
||||
)
|
||||
except ValueError:
|
||||
return None
|
||||
return naive.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def file_digest(path: Path) -> str:
|
||||
"""Return the SHA-256 of ``path``, read incrementally."""
|
||||
digest = hashlib.sha256()
|
||||
|
||||
@ -14,6 +14,7 @@ unlocks the PC; nothing here may delete, move, or rewrite those files.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from typing import TYPE_CHECKING
|
||||
@ -27,6 +28,16 @@ _logger = logging.getLogger(__name__)
|
||||
PHONE_EXPORT_DIR = "/sdcard/Documents/RunnerUp"
|
||||
ACTIVITY_SUFFIXES = (".tcx", ".gpx", ".fit")
|
||||
_ADB_TIMEOUT = 60
|
||||
# Pins which phone to pull from when more than one device is attached.
|
||||
# Without it every adb call fails with "more than one device/emulator" --
|
||||
# silently, as a warning on a path nobody watches -- the moment a second
|
||||
# device shows up. Left unset, the single attached device is used.
|
||||
ADB_SERIAL_ENV = "ENDURAIN_ADB_SERIAL"
|
||||
# Pins which phone to pull from when more than one device is attached. Without
|
||||
# it every adb call fails with "more than one device/emulator" -- silently, as
|
||||
# a warning on the fallback path -- the moment a second device (a spare handset,
|
||||
# an emulator) shows up. Left unset the single attached device is used.
|
||||
ADB_SERIAL_ENV = "ENDURAIN_ADB_SERIAL"
|
||||
# Resolved at call time so a missing adb is reported as "no device" rather than
|
||||
# crashing the whole import run.
|
||||
_ADB_BIN = shutil.which("adb") or "adb"
|
||||
@ -48,11 +59,17 @@ def inbox_files(inbox: Path) -> list[Path]:
|
||||
return sorted(found, key=lambda p: p.stat().st_mtime)
|
||||
|
||||
|
||||
def _adb(args: list[str]) -> tuple[bool, str]:
|
||||
"""Run an adb command, returning (ok, output)."""
|
||||
def _adb(args: list[str], serial: str | None = None) -> tuple[bool, str]:
|
||||
"""Run an adb command, returning (ok, output).
|
||||
|
||||
``serial`` is threaded into the argv as ``-s`` so every call targets one
|
||||
named device. Omitting it is only correct for ``devices`` itself, which is
|
||||
what resolves the serial in the first place.
|
||||
"""
|
||||
prefix = ["-s", serial] if serial else []
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[_ADB_BIN, *args],
|
||||
[_ADB_BIN, *prefix, *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=_ADB_TIMEOUT,
|
||||
@ -63,15 +80,54 @@ def _adb(args: list[str]) -> tuple[bool, str]:
|
||||
return proc.returncode == 0, proc.stdout + proc.stderr
|
||||
|
||||
|
||||
def phone_attached() -> bool:
|
||||
"""True when exactly one adb device is in the 'device' state."""
|
||||
def attached_serials() -> list[str]:
|
||||
"""Return the serials of every device in the 'device' state."""
|
||||
ok, out = _adb(["devices"])
|
||||
if not ok:
|
||||
return False
|
||||
states = [
|
||||
line.split("\t")[1].strip() for line in out.splitlines()[1:] if "\t" in line
|
||||
]
|
||||
return "device" in states
|
||||
return []
|
||||
serials = []
|
||||
for line in out.splitlines()[1:]:
|
||||
if "\t" not in line:
|
||||
continue
|
||||
serial, state = line.split("\t", 1)
|
||||
if state.strip() == "device":
|
||||
serials.append(serial.strip())
|
||||
return serials
|
||||
|
||||
|
||||
def resolve_serial() -> str | None:
|
||||
"""Pick the phone to pull from, or None when the choice is not clear.
|
||||
|
||||
Ambiguity is resolved by configuration, never by guessing: with several
|
||||
devices attached and no ``ENDURAIN_ADB_SERIAL`` set, this returns None so
|
||||
the fallback is skipped outright. The alternative -- letting bare adb pick
|
||||
-- is what produced "more than one device/emulator" as a warning on a path
|
||||
nobody watches.
|
||||
"""
|
||||
pinned = os.environ.get(ADB_SERIAL_ENV, "").strip()
|
||||
serials = attached_serials()
|
||||
if pinned:
|
||||
if pinned in serials:
|
||||
return pinned
|
||||
_logger.warning(
|
||||
"%s=%s is set but that device is not attached (attached: %s)",
|
||||
ADB_SERIAL_ENV,
|
||||
pinned,
|
||||
", ".join(serials) or "none",
|
||||
)
|
||||
return None
|
||||
if not serials:
|
||||
return None
|
||||
if len(serials) > 1:
|
||||
_logger.warning(
|
||||
"%d devices attached (%s); set %s to choose one. Skipping the "
|
||||
"phone fallback rather than pulling from an arbitrary device.",
|
||||
len(serials),
|
||||
", ".join(serials),
|
||||
ADB_SERIAL_ENV,
|
||||
)
|
||||
return None
|
||||
return serials[0]
|
||||
|
||||
|
||||
def pull_from_phone(inbox: Path) -> list[Path]:
|
||||
@ -81,11 +137,12 @@ def pull_from_phone(inbox: Path) -> list[Path]:
|
||||
left alone; content-level deduplication happens later against the ledger,
|
||||
so a name collision here is not authoritative either way.
|
||||
"""
|
||||
if not phone_attached():
|
||||
_logger.info("no adb device attached; skipping phone fallback")
|
||||
serial = resolve_serial()
|
||||
if serial is None:
|
||||
_logger.info("no adb device selected; skipping phone fallback")
|
||||
return []
|
||||
|
||||
ok, out = _adb(["shell", "ls", PHONE_EXPORT_DIR])
|
||||
ok, out = _adb(["shell", "ls", PHONE_EXPORT_DIR], serial)
|
||||
if not ok:
|
||||
_logger.warning("could not list %s: %s", PHONE_EXPORT_DIR, out.strip())
|
||||
return []
|
||||
@ -102,7 +159,7 @@ def pull_from_phone(inbox: Path) -> list[Path]:
|
||||
if target.exists():
|
||||
continue
|
||||
staged = inbox / f".{name}.partial"
|
||||
ok, err = _adb(["pull", f"{PHONE_EXPORT_DIR}/{name}", str(staged)])
|
||||
ok, err = _adb(["pull", f"{PHONE_EXPORT_DIR}/{name}", str(staged)], serial)
|
||||
if not ok:
|
||||
_logger.warning("adb pull failed for %s: %s", name, err.strip())
|
||||
staged.unlink(missing_ok=True)
|
||||
|
||||
71
python_pkg/endurain_import/tests/conftest.py
Normal file
71
python_pkg/endurain_import/tests/conftest.py
Normal file
@ -0,0 +1,71 @@
|
||||
"""Shared fixtures for the endurain_import orchestration tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from python_pkg.endurain_import.upload import Result
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def inbox(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "inbox"
|
||||
path.mkdir()
|
||||
return path
|
||||
|
||||
|
||||
def _file(inbox: Path, name: str = "RunnerUp_ts_Running.tcx") -> Path:
|
||||
path = inbox / name
|
||||
path.write_text("<xml/>")
|
||||
return path
|
||||
|
||||
|
||||
class _StubMainClient:
|
||||
"""Stands in for EndurainClient across a whole main() run."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
result: Result,
|
||||
*,
|
||||
reachable: bool = True,
|
||||
remote: list[datetime] | None = None,
|
||||
) -> None:
|
||||
self._result = result
|
||||
self._reachable = reachable
|
||||
self._remote = remote
|
||||
self.uploads = 0
|
||||
|
||||
def about(self) -> dict[str, object]:
|
||||
if not self._reachable:
|
||||
message = "connection refused"
|
||||
raise OSError(message)
|
||||
return {"version": "test"}
|
||||
|
||||
def recent_start_times(self, _user_id: int) -> list[datetime] | None:
|
||||
return self._remote
|
||||
|
||||
def upload(self, _path: Path) -> Result:
|
||||
self.uploads += 1
|
||||
return self._result
|
||||
|
||||
|
||||
def _env(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path, **extra: str
|
||||
) -> None:
|
||||
monkeypatch.setenv("ENDURAIN_API_KEY", "k")
|
||||
monkeypatch.setenv("ENDURAIN_INBOX", str(inbox))
|
||||
monkeypatch.setenv("ENDURAIN_STATE", str(tmp_path / "state"))
|
||||
monkeypatch.setenv("ENDURAIN_NO_ADB", "1")
|
||||
for key, value in extra.items():
|
||||
monkeypatch.setenv(key, value)
|
||||
|
||||
|
||||
def _patch_client(monkeypatch: pytest.MonkeyPatch, client: _StubMainClient) -> None:
|
||||
import python_pkg.endurain_import.__main__ as mod
|
||||
|
||||
monkeypatch.setattr(mod, "EndurainClient", lambda *_a, **_k: client)
|
||||
119
python_pkg/endurain_import/tests/test_device_selection.py
Normal file
119
python_pkg/endurain_import/tests/test_device_selection.py
Normal file
@ -0,0 +1,119 @@
|
||||
"""Which phone the adb fallback pulls from.
|
||||
|
||||
Ambiguity is settled by configuration, never by guessing: the old check
|
||||
returned true for ANY attached device, after which every call failed with
|
||||
"more than one device/emulator" as a warning on a path nobody watches.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
|
||||
import pytest
|
||||
|
||||
from python_pkg.endurain_import import sources
|
||||
|
||||
|
||||
def _devices(*serials: str) -> str:
|
||||
body = "".join(f"{s}\tdevice\n" for s in serials)
|
||||
return f"List of devices attached\n{body}"
|
||||
|
||||
|
||||
def test_single_device_is_selected_without_configuration(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv(sources.ADB_SERIAL_ENV, raising=False)
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a, _s=None: (True, _devices("ONE")))
|
||||
assert sources.resolve_serial() == "ONE"
|
||||
|
||||
|
||||
def test_no_devices_resolves_to_none(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.delenv(sources.ADB_SERIAL_ENV, raising=False)
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a, _s=None: (True, _devices()))
|
||||
assert sources.resolve_serial() is None
|
||||
|
||||
|
||||
def test_several_devices_without_a_pin_is_refused(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Two devices must not silently become 'adb picks whichever it likes'."""
|
||||
monkeypatch.delenv(sources.ADB_SERIAL_ENV, raising=False)
|
||||
monkeypatch.setattr(
|
||||
sources, "_adb", lambda _a, _s=None: (True, _devices("ONE", "TWO"))
|
||||
)
|
||||
assert sources.resolve_serial() is None
|
||||
|
||||
|
||||
def test_pinned_serial_wins_when_several_are_attached(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setenv(sources.ADB_SERIAL_ENV, "TWO")
|
||||
monkeypatch.setattr(
|
||||
sources, "_adb", lambda _a, _s=None: (True, _devices("ONE", "TWO"))
|
||||
)
|
||||
assert sources.resolve_serial() == "TWO"
|
||||
|
||||
|
||||
def test_pinned_serial_that_is_absent_resolves_to_none(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setenv(sources.ADB_SERIAL_ENV, "GONE")
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a, _s=None: (True, _devices("ONE")))
|
||||
assert sources.resolve_serial() is None
|
||||
|
||||
|
||||
def test_blank_pin_falls_back_to_the_single_device(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setenv(sources.ADB_SERIAL_ENV, " ")
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a, _s=None: (True, _devices("ONE")))
|
||||
assert sources.resolve_serial() == "ONE"
|
||||
|
||||
|
||||
def test_devices_output_without_tabs_is_ignored(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.delenv(sources.ADB_SERIAL_ENV, raising=False)
|
||||
monkeypatch.setattr(
|
||||
sources,
|
||||
"_adb",
|
||||
lambda _a, _s=None: (True, "List of devices attached\n* daemon started *\n"),
|
||||
)
|
||||
assert sources.attached_serials() == []
|
||||
|
||||
|
||||
def test_serial_is_passed_to_every_adb_call(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Without -s, a second attached device breaks every call with a warning."""
|
||||
seen: list[list[str]] = []
|
||||
|
||||
def _adb(args: list[str], serial: str | None = None) -> tuple[bool, str]:
|
||||
seen.append(["-s", serial or "", *args])
|
||||
if args[0] == "shell":
|
||||
return True, "RunnerUp_ts_Running.tcx\n"
|
||||
Path(args[2]).write_text("pulled")
|
||||
return True, ""
|
||||
|
||||
monkeypatch.setattr(sources, "resolve_serial", lambda: "PIXEL")
|
||||
monkeypatch.setattr(sources, "_adb", _adb)
|
||||
assert len(sources.pull_from_phone(tmp_path)) == 1
|
||||
assert all(call[1] == "PIXEL" for call in seen)
|
||||
|
||||
|
||||
def test_adb_builds_argv_with_the_serial(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
captured: dict[str, list[str]] = {}
|
||||
|
||||
class _Proc:
|
||||
returncode = 0
|
||||
stdout = "ok"
|
||||
stderr = ""
|
||||
|
||||
def _run(argv: list[str], **_kwargs: object) -> _Proc:
|
||||
captured["argv"] = argv
|
||||
return _Proc()
|
||||
|
||||
monkeypatch.setattr(subprocess, "run", _run)
|
||||
sources._adb(["shell", "ls"], "PIXEL")
|
||||
assert captured["argv"][1:4] == ["-s", "PIXEL", "shell"]
|
||||
@ -2,11 +2,13 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from python_pkg.endurain_import import ledger
|
||||
from python_pkg.endurain_import.ledger import (
|
||||
Entry,
|
||||
Ledger,
|
||||
@ -93,3 +95,40 @@ def test_non_dict_ledger_is_ignored(tmp_path: Path) -> None:
|
||||
path = tmp_path / "l.json"
|
||||
path.write_text("[]")
|
||||
assert len(Ledger(path)) == 0
|
||||
|
||||
|
||||
def test_activity_start_reads_local_wall_time() -> None:
|
||||
"""RunnerUp names exports in local time, so 00:58 is the previous day UTC."""
|
||||
parsed = ledger.activity_start(Path("RunnerUp_2026-08-14-00-58-45_Running.tcx"))
|
||||
assert parsed is not None
|
||||
assert parsed.tzinfo is not None
|
||||
expected = datetime(
|
||||
2026, 8, 14, 0, 58, 45, tzinfo=datetime.now().astimezone().tzinfo
|
||||
).astimezone(timezone.utc)
|
||||
assert parsed == expected
|
||||
|
||||
|
||||
def test_activity_start_ignores_the_device_model() -> None:
|
||||
plain = ledger.activity_start(Path("RunnerUp_2026-08-22-23-51-04_Running.tcx"))
|
||||
with_model = ledger.activity_start(
|
||||
Path("RunnerUp_Pixel_6a_2026-08-22-23-51-04_Running.gpx")
|
||||
)
|
||||
assert plain == with_model
|
||||
|
||||
|
||||
def test_activity_start_of_an_unparseable_name_is_none() -> None:
|
||||
assert ledger.activity_start(Path("whatever.tcx")) is None
|
||||
|
||||
|
||||
def test_activity_start_rejects_a_non_timestamp() -> None:
|
||||
assert ledger.activity_start(Path("RunnerUp_notatimestamp_Running.tcx")) is None
|
||||
|
||||
|
||||
def test_activity_start_rejects_an_impossible_date() -> None:
|
||||
assert (
|
||||
ledger.activity_start(Path("RunnerUp_2026-13-45-99-99-99_Running.tcx")) is None
|
||||
)
|
||||
|
||||
|
||||
def test_activity_start_rejects_wrong_field_count() -> None:
|
||||
assert ledger.activity_start(Path("RunnerUp_2026-08-22_Running.tcx")) is None
|
||||
|
||||
@ -7,56 +7,15 @@ from pathlib import Path
|
||||
import pytest
|
||||
|
||||
from python_pkg.endurain_import.__main__ import _route_rejected, main
|
||||
from python_pkg.endurain_import.tests.conftest import (
|
||||
_env,
|
||||
_file,
|
||||
_patch_client,
|
||||
_StubMainClient,
|
||||
)
|
||||
from python_pkg.endurain_import.upload import Outcome, Result
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def inbox(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "inbox"
|
||||
path.mkdir()
|
||||
return path
|
||||
|
||||
|
||||
def _file(inbox: Path, name: str = "RunnerUp_ts_Running.tcx") -> Path:
|
||||
path = inbox / name
|
||||
path.write_text("<xml/>")
|
||||
return path
|
||||
|
||||
|
||||
class _StubMainClient:
|
||||
"""Stands in for EndurainClient across a whole main() run."""
|
||||
|
||||
def __init__(self, result: Result, *, reachable: bool = True) -> None:
|
||||
self._result = result
|
||||
self._reachable = reachable
|
||||
|
||||
def about(self) -> dict[str, object]:
|
||||
if not self._reachable:
|
||||
message = "connection refused"
|
||||
raise OSError(message)
|
||||
return {"version": "test"}
|
||||
|
||||
def upload(self, _path: Path) -> Result:
|
||||
return self._result
|
||||
|
||||
|
||||
def _env(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path, **extra: str
|
||||
) -> None:
|
||||
monkeypatch.setenv("ENDURAIN_API_KEY", "k")
|
||||
monkeypatch.setenv("ENDURAIN_INBOX", str(inbox))
|
||||
monkeypatch.setenv("ENDURAIN_STATE", str(tmp_path / "state"))
|
||||
monkeypatch.setenv("ENDURAIN_NO_ADB", "1")
|
||||
for key, value in extra.items():
|
||||
monkeypatch.setenv(key, value)
|
||||
|
||||
|
||||
def _patch_client(monkeypatch: pytest.MonkeyPatch, client: _StubMainClient) -> None:
|
||||
import python_pkg.endurain_import.__main__ as mod
|
||||
|
||||
monkeypatch.setattr(mod, "EndurainClient", lambda *_a, **_k: client)
|
||||
|
||||
|
||||
def test_main_returns_one_when_unreachable(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path
|
||||
) -> None:
|
||||
|
||||
111
python_pkg/endurain_import/tests/test_remote_dedupe.py
Normal file
111
python_pkg/endurain_import/tests/test_remote_dedupe.py
Normal file
@ -0,0 +1,111 @@
|
||||
"""Skipping runs Endurain already holds.
|
||||
|
||||
RunnerUp's own uploader writes straight to the server and leaves no ledger
|
||||
trace, so the server is the only place these runs are visible.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from python_pkg.endurain_import.__main__ import main
|
||||
from python_pkg.endurain_import.tests.conftest import (
|
||||
_env,
|
||||
_file,
|
||||
_patch_client,
|
||||
_StubMainClient,
|
||||
)
|
||||
from python_pkg.endurain_import.upload import Outcome, Result
|
||||
|
||||
# The name encodes 23:51:04 LOCAL time; the UTC instant depends on the zone,
|
||||
# so these derive it the same way production does rather than hardcoding it.
|
||||
_RUN_NAME = "RunnerUp_2026-08-22-23-51-04_Running.tcx"
|
||||
|
||||
|
||||
def _run_start() -> datetime:
|
||||
return datetime(
|
||||
2026, 8, 22, 23, 51, 4, tzinfo=datetime.now().astimezone().tzinfo
|
||||
).astimezone(timezone.utc)
|
||||
|
||||
|
||||
def test_run_already_in_endurain_is_skipped_not_reuploaded(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path
|
||||
) -> None:
|
||||
"""RunnerUp's own uploader leaves no ledger trace; only the server knows."""
|
||||
_env(monkeypatch, tmp_path, inbox)
|
||||
_file(inbox, _RUN_NAME)
|
||||
client = _StubMainClient(Result(Outcome.OK, 1, "ok"), remote=[_run_start()])
|
||||
_patch_client(monkeypatch, client)
|
||||
assert main() == 0
|
||||
assert client.uploads == 0
|
||||
assert (inbox / "processed" / _RUN_NAME).exists()
|
||||
|
||||
|
||||
def test_remote_skip_is_recorded_so_the_next_run_needs_no_query(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path
|
||||
) -> None:
|
||||
_env(monkeypatch, tmp_path, inbox)
|
||||
_file(inbox, _RUN_NAME)
|
||||
_patch_client(
|
||||
monkeypatch,
|
||||
_StubMainClient(Result(Outcome.OK, 1, "ok"), remote=[_run_start()]),
|
||||
)
|
||||
assert main() == 0
|
||||
assert _RUN_NAME in (tmp_path / "state" / "ledger.json").read_text()
|
||||
|
||||
|
||||
def test_a_genuinely_new_run_still_uploads(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path
|
||||
) -> None:
|
||||
"""The remote check must not become a blanket refusal to upload."""
|
||||
_env(monkeypatch, tmp_path, inbox)
|
||||
_file(inbox, _RUN_NAME)
|
||||
other = datetime(2026, 1, 1, tzinfo=timezone.utc)
|
||||
client = _StubMainClient(Result(Outcome.OK, 7, "ok"), remote=[other])
|
||||
_patch_client(monkeypatch, client)
|
||||
assert main() == 0
|
||||
assert client.uploads == 1
|
||||
|
||||
|
||||
def test_unavailable_query_falls_back_to_the_ledger(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path
|
||||
) -> None:
|
||||
"""A failed lookup must not read as 'nothing exists' -- nor block imports."""
|
||||
_env(monkeypatch, tmp_path, inbox)
|
||||
_file(inbox, _RUN_NAME)
|
||||
client = _StubMainClient(Result(Outcome.OK, 1, "ok"), remote=None)
|
||||
_patch_client(monkeypatch, client)
|
||||
assert main() == 0
|
||||
assert client.uploads == 1
|
||||
|
||||
|
||||
def test_unparseable_filename_falls_back_to_the_ledger(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path
|
||||
) -> None:
|
||||
"""No timestamp in the name means the remote check cannot apply."""
|
||||
_env(monkeypatch, tmp_path, inbox)
|
||||
_file(inbox, "mystery.tcx")
|
||||
client = _StubMainClient(Result(Outcome.OK, 1, "ok"), remote=[_run_start()])
|
||||
_patch_client(monkeypatch, client)
|
||||
assert main() == 0
|
||||
assert client.uploads == 1
|
||||
|
||||
|
||||
def test_user_id_is_configurable(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, inbox: Path
|
||||
) -> None:
|
||||
_env(monkeypatch, tmp_path, inbox, ENDURAIN_USER_ID="7")
|
||||
_file(inbox, _RUN_NAME)
|
||||
seen: list[int] = []
|
||||
|
||||
class _Recording(_StubMainClient):
|
||||
def recent_start_times(self, user_id: int) -> list[datetime] | None:
|
||||
seen.append(user_id)
|
||||
return []
|
||||
|
||||
_patch_client(monkeypatch, _Recording(Result(Outcome.OK, 1, "ok")))
|
||||
assert main() == 0
|
||||
assert seen == [7]
|
||||
139
python_pkg/endurain_import/tests/test_remote_lookup.py
Normal file
139
python_pkg/endurain_import/tests/test_remote_lookup.py
Normal file
@ -0,0 +1,139 @@
|
||||
"""Asking Endurain what it already holds.
|
||||
|
||||
A failed lookup must return None, never an empty list: read as "nothing
|
||||
matches" it would upload a duplicate on every transport hiccup.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
import requests
|
||||
|
||||
from python_pkg.endurain_import import upload
|
||||
|
||||
|
||||
class _Resp:
|
||||
def __init__(self, status: int, payload: object, *, bad_json: bool = False) -> None:
|
||||
self.status_code = status
|
||||
self._payload = payload
|
||||
self._bad_json = bad_json
|
||||
|
||||
def json(self) -> object:
|
||||
if self._bad_json:
|
||||
message = "not json"
|
||||
raise ValueError(message)
|
||||
return self._payload
|
||||
|
||||
|
||||
def _client_with_get(
|
||||
monkeypatch: pytest.MonkeyPatch, result: object
|
||||
) -> upload.EndurainClient:
|
||||
client = upload.EndurainClient("http://x", "k")
|
||||
|
||||
def _get(*_a: object, **_k: object) -> object:
|
||||
if isinstance(result, Exception):
|
||||
raise result
|
||||
return result
|
||||
|
||||
monkeypatch.setattr(client._session, "get", _get)
|
||||
return client
|
||||
|
||||
|
||||
def test_recent_start_times_parses_offsets(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
client = _client_with_get(
|
||||
monkeypatch,
|
||||
_Resp(200, [{"start_time": "2026-08-22T23:51:05+02:00"}]),
|
||||
)
|
||||
times = client.recent_start_times(1)
|
||||
assert times is not None
|
||||
assert times[0] == datetime(2026, 8, 22, 21, 51, 5, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def test_recent_start_times_treats_naive_as_utc(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
client = _client_with_get(
|
||||
monkeypatch, _Resp(200, [{"start_time": "2026-08-22T21:51:05"}])
|
||||
)
|
||||
times = client.recent_start_times(1)
|
||||
assert times == [datetime(2026, 8, 22, 21, 51, 5, tzinfo=timezone.utc)]
|
||||
|
||||
|
||||
def test_recent_start_times_accepts_zulu(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
client = _client_with_get(
|
||||
monkeypatch, _Resp(200, [{"start_time": "2026-08-22T21:51:05Z"}])
|
||||
)
|
||||
times = client.recent_start_times(1)
|
||||
assert times == [datetime(2026, 8, 22, 21, 51, 5, tzinfo=timezone.utc)]
|
||||
|
||||
|
||||
def test_unparseable_entries_are_dropped_not_fatal(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
client = _client_with_get(
|
||||
monkeypatch,
|
||||
_Resp(
|
||||
200,
|
||||
[
|
||||
{"start_time": "nonsense"},
|
||||
{"start_time": None},
|
||||
{"start_time": ""},
|
||||
{"no_start_time": 1},
|
||||
"not-a-dict",
|
||||
{"start_time": "2026-08-22T21:51:05Z"},
|
||||
],
|
||||
),
|
||||
)
|
||||
assert client.recent_start_times(1) == [
|
||||
datetime(2026, 8, 22, 21, 51, 5, tzinfo=timezone.utc)
|
||||
]
|
||||
|
||||
|
||||
def test_auth_failure_is_unknown_not_empty(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""401 must not read as 'no activities exist' -- that would duplicate."""
|
||||
client = _client_with_get(monkeypatch, _Resp(401, None))
|
||||
assert client.recent_start_times(1) is None
|
||||
|
||||
|
||||
def test_transport_failure_is_unknown(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
client = _client_with_get(monkeypatch, requests.ConnectionError("down"))
|
||||
assert client.recent_start_times(1) is None
|
||||
|
||||
|
||||
def test_non_json_body_is_unknown(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
client = _client_with_get(monkeypatch, _Resp(200, None, bad_json=True))
|
||||
assert client.recent_start_times(1) is None
|
||||
|
||||
|
||||
def test_non_list_body_is_unknown(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
client = _client_with_get(monkeypatch, _Resp(200, {"detail": "nope"}))
|
||||
assert client.recent_start_times(1) is None
|
||||
|
||||
|
||||
def test_already_present_tolerates_recording_skew() -> None:
|
||||
"""The file said 21:51:04Z; Endurain stored 21:51:05Z. Same run."""
|
||||
start = datetime(2026, 8, 22, 21, 51, 4, tzinfo=timezone.utc)
|
||||
known = [datetime(2026, 8, 22, 21, 51, 5, tzinfo=timezone.utc)]
|
||||
assert upload.already_present(start, known)
|
||||
|
||||
|
||||
def test_already_present_rejects_a_different_run() -> None:
|
||||
start = datetime(2026, 8, 22, 21, 51, 4, tzinfo=timezone.utc)
|
||||
known = [datetime(2026, 8, 22, 19, 0, 0, tzinfo=timezone.utc)]
|
||||
assert not upload.already_present(start, known)
|
||||
|
||||
|
||||
def test_tolerance_boundary_is_inclusive() -> None:
|
||||
start = datetime(2026, 8, 22, 21, 51, 4, tzinfo=timezone.utc)
|
||||
inside = [datetime(2026, 8, 22, 21, 53, 4, tzinfo=timezone.utc)]
|
||||
outside = [datetime(2026, 8, 22, 21, 53, 5, tzinfo=timezone.utc)]
|
||||
assert upload.already_present(start, inside)
|
||||
assert not upload.already_present(start, outside)
|
||||
|
||||
|
||||
def test_already_present_against_nothing_is_false() -> None:
|
||||
assert not upload.already_present(
|
||||
datetime(2026, 8, 22, 21, 51, 4, tzinfo=timezone.utc), []
|
||||
)
|
||||
@ -49,40 +49,42 @@ def test_inbox_is_ordered_oldest_first(tmp_path: Path) -> None:
|
||||
]
|
||||
|
||||
|
||||
def test_phone_attached_parses_adb_devices(
|
||||
def test_attached_serials_parses_adb_devices(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(
|
||||
sources,
|
||||
"_adb",
|
||||
lambda _a: (True, "List of devices attached\nABC123\tdevice\n"),
|
||||
lambda _a, _s=None: (True, "List of devices attached\nABC123\tdevice\n"),
|
||||
)
|
||||
assert sources.phone_attached()
|
||||
assert sources.attached_serials() == ["ABC123"]
|
||||
|
||||
|
||||
def test_phone_unauthorised_is_not_attached(
|
||||
def test_unauthorised_device_is_not_listed(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(
|
||||
sources,
|
||||
"_adb",
|
||||
lambda _a: (True, "List of devices attached\nABC123\tunauthorized\n"),
|
||||
lambda _a, _s=None: (True, "List of devices attached\nABC123\tunauthorized\n"),
|
||||
)
|
||||
assert not sources.phone_attached()
|
||||
assert sources.attached_serials() == []
|
||||
|
||||
|
||||
def test_pull_skips_when_no_device(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
monkeypatch.setattr(sources, "phone_attached", lambda: False)
|
||||
monkeypatch.setattr(sources, "resolve_serial", lambda: None)
|
||||
assert sources.pull_from_phone(tmp_path) == []
|
||||
|
||||
|
||||
def test_pull_ignores_already_present_files(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
monkeypatch.setattr(sources, "phone_attached", lambda: True)
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a: (True, "RunnerUp_ts_Running.tcx\n"))
|
||||
monkeypatch.setattr(sources, "resolve_serial", lambda: "ABC123")
|
||||
monkeypatch.setattr(
|
||||
sources, "_adb", lambda _a, _s=None: (True, "RunnerUp_ts_Running.tcx\n")
|
||||
)
|
||||
(tmp_path / "RunnerUp_ts_Running.tcx").write_text("already here")
|
||||
assert sources.pull_from_phone(tmp_path) == []
|
||||
|
||||
@ -91,9 +93,9 @@ def test_failed_pull_leaves_no_partial_file(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""A half-transferred file must never look like a complete activity."""
|
||||
monkeypatch.setattr(sources, "phone_attached", lambda: True)
|
||||
monkeypatch.setattr(sources, "resolve_serial", lambda: "ABC123")
|
||||
|
||||
def _adb(args: list[str]) -> tuple[bool, str]:
|
||||
def _adb(args: list[str], _serial: str | None = None) -> tuple[bool, str]:
|
||||
if args[0] == "shell":
|
||||
return True, "RunnerUp_ts_Running.tcx\n"
|
||||
return False, "pull failed"
|
||||
@ -137,27 +139,27 @@ def test_adb_returns_output_on_success(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
assert out == "out"
|
||||
|
||||
|
||||
def test_phone_attached_false_when_adb_fails(
|
||||
def test_no_serials_when_adb_fails(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a: (False, "boom"))
|
||||
assert not sources.phone_attached()
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a, _s=None: (False, "boom"))
|
||||
assert sources.attached_serials() == []
|
||||
|
||||
|
||||
def test_pull_reports_unlistable_directory(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
monkeypatch.setattr(sources, "phone_attached", lambda: True)
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a: (False, "no such dir"))
|
||||
monkeypatch.setattr(sources, "resolve_serial", lambda: "ABC123")
|
||||
monkeypatch.setattr(sources, "_adb", lambda _a, _s=None: (False, "no such dir"))
|
||||
assert sources.pull_from_phone(tmp_path) == []
|
||||
|
||||
|
||||
def test_successful_pull_moves_into_place(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
monkeypatch.setattr(sources, "phone_attached", lambda: True)
|
||||
monkeypatch.setattr(sources, "resolve_serial", lambda: "ABC123")
|
||||
|
||||
def _adb(args: list[str]) -> tuple[bool, str]:
|
||||
def _adb(args: list[str], _serial: str | None = None) -> tuple[bool, str]:
|
||||
if args[0] == "shell":
|
||||
return True, "RunnerUp_ts_Running.tcx\nnotes.txt\n"
|
||||
Path(args[2]).write_text("pulled")
|
||||
|
||||
@ -14,6 +14,7 @@ rejected" and "might have been committed" decides whether a retry is safe:
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from enum import Enum
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Any, Protocol
|
||||
@ -26,6 +27,15 @@ if TYPE_CHECKING:
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
UPLOAD_PATH = "/api/v1/activities/create/upload"
|
||||
# Endurain exposes no "does this run exist" query, so existence is decided by
|
||||
# fetching a recent page and comparing start times client-side.
|
||||
RECENT_PATH = "/api/v1/activities/user/{user_id}/page_number/1/num_records/{n}"
|
||||
_RECENT_RECORDS = 50
|
||||
# Start times are compared with a tolerance because the recorded value is not
|
||||
# the one in the file: a run whose export embeds 21:51:04Z was stored by
|
||||
# Endurain as 21:51:05Z. Two genuinely distinct runs starting inside two
|
||||
# minutes of each other is not a real scenario, so the window is generous.
|
||||
_START_TOLERANCE_SECONDS = 120
|
||||
# The API key goes in X-API-Key. Endurain's own OpenAPI schema declares the
|
||||
# APIKeyHeader security scheme as name "X-Client-Type", which is wrong --
|
||||
# verified against the running server: X-API-Key with a bad key returns
|
||||
@ -48,6 +58,10 @@ class SupportsUpload(Protocol):
|
||||
"""Upload one activity file and report what happened."""
|
||||
... # pragma: no cover - structural typing only, never executed
|
||||
|
||||
def recent_start_times(self, user_id: int) -> list[datetime] | None:
|
||||
"""Return recent activity start times, or None if unknowable."""
|
||||
... # pragma: no cover - structural typing only, never executed
|
||||
|
||||
|
||||
class Outcome(Enum):
|
||||
"""What happened to one upload attempt."""
|
||||
@ -82,6 +96,42 @@ class EndurainClient:
|
||||
body: dict[str, Any] = resp.json()
|
||||
return body
|
||||
|
||||
def recent_start_times(self, user_id: int) -> list[datetime] | None:
|
||||
"""Return start times of recent activities, or None if unknowable.
|
||||
|
||||
None means "the question could not be answered" -- a transport failure,
|
||||
an auth failure, or a body that did not parse. It is deliberately not
|
||||
an empty list: callers must not read a failed lookup as "no matching
|
||||
activity exists" and upload on the strength of it.
|
||||
"""
|
||||
url = f"{self._base}{RECENT_PATH}".format(user_id=user_id, n=_RECENT_RECORDS)
|
||||
try:
|
||||
resp = self._session.get(url, timeout=30)
|
||||
except requests.RequestException as exc:
|
||||
_logger.warning("could not list recent activities: %s", exc)
|
||||
return None
|
||||
if resp.status_code >= _HTTP_BAD_REQUEST:
|
||||
_logger.warning(
|
||||
"could not list recent activities: %s (the API key needs the "
|
||||
"activities:read scope for duplicate detection)",
|
||||
resp.status_code,
|
||||
)
|
||||
return None
|
||||
try:
|
||||
body = resp.json()
|
||||
except ValueError:
|
||||
_logger.warning("recent activities response was not JSON")
|
||||
return None
|
||||
if not isinstance(body, list):
|
||||
_logger.warning("recent activities response was not a list")
|
||||
return None
|
||||
return [
|
||||
parsed
|
||||
for item in body
|
||||
if isinstance(item, dict)
|
||||
and (parsed := _parse_start(item.get("start_time"))) is not None
|
||||
]
|
||||
|
||||
def upload(self, path: Path) -> Result:
|
||||
"""Upload one activity file."""
|
||||
try:
|
||||
@ -120,3 +170,26 @@ def _activity_id(resp: requests.Response) -> int | None:
|
||||
if isinstance(value, int):
|
||||
return value
|
||||
return None
|
||||
|
||||
|
||||
def _parse_start(raw: object) -> datetime | None:
|
||||
"""Parse an Endurain start_time into an aware UTC datetime."""
|
||||
if not isinstance(raw, str) or not raw:
|
||||
return None
|
||||
try:
|
||||
parsed = datetime.fromisoformat(raw.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
# Endurain returns local-offset timestamps; a naive value is read as UTC
|
||||
# so the comparison below never comes down to the reader's timezone.
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed.astimezone(timezone.utc)
|
||||
|
||||
|
||||
def already_present(start: datetime, known: list[datetime]) -> bool:
|
||||
"""True when ``start`` matches a known activity within the tolerance."""
|
||||
return any(
|
||||
abs((start - other).total_seconds()) <= _START_TOLERANCE_SECONDS
|
||||
for other in known
|
||||
)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user