utils/crdt-sync/crdt_sync/_github.py
Krzysztof Rudnicki 02e2e13790 crdt_sync: add a Firebase Realtime Database transport behind a RemoteStore seam
Extracts the three-method remote surface GitHub was reached through into a
`RemoteStore` interface, then implements it against Firebase Realtime
Database over REST, so apps can move backend with a constructor change.

Why REST and not the SDKs: official FlutterFire has no Linux desktop
support, and `wake_alarm`'s PC side is a dependency-free systemd job. The
RTDB REST endpoints are plain HTTPS and behave identically on Linux,
Android and headless. Firebase is a transport, never the datastore -- each
app keeps its local log as the source of truth, so offline-first survives.

Why RTDB and not Firestore: on the Spark plan RTDB bills only storage and
bandwidth with no per-operation quota, so a runaway loop cannot exhaust a
daily budget and silently stop syncing mid-day.

Two savings make the no-cost tier a guarantee rather than a hope. Measured
against the GitHub-backed history this replaces: 4894 of 5542 commits
(88.3%) were byte-identical no-ops, and every tick re-downloaded every
peer's whole log. Both are now driven by one sha256 revision -- an
unchanged log is not re-pushed, and an unchanged peer is not re-fetched.
That turns egress from "ticks x log size" (~700 MB/month, and growing with
the log) into "edits" (~40 MB/month against a 10 GiB cap).

Revisions are one node per device, written only by its owner, rather than
one shared map -- a whole-map write would erase every peer's entry, after
which those peers would look permanently unchanged and never be fetched
again. Per-device keys make that impossible rather than merely avoided.

`.` is illegal in an RTDB key and the REST `.json` is a format suffix, so
paths get a reversible `~XX` escape (`log.json` -> `log~2Ejson`). Lossy
extension-stripping would break `todo`, whose listings return filenames.

`MirrorStore` / `MirrorSyncClient` dual-writes during the cutover. Reads
consult *both* backends: with primary-only reads a migrated device would
never see an un-migrated peer's writes while still mirroring its own back
-- silent one-directional convergence with no error raised. Writes are
asymmetric on purpose: a primary failure fails the tick, a mirror failure
is logged and survived.

Auth is Firebase email/password over identitytoolkit REST, with mandatory
refresh-token rotation. No service-account key anywhere: it would bypass
the security rules and is trivially extractable from an APK.

Verified live against a real database, not only mocks: sign-in and refresh,
key escaping round-tripping byte-for-byte, an unchanged tick writing
nothing, two devices converging, a revoked session throwing rather than
returning null, and a log written by the Python client being read by the
Dart one.

No app is migrated by this commit; every app still runs on GitHub.

Dart: 194 tests. Python: 203 tests at 100% branch coverage. No lint
suppressions added.
2026-08-03 18:14:25 +02:00

250 lines
8.9 KiB
Python

"""Minimal GitHub Contents API client used as dumb file storage.
GitHub is used purely as file storage via the REST Contents API, not a git
clone -- there is no working tree and no git-level merge; the only merge is
the domain-level one in :mod:`crdt_sync._log`. Extracted from diet_guard's
``_sync_github.py`` (itself ported in spirit from ``todo``'s sync
transport), generalized to be domain-agnostic.
"""
from __future__ import annotations
import base64
import requests
from crdt_sync._remote import RemoteNotFoundError, RemoteSyncError
_API_BASE = "https://api.github.com"
_HTTP_NOT_FOUND = 404
_DEFAULT_TIMEOUT_SECONDS = 15
class GitHubSyncError(RemoteSyncError):
"""Raised for a GitHub API failure the caller must not silently ignore."""
class RepoNotFoundError(GitHubSyncError, RemoteNotFoundError):
"""Raised when the configured repo itself is unreachable.
Distinguished from a path-404 (nothing pushed to that path yet, which is
benign -- it just means no other device has synced before) so the caller
can tell "the repo name is wrong or the token isn't scoped to it" apart
from "no other device has synced yet".
Derives from both so existing ``except GitHubSyncError`` handlers still
cover it *and* backend-neutral callers can catch "the remote itself is
missing" without naming GitHub.
"""
class GitHubSyncClient:
"""Thin wrapper around the subset of the Contents API sync needs."""
def __init__(
self,
owner: str,
repo: str,
token: str,
*,
timeout_seconds: float = _DEFAULT_TIMEOUT_SECONDS,
) -> None:
"""Create a client scoped to one repo, authenticated with ``token``.
Args:
owner: The repo owner/org (e.g. ``"kuhyx"``).
repo: The repo name (e.g. ``"my-app-sync"``).
token: A GitHub PAT with contents read/write on that repo.
timeout_seconds: Per-request network timeout.
"""
self._owner = owner
self._repo = repo
self._timeout_seconds = timeout_seconds
self._headers = {
"Authorization": f"Bearer {token}",
"Accept": "application/vnd.github+json",
}
def _contents_url(self, path: str) -> str:
return f"{_API_BASE}/repos/{self._owner}/{self._repo}/contents/{path}"
def _get(self, path: str) -> requests.Response:
try:
return requests.get(
self._contents_url(path),
headers=self._headers,
timeout=self._timeout_seconds,
)
except requests.RequestException as exc:
msg = f"network error reading {path}"
raise GitHubSyncError(msg) from exc
def _repo_exists(self) -> bool:
try:
response = requests.get(
f"{_API_BASE}/repos/{self._owner}/{self._repo}",
headers=self._headers,
timeout=self._timeout_seconds,
)
except requests.RequestException:
return False
return response.ok
def _raise_for_missing_path(self, path: str) -> None:
"""Raise :class:`RepoNotFoundError` only if the repo is unreachable.
A 404 on a path within a reachable repo just means nothing has been
pushed there yet, which is not an error worth raising on.
"""
if not self._repo_exists():
msg = (
f"{self._owner}/{self._repo} not found, private without "
f"access, or the token lacks contents permission "
f"(while reading {path})"
)
raise RepoNotFoundError(msg)
def get_file_text(self, path: str) -> str | None:
"""Return the decoded text content at ``path``, or None if unused.
Args:
path: A repo-relative file path, e.g. ``"devices/pc/log.json"``.
Returns:
The file's text content, or None if nothing has been pushed
there yet (but the repo itself is reachable).
Raises:
RepoNotFoundError: If the repo itself is unreachable.
GitHubSyncError: For any other non-2xx response or network error.
"""
response = self._get(path)
if response.status_code == _HTTP_NOT_FOUND:
self._raise_for_missing_path(path)
return None
if not response.ok:
msg = f"GET {path} failed: {response.status_code}"
raise GitHubSyncError(msg)
data = response.json()
content = data.get("content", "") if isinstance(data, dict) else ""
return base64.b64decode(content).decode("utf-8")
def _existing_sha(self, path: str) -> str | None:
response = self._get(path)
if response.status_code == _HTTP_NOT_FOUND:
self._raise_for_missing_path(path)
return None
if not response.ok:
msg = f"GET {path} (for sha) failed: {response.status_code}"
raise GitHubSyncError(msg)
data = response.json()
sha = data.get("sha") if isinstance(data, dict) else None
return sha if isinstance(sha, str) else None
def put_file_text(self, path: str, text: str, *, message: str) -> None:
"""Create or update the file at ``path`` with ``text``.
Args:
path: A repo-relative file path.
text: The full new content (this device's complete merged log).
message: The commit message for this push.
Raises:
GitHubSyncError: On any non-2xx response or network error.
"""
sha = self._existing_sha(path)
payload: dict[str, object] = {
"message": message,
"content": base64.b64encode(text.encode("utf-8")).decode("ascii"),
}
if sha is not None:
payload["sha"] = sha
try:
response = requests.put(
self._contents_url(path),
headers=self._headers,
json=payload,
timeout=self._timeout_seconds,
)
except requests.RequestException as exc:
msg = f"network error pushing {path}"
raise GitHubSyncError(msg) from exc
if not response.ok:
msg = f"PUT {path} failed: {response.status_code}"
raise GitHubSyncError(msg)
def list_directory(self, path: str) -> list[str]:
"""Return the entry names directly under ``path`` (empty if unused).
Args:
path: A repo-relative directory path, e.g. ``"devices"``.
Raises:
RepoNotFoundError: If the repo itself is unreachable.
GitHubSyncError: For any other non-2xx response or network error.
"""
response = self._get(path)
if response.status_code == _HTTP_NOT_FOUND:
self._raise_for_missing_path(path)
return []
if not response.ok:
msg = f"GET {path} (list) failed: {response.status_code}"
raise GitHubSyncError(msg)
data = response.json()
if not isinstance(data, list):
return []
return [
item["name"]
for item in data
if isinstance(item, dict) and isinstance(item.get("name"), str)
]
def can_access_repo(self) -> bool:
"""Return whether the token can read this repo (a connection test).
Hits the bare repo endpoint, so it succeeds even before any file has
been pushed. Never raises -- a network failure or missing repo is
reported as ``False``.
The GitHub-era name for :meth:`can_access_remote`; kept because app
settings screens call it. Both do the same probe.
"""
return self._repo_exists()
def can_access_remote(self) -> bool:
"""Return whether the token can reach the remote (a connection test).
The :class:`crdt_sync.RemoteStore` spelling of
:meth:`can_access_repo`.
"""
return self._repo_exists()
def delete_file(self, path: str, *, message: str = "crdt_sync: delete") -> None:
"""Delete the file at ``path``, resolving its current sha itself.
A no-op if ``path`` does not exist, so a repeated cleanup is safe.
Args:
path: A repo-relative file path.
message: The commit message for the deletion.
Raises:
GitHubSyncError: On any non-2xx response or network error.
"""
sha = self._existing_sha(path)
if sha is None:
return
try:
response = requests.delete(
self._contents_url(path),
headers=self._headers,
json={"message": message, "sha": sha},
timeout=self._timeout_seconds,
)
except requests.RequestException as exc:
msg = f"network error deleting {path}"
raise GitHubSyncError(msg) from exc
if not response.ok:
msg = f"DELETE {path} failed: {response.status_code}"
raise GitHubSyncError(msg)