mirror of
https://github.com/kuhyx/utils.git
synced 2026-11-17 21:25:05 +01:00
rtdb_relay 0.1.0: sealed RTDB request relay
The tutor's generic relay transport (layout rules, sealer, REST client, SSE, mirror, unlock) moved here unchanged in format, plus a generic RequestServer (kind allowlist, sealed-only requests, lease, visible handler failures) and a presence heartbeat for a second consumer (todoImplementer). The tutor's contract vectors pin the byte format. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
0d9974c194
commit
6d72a7c804
90
rtdb_relay/README.md
Normal file
90
rtdb_relay/README.md
Normal file
@ -0,0 +1,90 @@
|
||||
# rtdb_relay
|
||||
|
||||
A sealed request relay over Firebase RTDB: a phone app writes a request, a
|
||||
worker on the PC answers it. No VPN, no open port: both sides talk only to
|
||||
RTDB over HTTPS. Access needs the one Google account the RTDB rules allow,
|
||||
and every value is sealed end to end with the shared crdt-sync passphrase.
|
||||
|
||||
Extracted from `kuhyx/tutor` (`tutor/relay/`), whose phone app has used this
|
||||
wire format since 2026-10. The tutor's contract vectors
|
||||
(`rtdb_relay/tests/data/e2e_vectors.json`) pin the byte format; the tests
|
||||
check that the package seals and opens exactly those bytes.
|
||||
|
||||
Python 3.14+ (tested on 3.15.2). Install from a release tag:
|
||||
|
||||
```toml
|
||||
"rtdb-relay @ git+https://github.com/kuhyx/utils@rtdb-relay-v0.1.0#subdirectory=rtdb_relay"
|
||||
```
|
||||
|
||||
## Wire format (under the app's root, e.g. `todoimpl/v1-e2e`)
|
||||
|
||||
| Path | Who writes | Value |
|
||||
|---|---|---|
|
||||
| `requests/{rid}/req` | client | sealed `{kind, dev, created, sent_at, body}`; `created` clear |
|
||||
| `requests/{rid}/claim` | worker | clear `{by, lease_until, attempt, phase, at}` (ms) |
|
||||
| `requests/{rid}/result` | worker | sealed `{ok, kind, ...data}` or `{ok: false, kind, code, error, retry}`; `at` clear |
|
||||
| `presence/workers/{worker}` | worker | sealed `{host, version}`; `beat` clear (server time) |
|
||||
| `<root minus -e2e>/e2e` | first unlock | the Argon2id key header (crdt_sync) |
|
||||
|
||||
`rid` is a ULID (`[0-9A-HJKMNP-TV-Z]{26}`); any other key is never used in
|
||||
a path. A sealed unit is `{"_e2e": <XChaCha20-Poly1305 blob>, <clear
|
||||
siblings>}`, the blob being crdt_sync's `E2eCipher` over compact JSON with the
|
||||
unit's full path (`<root>/<path>`) as AAD, so a blob moved elsewhere does not
|
||||
open.
|
||||
|
||||
## Python API (the PC worker)
|
||||
|
||||
```python
|
||||
from crdt_sync import E2eCipher, E2eLayout
|
||||
from rtdb_relay.layout import Layout, Rule, E2eLayoutError, parts_of
|
||||
from rtdb_relay.seal import Sealer, SEALED
|
||||
from rtdb_relay.rtdb import Rtdb, RtdbError, ConflictError, SERVER_TIME
|
||||
from rtdb_relay.unlock import unlock
|
||||
from rtdb_relay.server import RequestServer, RequestError
|
||||
from rtdb_relay import presence, sse, mirror
|
||||
```
|
||||
|
||||
- `Layout(root: str, clear: tuple[str, ...], units: dict[str, frozenset[str]])`:
|
||||
the tree rules. `*` matches one segment; a `clear` pattern covers its
|
||||
subtree; a unit maps to the clear fields it keeps. A write to a path with
|
||||
no rule raises `E2eLayoutError` (fail closed). A relay app needs at least
|
||||
`clear=("requests/*/claim",)` and units `requests/*/req: {"created"}`,
|
||||
`requests/*/result: {"at"}`, `presence/workers/*: {"beat"}`.
|
||||
`layout.classify(parts) -> Rule`, `layout.unit_path(path) -> str | None`.
|
||||
- `Rtdb(database_url: str, token: Callable[[], Awaitable[str]], client:
|
||||
httpx.AsyncClient | None = None, *, root: str, sealer: Sealer | None = None)`:
|
||||
`get(path, params=None)`, `get_etag(path)`, `put(path, value)`,
|
||||
`put_etag(path, value, if_match)`, `patch(path, dict)`, `delete(path,
|
||||
if_match=None)`, `open_stream(path, params=None)`, `at(root) -> Rtdb` (raw,
|
||||
unsealed), `aclose()`. Errors never contain the URL or the token.
|
||||
- `await unlock(raw_rtdb, passphrase: str, layout: E2eLayout, params:
|
||||
Argon2Params | None = None) -> E2eKey`: makes the header on first use;
|
||||
raises `E2eLockedError` on a missing or wrong passphrase. Then
|
||||
`Sealer(E2eCipher(key.master), layout)` and `Rtdb(..., root=layout.root,
|
||||
sealer=sealer)`.
|
||||
- `RequestServer(rtdb: Rtdb, worker: str, handlers: Mapping[str, Handler],
|
||||
now: Callable[[], float] = time.time)` with `Handler = Callable[[dict[str,
|
||||
Any]], Awaitable[dict[str, Any]]]` (request `body` in, result data out).
|
||||
`await server.run()` watches `requests` until cancelled; `await
|
||||
server.answer(rid, raw_node)` serves one. A handler refuses with
|
||||
`raise RequestError(code, message, retry=False)`. Guards: unknown kind ->
|
||||
`unknown_kind`; unsealed or undecryptable `req` -> `unsealed`; a handler
|
||||
bug -> `internal` (logged); long handlers renew the lease every 60 s.
|
||||
- `await presence.beat_forever(rtdb, worker, version)`: a beat every 30 s.
|
||||
- `sse.listen(rtdb, path) -> AsyncGenerator[sse.Event]` (reconnects with
|
||||
backoff, opens sealed units); `mirror.apply(tree, event) -> dict` keeps a
|
||||
local copy of a streamed node. To publish a sealed snapshot node, `await
|
||||
rtdb.put("<unit path>", value)`.
|
||||
|
||||
## Dart API (the Flutter client)
|
||||
|
||||
Not extracted yet: the tutor app's `lib/sync/` and `lib/relay/` still hold
|
||||
the client half. Until it moves here, a new client follows the wire format
|
||||
above (`requests/{rid}/req` sealed with crdt_sync's Dart `E2eCipher`, then
|
||||
listen on `requests/{rid}/result`).
|
||||
|
||||
## Develop
|
||||
|
||||
`uv venv --python 3.15 .venv && uv pip install -e . pytest==9.2.0
|
||||
pytest-cov==7.1.0 ruff==0.17.4 mypy==2.4.1`, then `scripts/run_subproject_tests.sh rtdb_relay`
|
||||
from the repo root (100 % branch coverage), `ruff check .` and `mypy rtdb_relay`.
|
||||
176
rtdb_relay/pyproject.toml
Normal file
176
rtdb_relay/pyproject.toml
Normal file
@ -0,0 +1,176 @@
|
||||
[project]
|
||||
name = "rtdb-relay"
|
||||
version = "0.1.0"
|
||||
description = "Sealed request relay over Firebase RTDB: PC worker (Python) + Flutter client (Dart)"
|
||||
requires-python = ">=3.14"
|
||||
dependencies = [
|
||||
"crdt-sync @ git+https://github.com/kuhyx/utils@crdt-sync-v0.11.1#subdirectory=crdt-sync",
|
||||
"httpx==0.28.1",
|
||||
]
|
||||
|
||||
# Ship the library and its tests (consumers run the contract vectors).
|
||||
[tool.setuptools]
|
||||
packages = ["rtdb_relay", "rtdb_relay.tests"]
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py314"
|
||||
include = ["*.py", "**/*.py"]
|
||||
exclude = [".git", ".venv", "__pycache__", "build", "dist", ".eggs"]
|
||||
|
||||
[tool.ruff.lint]
|
||||
select = ["ALL"]
|
||||
ignore = [
|
||||
"D203",
|
||||
"D213",
|
||||
"COM812",
|
||||
"ISC001",
|
||||
"S603",
|
||||
]
|
||||
fixable = ["ALL"]
|
||||
unfixable = []
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"**/tests/*.py" = ["ANN", "ARG", "CPY001", "D", "EM101", "PLC0415", "PLR2004", "S101", "SLF001", "TC002"]
|
||||
"**/_fake_*.py" = ["ANN401", "ARG002"]
|
||||
"**/test_*.py" = ["ANN", "ARG", "CPY001", "D", "EM101", "PLC0415", "PLR2004", "S101", "SLF001", "TC002"]
|
||||
|
||||
|
||||
[tool.ruff.lint.pydocstyle]
|
||||
convention = "google"
|
||||
|
||||
[tool.ruff.lint.isort]
|
||||
force-single-line = false
|
||||
force-sort-within-sections = true
|
||||
known-first-party = ["rtdb_relay"]
|
||||
|
||||
[tool.ruff.lint.flake8-quotes]
|
||||
docstring-quotes = "double"
|
||||
inline-quotes = "double"
|
||||
|
||||
[tool.ruff.lint.flake8-tidy-imports]
|
||||
ban-relative-imports = "all"
|
||||
|
||||
[tool.ruff.format]
|
||||
quote-style = "double"
|
||||
indent-style = "space"
|
||||
skip-magic-trailing-comma = false
|
||||
line-ending = "auto"
|
||||
docstring-code-format = true
|
||||
|
||||
[tool.mypy]
|
||||
python_version = "3.14"
|
||||
strict = true
|
||||
warn_return_any = true
|
||||
warn_unused_configs = true
|
||||
disallow_untyped_defs = true
|
||||
disallow_incomplete_defs = true
|
||||
check_untyped_defs = true
|
||||
disallow_untyped_decorators = true
|
||||
no_implicit_optional = true
|
||||
warn_redundant_casts = true
|
||||
warn_unused_ignores = true
|
||||
warn_no_return = true
|
||||
warn_unreachable = true
|
||||
# crdt_sync ships no py.typed: its names would all be "unfollowed".
|
||||
disallow_any_unimported = false
|
||||
disallow_any_explicit = false
|
||||
disallow_any_generics = true
|
||||
disallow_subclassing_any = true
|
||||
strict_equality = true
|
||||
extra_checks = true
|
||||
ignore_missing_imports = true
|
||||
show_error_codes = true
|
||||
color_output = true
|
||||
exclude = [".venv/"]
|
||||
|
||||
[tool.pylint.main]
|
||||
# The repo root on the path, so pylint resolves the package's own imports the
|
||||
# same way pytest does instead of reporting import-error for them.
|
||||
init-hook = "import sys; sys.path.insert(0, 'rtdb_relay')"
|
||||
analyse-fallback-blocks = true
|
||||
persistent = true
|
||||
jobs = 0
|
||||
py-version = "3.14"
|
||||
ignore = [".venv", "__pycache__"]
|
||||
# Test suites intentionally use patterns (protected-access, fixture-arg
|
||||
# shadowing) that don't apply to source code; ruff's per-file-ignores carry
|
||||
# the equivalent exemptions, so tests stay linted -- by ruff. This is the same
|
||||
# source-only pylint scope wake-alarm / diet-guard / testsAndMisc use. It has
|
||||
# to be `ignore-paths`, not `ignore = ["tests"]`: `ignore` only matches the
|
||||
# basename of what pylint is *given*, and pre-commit hands it explicit file
|
||||
# paths, so a directory name in `ignore` never fires under the hook.
|
||||
ignore-paths = ["^(.*/)?tests/.*$"]
|
||||
ignore-patterns = [".*\\.pyi$"]
|
||||
|
||||
[tool.pylint.basic]
|
||||
# snake_case, plus the ``do_<METHOD>`` names http.server dispatches on: the
|
||||
# stdlib looks handlers up as ``"do_" + command``, so ``do_GET`` is the only
|
||||
# spelling that works and cannot be renamed to satisfy the style.
|
||||
method-rgx = "([^\\W\\dA-Z][^\\WA-Z]{2,}|_[^\\WA-Z]*|__[^\\W\\dA-Z][^\\WA-Z]+__|do_[A-Z]+|of)$"
|
||||
|
||||
[tool.pylint.messages_control]
|
||||
enable = "all"
|
||||
disable = []
|
||||
|
||||
[tool.pylint.design]
|
||||
min-public-methods = 0
|
||||
max-module-lines = 1000
|
||||
max-attributes = 10
|
||||
|
||||
[tool.pylint.typecheck]
|
||||
generated-members = [
|
||||
".*\\.assert_called_once_with",
|
||||
".*\\.assert_called_once",
|
||||
".*\\.assert_called",
|
||||
".*\\.assert_not_called",
|
||||
".*\\.assert_any_call",
|
||||
".*\\.call_args",
|
||||
".*\\.call_args_list",
|
||||
".*\\.call_count",
|
||||
]
|
||||
|
||||
[tool.bandit]
|
||||
exclude_dirs = ["tests", ".venv"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["rtdb_relay/tests"]
|
||||
python_files = ["test_*.py", "*_test.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
addopts = [
|
||||
"-v",
|
||||
"--strict-markers",
|
||||
"--strict-config",
|
||||
"-ra",
|
||||
"--cov=rtdb_relay",
|
||||
"--cov-branch",
|
||||
"--cov-report=term-missing",
|
||||
"--cov-report=lcov",
|
||||
]
|
||||
filterwarnings = [
|
||||
"error",
|
||||
"ignore::DeprecationWarning",
|
||||
"default::pytest.PytestUnraisableExceptionWarning",
|
||||
]
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["rtdb_relay"]
|
||||
branch = true
|
||||
omit = [
|
||||
"*/__pycache__/*",
|
||||
"*/tests/*",
|
||||
"*/.venv/*",
|
||||
]
|
||||
|
||||
[tool.coverage.report]
|
||||
fail_under = 100
|
||||
show_missing = true
|
||||
skip_covered = false
|
||||
exclude_lines = [
|
||||
"pragma: no cover",
|
||||
"raise NotImplementedError",
|
||||
"raise AssertionError",
|
||||
"if TYPE_CHECKING:",
|
||||
'if __name__ == "__main__":',
|
||||
]
|
||||
partial_branches = ["pragma: no branch"]
|
||||
5
rtdb_relay/rtdb_relay/__init__.py
Normal file
5
rtdb_relay/rtdb_relay/__init__.py
Normal file
@ -0,0 +1,5 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""A sealed request relay over Firebase RTDB (PC worker side).
|
||||
|
||||
See ``README.md`` for the API and the wire format.
|
||||
"""
|
||||
81
rtdb_relay/rtdb_relay/layout.py
Normal file
81
rtdb_relay/rtdb_relay/layout.py
Normal file
@ -0,0 +1,81 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Which RTDB paths of an app's tree are clear, sealed units, or structure.
|
||||
|
||||
The tree keeps its shape: keys stay clear so ``orderBy="$key"``, PATCH and
|
||||
SSE paths work unchanged. Every *unit* (a node one writer writes whole,
|
||||
below which nothing is written) is stored sealed; a unit may keep a few
|
||||
clear siblings (server timestamps RTDB must resolve). ``clear`` prefixes
|
||||
hold ids, counters and lease times only. Any other path is refused by the
|
||||
sealer (fail closed), so a new node cannot go out in plaintext by accident.
|
||||
|
||||
Patterns are ``/``-separated; ``*`` matches one segment.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
class E2eLayoutError(ValueError):
|
||||
"""A write the layout has no rule for, or one below a sealed unit."""
|
||||
|
||||
|
||||
def parts_of(path: str) -> list[str]:
|
||||
"""``path``'s segments (no empty ones)."""
|
||||
return [p for p in path.split("/") if p]
|
||||
|
||||
|
||||
def _matches(pattern: list[str], parts: list[str]) -> bool:
|
||||
return len(pattern) == len(parts) and all(
|
||||
p in {"*", q} for p, q in zip(pattern, parts, strict=True)
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Rule:
|
||||
"""Where ``parts`` falls: ``clear``, ``unit``, ``below``, ``struct``, ``unknown``.
|
||||
|
||||
``depth`` is the unit's length for ``below``; ``keep`` the clear fields
|
||||
of a unit.
|
||||
"""
|
||||
|
||||
kind: str
|
||||
depth: int = 0
|
||||
keep: frozenset[str] = frozenset()
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Layout:
|
||||
"""An app's tree rules: ``root`` (e.g. ``tutor/v1-e2e``) plus patterns.
|
||||
|
||||
``clear`` patterns cover their whole subtree; ``units`` map a unit
|
||||
pattern to the clear fields it keeps beside the sealed blob.
|
||||
"""
|
||||
|
||||
root: str
|
||||
clear: tuple[str, ...]
|
||||
units: dict[str, frozenset[str]] = field(default_factory=dict)
|
||||
|
||||
def classify(self, parts: list[str]) -> Rule:
|
||||
"""The rule for the node at ``parts`` (relative to :attr:`root`)."""
|
||||
for pattern in self.clear:
|
||||
pat = pattern.split("/")
|
||||
if len(pat) <= len(parts) and _matches(pat, parts[: len(pat)]):
|
||||
return Rule("clear")
|
||||
for pattern, keep in self.units.items():
|
||||
pat = pattern.split("/")
|
||||
if _matches(pat, parts):
|
||||
return Rule("unit", len(pat), keep)
|
||||
if len(pat) < len(parts) and _matches(pat, parts[: len(pat)]):
|
||||
return Rule("below", len(pat), keep)
|
||||
for pattern in (*self.clear, *self.units):
|
||||
pat = pattern.split("/")
|
||||
if len(parts) < len(pat) and _matches(pat[: len(parts)], parts):
|
||||
return Rule("struct")
|
||||
return Rule("unknown")
|
||||
|
||||
def unit_path(self, path: str) -> str | None:
|
||||
"""The sealed unit ``path`` lies strictly below, or ``None``."""
|
||||
parts = parts_of(path)
|
||||
rule = self.classify(parts)
|
||||
return "/".join(parts[: rule.depth]) if rule.kind == "below" else None
|
||||
101
rtdb_relay/rtdb_relay/lease.py
Normal file
101
rtdb_relay/rtdb_relay/lease.py
Normal file
@ -0,0 +1,101 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Request ids and the claim lease on ``requests/{rid}/claim``.
|
||||
|
||||
The claim node is ``{"by", "lease_until", "attempt", "phase", "at"}`` (ms),
|
||||
the format the tutor's worker writes. One :class:`Lease` per request is the
|
||||
only writer of its claim: it holds the current ETag, so renewals never race
|
||||
each other into a 412 of our own making.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
from typing import TYPE_CHECKING, Any, Final
|
||||
|
||||
from rtdb_relay.rtdb import ConflictError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
from rtdb_relay.rtdb import Rtdb
|
||||
|
||||
RID: Final = re.compile(r"[0-9A-HJKMNP-TV-Z]{26}", re.ASCII)
|
||||
LEASE_S: Final = 300.0
|
||||
RENEW_S: Final = 60.0
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def valid_rid(rid: str) -> bool:
|
||||
"""Whether ``rid`` is a ULID (the only keys a path is built from)."""
|
||||
return bool(RID.fullmatch(rid))
|
||||
|
||||
|
||||
def ms(t: float) -> int:
|
||||
"""Unix seconds -> unix ms."""
|
||||
return int(t * 1000)
|
||||
|
||||
|
||||
def claimable(claim: object, now: float, own: str = "") -> bool:
|
||||
"""Null, a lease that ran out, or ``own`` (a dead process of this worker).
|
||||
|
||||
The caller has checked there is no result.
|
||||
"""
|
||||
if not isinstance(claim, dict):
|
||||
return True
|
||||
lease = claim.get("lease_until")
|
||||
mine = bool(own) and claim.get("by") == own
|
||||
return mine or not isinstance(lease, (int, float)) or lease < ms(now)
|
||||
|
||||
|
||||
class Lease:
|
||||
"""A claim this worker holds: attempt number, phase and current ETag."""
|
||||
|
||||
def __init__(
|
||||
self, rtdb: Rtdb, rid: str, worker: str, now: Callable[[], float]
|
||||
) -> None:
|
||||
"""Not yet taken; see :meth:`take`."""
|
||||
self._rtdb, self.rid, self._worker, self._now = rtdb, rid, worker, now
|
||||
self.attempt = 0
|
||||
self.phase = "claimed"
|
||||
self._etag = ""
|
||||
|
||||
@property
|
||||
def path(self) -> str:
|
||||
"""``requests/{rid}/claim``."""
|
||||
return f"requests/{self.rid}/claim"
|
||||
|
||||
def _node(self) -> dict[str, Any]:
|
||||
now = self._now()
|
||||
return {
|
||||
"by": self._worker,
|
||||
"lease_until": ms(now + LEASE_S),
|
||||
"attempt": self.attempt,
|
||||
"phase": self.phase,
|
||||
"at": ms(now),
|
||||
}
|
||||
|
||||
async def take(self) -> bool:
|
||||
"""Claim the request (False: someone else holds it, or lost the race)."""
|
||||
current, etag = await self._rtdb.get_etag(self.path)
|
||||
if not claimable(current, self._now(), self._worker):
|
||||
return False
|
||||
before = current.get("attempt", 0) if isinstance(current, dict) else 0
|
||||
self.attempt = (before if isinstance(before, int) else 0) + 1
|
||||
try:
|
||||
_, self._etag = await self._rtdb.put_etag(self.path, self._node(), etag)
|
||||
except ConflictError:
|
||||
_logger.warning("another worker claimed %s first", self.rid)
|
||||
return False
|
||||
return True
|
||||
|
||||
async def renew(self, phase: str | None = None) -> None:
|
||||
"""Extend the lease (and move to ``phase``); a lost race is only logged."""
|
||||
if phase is not None:
|
||||
self.phase = phase
|
||||
try:
|
||||
_, self._etag = await self._rtdb.put_etag(
|
||||
self.path, self._node(), self._etag
|
||||
)
|
||||
except ConflictError:
|
||||
_logger.warning("claim on %s lost to another writer", self.rid)
|
||||
46
rtdb_relay/rtdb_relay/mirror.py
Normal file
46
rtdb_relay/rtdb_relay/mirror.py
Normal file
@ -0,0 +1,46 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""A local copy of a streamed RTDB node, kept current from put/patch events."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from rtdb_relay.sse import Event
|
||||
|
||||
|
||||
def _parts(path: str) -> list[str]:
|
||||
return [p for p in path.split("/") if p]
|
||||
|
||||
|
||||
def _set(tree: dict[str, Any], parts: list[str], value: object) -> dict[str, Any]:
|
||||
"""``tree`` with ``value`` at ``parts`` (``None`` deletes, empty parents go)."""
|
||||
head, rest = parts[0], parts[1:]
|
||||
if rest:
|
||||
child = tree.get(head)
|
||||
value = _set(child if isinstance(child, dict) else {}, rest, value)
|
||||
if not value:
|
||||
value = None
|
||||
if value is None:
|
||||
tree.pop(head, None)
|
||||
else:
|
||||
tree[head] = value
|
||||
return tree
|
||||
|
||||
|
||||
def apply(tree: dict[str, Any], event: Event) -> dict[str, Any]:
|
||||
"""The mirrored node after ``event`` (a new dict on a root ``put``)."""
|
||||
parts = _parts(event.path)
|
||||
if event.kind == "put":
|
||||
if not parts:
|
||||
return event.data if isinstance(event.data, dict) else {}
|
||||
return _set(tree, parts, event.data)
|
||||
for key, value in (event.data or {}).items():
|
||||
tree = _set(tree, [*parts, *_parts(key)], value)
|
||||
return tree
|
||||
|
||||
|
||||
def first_key(path: str) -> str | None:
|
||||
"""The top-level key an event path touches (``None`` for the root)."""
|
||||
parts = _parts(path)
|
||||
return parts[0] if parts else None
|
||||
39
rtdb_relay/rtdb_relay/presence.py
Normal file
39
rtdb_relay/rtdb_relay/presence.py
Normal file
@ -0,0 +1,39 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""The worker's heartbeat at ``presence/workers/{worker}``.
|
||||
|
||||
Clients judge the worker alive from when beats stop arriving (``beat`` is an
|
||||
RTDB server timestamp, kept clear beside the sealed unit), never by
|
||||
comparing it to their own clock.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import socket
|
||||
from typing import TYPE_CHECKING, Final
|
||||
|
||||
from rtdb_relay.rtdb import SERVER_TIME
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Awaitable, Callable
|
||||
|
||||
from rtdb_relay.rtdb import Rtdb
|
||||
|
||||
BEAT_S: Final = 30.0
|
||||
|
||||
|
||||
def node(version: str) -> dict[str, object]:
|
||||
"""One beat: server time, the host and the worker's build."""
|
||||
return {"beat": SERVER_TIME, "host": socket.gethostname(), "version": version}
|
||||
|
||||
|
||||
async def beat_forever(
|
||||
rtdb: Rtdb,
|
||||
worker: str,
|
||||
version: str,
|
||||
sleep: Callable[[float], Awaitable[None]] = asyncio.sleep,
|
||||
) -> None:
|
||||
"""Write a beat every :data:`BEAT_S` until cancelled."""
|
||||
while True:
|
||||
await rtdb.put(f"presence/workers/{worker}", node(version))
|
||||
await sleep(BEAT_S)
|
||||
197
rtdb_relay/rtdb_relay/rtdb.py
Normal file
197
rtdb_relay/rtdb_relay/rtdb.py
Normal file
@ -0,0 +1,197 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""A minimal async RTDB REST client: GET/PUT/PATCH/DELETE, ETags, server time.
|
||||
|
||||
Under the app's root every value goes through a ``seal.Sealer`` (sealed on
|
||||
write, opened on read) once ``sealer`` is set; ``at()`` gives the raw
|
||||
client the e2e header and one-time copies use.
|
||||
|
||||
Every call authenticates with ``?auth=<idToken>`` (RTDB REST has no header
|
||||
form), so nothing here ever formats a URL or an ``httpx`` error into a
|
||||
message or a log line: errors name the method, the path and the status only.
|
||||
httpx's own request log (INFO, full URL) is silenced for the same reason.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Any, Final
|
||||
from urllib.parse import quote
|
||||
|
||||
import httpx
|
||||
|
||||
from rtdb_relay.layout import parts_of
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator, Awaitable, Callable
|
||||
|
||||
from rtdb_relay.seal import Sealer
|
||||
|
||||
SERVER_TIME: Final = {".sv": "timestamp"}
|
||||
NULL_ETAG: Final = "null_etag"
|
||||
_PRECONDITION_FAILED: Final = 412
|
||||
_TIMEOUT: Final = httpx.Timeout(30.0, connect=15.0)
|
||||
# The SSE read timeout: RTDB sends keep-alive every 30 s.
|
||||
_STREAM_TIMEOUT: Final = httpx.Timeout(30.0, connect=15.0, read=75.0)
|
||||
|
||||
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||
logging.getLogger("httpcore").setLevel(logging.WARNING)
|
||||
|
||||
|
||||
class RtdbError(RuntimeError):
|
||||
"""A request failed; the message never contains the URL or the token."""
|
||||
|
||||
|
||||
class ConflictError(RtdbError):
|
||||
"""A conditional write lost the race (HTTP 412): the current value and ETag."""
|
||||
|
||||
def __init__(self, path: str, value: object, etag: str) -> None:
|
||||
"""Keep what the server holds now."""
|
||||
super().__init__(f"conditional write lost on {path}")
|
||||
self.value = value
|
||||
self.etag = etag
|
||||
|
||||
|
||||
class Rtdb:
|
||||
"""REST calls under ``root``; ``token`` yields a fresh id token."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
database_url: str,
|
||||
token: Callable[[], Awaitable[str]],
|
||||
client: httpx.AsyncClient | None = None,
|
||||
*,
|
||||
root: str,
|
||||
sealer: Sealer | None = None,
|
||||
) -> None:
|
||||
"""Bind the database; ``client`` is injectable (tests use a fake)."""
|
||||
self._base = database_url.rstrip("/")
|
||||
self._token = token
|
||||
self.client = client or httpx.AsyncClient(timeout=_TIMEOUT)
|
||||
self.root = root
|
||||
self.sealer = sealer
|
||||
|
||||
def at(self, root: str) -> Rtdb:
|
||||
"""A raw (unsealed) client on ``root`` sharing this one's connection."""
|
||||
return type(self)(self._base, self._token, self.client, root=root)
|
||||
|
||||
def url(self, path: str) -> str:
|
||||
"""The REST URL of ``path`` (relative to the root), no token."""
|
||||
parts = [*parts_of(self.root), *parts_of(path)]
|
||||
return f"{self._base}/{'/'.join(quote(p, safe='') for p in parts)}.json"
|
||||
|
||||
def _seal(self, path: str, value: object) -> object:
|
||||
return value if self.sealer is None else self.sealer.seal(path, value)
|
||||
|
||||
def opened(self, path: str, value: object) -> object:
|
||||
"""``value`` read at ``path`` with its sealed units opened."""
|
||||
return value if self.sealer is None else self.sealer.open(path, value)
|
||||
|
||||
async def params(self, extra: dict[str, str] | None = None) -> dict[str, str]:
|
||||
"""Query parameters: the auth token plus ``extra``."""
|
||||
return {"auth": await self._token(), **(extra or {})}
|
||||
|
||||
async def _call(
|
||||
self,
|
||||
method: str,
|
||||
path: str,
|
||||
*,
|
||||
value: object = None,
|
||||
headers: dict[str, str] | None = None,
|
||||
params: dict[str, str] | None = None,
|
||||
) -> httpx.Response:
|
||||
kwargs: dict[str, Any] = {"params": await self.params(params)}
|
||||
if method in {"PUT", "PATCH"}:
|
||||
kwargs["content"] = json.dumps(value).encode() # json=None sends no body
|
||||
try:
|
||||
response = await self.client.request(
|
||||
method, self.url(path), headers=headers or {}, **kwargs
|
||||
)
|
||||
except httpx.HTTPError as err:
|
||||
msg = f"{method} {path}: {type(err).__name__}"
|
||||
raise RtdbError(msg) from None
|
||||
if response.status_code == _PRECONDITION_FAILED:
|
||||
current = self.opened(path, response.json())
|
||||
raise ConflictError(path, current, response.headers.get("etag", ""))
|
||||
if response.is_error:
|
||||
msg = f"{method} {path}: HTTP {response.status_code}"
|
||||
raise RtdbError(msg)
|
||||
return response
|
||||
|
||||
async def get(self, path: str, params: dict[str, str] | None = None) -> object:
|
||||
"""The value at ``path`` (``None`` when absent).
|
||||
|
||||
Below a sealed unit the whole unit is read and opened, then walked.
|
||||
"""
|
||||
unit = None if self.sealer is None else self.sealer.unit_path(path)
|
||||
if unit is not None:
|
||||
node: Any = await self.get(unit)
|
||||
for part in parts_of(path)[len(parts_of(unit)) :]:
|
||||
node = node.get(part) if isinstance(node, dict) else None
|
||||
return node
|
||||
value: object = (await self._call("GET", path, params=params)).json()
|
||||
return self.opened(path, value)
|
||||
|
||||
async def get_etag(self, path: str) -> tuple[Any, str]:
|
||||
"""The value at ``path`` and its ETag (``null_etag`` when absent)."""
|
||||
response = await self._call("GET", path, headers={"X-Firebase-ETag": "true"})
|
||||
value = self.opened(path, response.json())
|
||||
return value, response.headers.get("etag", NULL_ETAG)
|
||||
|
||||
async def put(self, path: str, value: object) -> object:
|
||||
"""Write ``value``; returns it as stored (server timestamps resolved)."""
|
||||
sealed = self._seal(path, value)
|
||||
stored: object = (await self._call("PUT", path, value=sealed)).json()
|
||||
return self.opened(path, stored)
|
||||
|
||||
async def put_etag(
|
||||
self, path: str, value: object, if_match: str
|
||||
) -> tuple[Any, str]:
|
||||
"""A conditional PUT that also returns the new ETag."""
|
||||
headers = {"if-match": if_match, "X-Firebase-ETag": "true"}
|
||||
sealed = self._seal(path, value)
|
||||
response = await self._call("PUT", path, value=sealed, headers=headers)
|
||||
return self.opened(path, response.json()), response.headers.get("etag", "")
|
||||
|
||||
async def patch(self, path: str, value: dict[str, Any]) -> None:
|
||||
"""Merge ``value``'s children into ``path`` (``None`` children delete)."""
|
||||
if self.sealer is not None:
|
||||
value = self.sealer.seal_patch(path, value)
|
||||
await self._call("PATCH", path, value=value)
|
||||
|
||||
async def delete(self, path: str, if_match: str | None = None) -> None:
|
||||
"""Remove ``path``; with ``if_match`` only if unchanged."""
|
||||
headers = {"if-match": if_match} if if_match else {}
|
||||
await self._call("DELETE", path, headers=headers)
|
||||
|
||||
async def open_stream(
|
||||
self, path: str, params: dict[str, str] | None = None
|
||||
) -> AsyncIterator[str]:
|
||||
"""The raw SSE lines of one connection to ``path`` (see ``sse``)."""
|
||||
request = self.client.build_request(
|
||||
"GET",
|
||||
self.url(path),
|
||||
params=await self.params(params),
|
||||
headers={"Accept": "text/event-stream"},
|
||||
timeout=_STREAM_TIMEOUT,
|
||||
)
|
||||
try:
|
||||
response = await self.client.send(request, stream=True)
|
||||
except httpx.HTTPError as err:
|
||||
msg = f"STREAM {path}: {type(err).__name__}"
|
||||
raise RtdbError(msg) from None
|
||||
try:
|
||||
if response.is_error:
|
||||
msg = f"STREAM {path}: HTTP {response.status_code}"
|
||||
raise RtdbError(msg)
|
||||
async for line in response.aiter_lines():
|
||||
yield line
|
||||
except httpx.HTTPError as err:
|
||||
msg = f"STREAM {path}: {type(err).__name__}"
|
||||
raise RtdbError(msg) from None
|
||||
finally:
|
||||
await response.aclose()
|
||||
|
||||
async def aclose(self) -> None:
|
||||
"""Close the HTTP client."""
|
||||
await self.client.aclose()
|
||||
133
rtdb_relay/rtdb_relay/seal.py
Normal file
133
rtdb_relay/rtdb_relay/seal.py
Normal file
@ -0,0 +1,133 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Sealing writes and opening reads of an app's RTDB tree.
|
||||
|
||||
A unit is stored as ``{"_e2e": <XChaCha20-Poly1305 blob>, <clear siblings>}``;
|
||||
the blob is crdt_sync's ``E2eCipher`` with the unit's full path as AAD, so
|
||||
a blob copied to another path does not open. The byte format is pinned by
|
||||
``tests/data/e2e_vectors.json`` (the tutor's contract vectors).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Any, Final
|
||||
|
||||
from crdt_sync import E2eCipher, E2eDecryptError
|
||||
|
||||
from rtdb_relay.layout import E2eLayoutError, Layout, parts_of
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from rtdb_relay.layout import Rule
|
||||
|
||||
SEALED: Final = "_e2e"
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def pruned(value: object) -> object:
|
||||
"""``value`` as RTDB would keep it: no null members, no empty objects."""
|
||||
if not isinstance(value, dict):
|
||||
return value
|
||||
out = {k: pruned(v) for k, v in value.items()}
|
||||
return {k: v for k, v in out.items() if v is not None} or None
|
||||
|
||||
|
||||
def dumps(value: object) -> str:
|
||||
"""Compact JSON, UTF-8 kept (the sealed plaintext format)."""
|
||||
return json.dumps(value, separators=(",", ":"), ensure_ascii=False)
|
||||
|
||||
|
||||
class Sealer:
|
||||
"""Seals writes and opens reads of the tree ``layout`` describes."""
|
||||
|
||||
def __init__(
|
||||
self, cipher: E2eCipher, layout: Layout, root: str | None = None
|
||||
) -> None:
|
||||
"""Use ``cipher`` for every unit; AAD prefix ``root`` (default the layout's)."""
|
||||
self._cipher = cipher
|
||||
self.layout = layout
|
||||
self.root = layout.root if root is None else root
|
||||
|
||||
def classify(self, parts: list[str]) -> Rule:
|
||||
"""The layout's rule for ``parts``."""
|
||||
return self.layout.classify(parts)
|
||||
|
||||
def unit_path(self, path: str) -> str | None:
|
||||
"""The sealed unit ``path`` lies strictly below, or ``None``."""
|
||||
return self.layout.unit_path(path)
|
||||
|
||||
def _aad(self, parts: list[str]) -> str:
|
||||
return "/".join([self.root, *parts])
|
||||
|
||||
# -- writes ---------------------------------------------------------------
|
||||
def seal(self, path: str, value: object) -> object:
|
||||
"""``value`` as it may be stored at ``path``; refuses unknown paths."""
|
||||
return self._seal(parts_of(path), value)
|
||||
|
||||
def seal_patch(self, path: str, body: dict[str, Any]) -> dict[str, Any]:
|
||||
"""A PATCH body for ``path``: each (multi-segment) key sealed on its own."""
|
||||
if self.classify(parts_of(path)).kind in {"unit", "below"}:
|
||||
msg = f"PATCH into sealed unit {path}"
|
||||
raise E2eLayoutError(msg)
|
||||
base = parts_of(path)
|
||||
return {k: self._seal([*base, *parts_of(k)], v) for k, v in body.items()}
|
||||
|
||||
def _seal(self, parts: list[str], value: object) -> object:
|
||||
rule = self.classify(parts)
|
||||
if rule.kind == "clear" or value is None:
|
||||
return value
|
||||
if rule.kind == "unit":
|
||||
return self._seal_unit(parts, value, rule.keep)
|
||||
if rule.kind == "struct" and isinstance(value, dict):
|
||||
return {k: self._seal([*parts, *parts_of(k)], v) for k, v in value.items()}
|
||||
where = "/".join(parts) or "/"
|
||||
msg = f"no e2e rule for a {type(value).__name__} at {where} ({rule.kind})"
|
||||
raise E2eLayoutError(msg)
|
||||
|
||||
def _seal_unit(
|
||||
self, parts: list[str], value: object, keep: frozenset[str]
|
||||
) -> object:
|
||||
value = pruned(value)
|
||||
if value is None:
|
||||
return None # RTDB stores no null and no empty object
|
||||
clear: dict[str, Any] = {}
|
||||
if isinstance(value, dict) and keep:
|
||||
clear = {k: v for k, v in value.items() if k in keep}
|
||||
value = {k: v for k, v in value.items() if k not in keep}
|
||||
blob = self._cipher.seal(dumps(value), path=self._aad(parts))
|
||||
return {SEALED: blob, **clear}
|
||||
|
||||
# -- reads ----------------------------------------------------------------
|
||||
def open(self, path: str, value: object) -> object:
|
||||
"""``value`` read at ``path``, every unit in it opened.
|
||||
|
||||
A unit that does not open (tampered, other key) reads as absent.
|
||||
Legacy plaintext (a unit without ``_e2e``) is returned as stored.
|
||||
"""
|
||||
return self._open(parts_of(path), value)
|
||||
|
||||
def _open(self, parts: list[str], value: object) -> object:
|
||||
rule = self.classify(parts)
|
||||
if rule.kind == "unit":
|
||||
return self._open_unit(parts, value)
|
||||
if rule.kind == "below":
|
||||
msg = f"read below sealed unit: {'/'.join(parts)}"
|
||||
raise E2eLayoutError(msg)
|
||||
if rule.kind == "struct" and isinstance(value, dict):
|
||||
out = {k: self._open([*parts, k], v) for k, v in value.items()}
|
||||
return {k: v for k, v in out.items() if v is not None}
|
||||
if rule.kind == "struct" and isinstance(value, list):
|
||||
return [self._open([*parts, str(i)], v) for i, v in enumerate(value)]
|
||||
return value
|
||||
|
||||
def _open_unit(self, parts: list[str], value: object) -> object:
|
||||
if not isinstance(value, dict) or not isinstance(value.get(SEALED), str):
|
||||
return value # legacy plaintext, accepted during a migration
|
||||
clear = {k: v for k, v in value.items() if k != SEALED}
|
||||
try:
|
||||
text = self._cipher.open(value[SEALED], path=self._aad(parts))
|
||||
data = json.loads(text)
|
||||
except E2eDecryptError, ValueError:
|
||||
_logger.warning("undecryptable node %s dropped", "/".join(parts))
|
||||
return None
|
||||
return {**data, **clear} if isinstance(data, dict) else data
|
||||
151
rtdb_relay/rtdb_relay/server.py
Normal file
151
rtdb_relay/rtdb_relay/server.py
Normal file
@ -0,0 +1,151 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Serve ``requests/{rid}`` with registered kind handlers (the PC side).
|
||||
|
||||
A client writes ``requests/{rid}/req`` = ``{"kind", "dev", "created",
|
||||
"sent_at", "body"}`` (sealed). The server claims it (:class:`Lease`), runs
|
||||
the handler for ``kind`` and writes ``requests/{rid}/result`` =
|
||||
``{"ok": true, "kind", "at", **data}`` or ``{"ok": false, "kind", "code",
|
||||
"error", "retry", "at"}``. Guards (fail closed):
|
||||
|
||||
- only kinds in ``handlers`` run; any other kind gets ``unknown_kind``;
|
||||
- an unsealed (plaintext) or undecryptable ``req`` gets ``unsealed``;
|
||||
- a non-ULID rid is never used in a path (skipped);
|
||||
- a handler bug becomes a visible ``internal`` result, never a silent drop.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import contextlib
|
||||
import logging
|
||||
import time
|
||||
from typing import TYPE_CHECKING, Any, Final
|
||||
|
||||
from rtdb_relay import mirror, sse
|
||||
from rtdb_relay.lease import RENEW_S, Lease, valid_rid
|
||||
from rtdb_relay.rtdb import SERVER_TIME
|
||||
from rtdb_relay.seal import SEALED
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Awaitable, Callable, Mapping
|
||||
|
||||
from rtdb_relay.rtdb import Rtdb
|
||||
|
||||
type Handler = Callable[[dict[str, Any]], Awaitable[dict[str, Any]]]
|
||||
_logger = logging.getLogger(__name__)
|
||||
UNKNOWN_KIND: Final = "unknown_kind"
|
||||
UNSEALED: Final = "unsealed"
|
||||
INTERNAL: Final = "internal"
|
||||
|
||||
|
||||
class RequestError(Exception):
|
||||
"""A handler's refusal: ``code`` and ``message`` reach the client."""
|
||||
|
||||
def __init__(self, code: str, message: str, *, retry: bool = False) -> None:
|
||||
"""Fail the request with ``code``; ``retry`` tells the client it may."""
|
||||
super().__init__(message)
|
||||
self.code, self.message, self.retry = code, message, retry
|
||||
|
||||
|
||||
def failure(
|
||||
kind: object, code: str, message: str, *, retry: bool = False
|
||||
) -> dict[str, Any]:
|
||||
"""A failed ``result`` node (``at`` resolved by the server)."""
|
||||
return {
|
||||
"ok": False,
|
||||
"kind": kind if isinstance(kind, str) else "",
|
||||
"code": code,
|
||||
"error": message,
|
||||
"retry": retry,
|
||||
"at": SERVER_TIME,
|
||||
}
|
||||
|
||||
|
||||
class RequestServer:
|
||||
"""Claims and answers requests under ``rtdb``'s root."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
rtdb: Rtdb,
|
||||
worker: str,
|
||||
handlers: Mapping[str, Handler],
|
||||
now: Callable[[], float] = time.time,
|
||||
) -> None:
|
||||
"""Serve with ``rtdb`` (sealed client); ``worker`` names this process."""
|
||||
self.rtdb, self.worker, self.handlers, self._now = rtdb, worker, handlers, now
|
||||
self._raw = rtdb.at(rtdb.root)
|
||||
self._busy: set[str] = set()
|
||||
self.tasks: set[asyncio.Task[None]] = set()
|
||||
|
||||
def request_of(self, rid: str, raw: object) -> dict[str, Any] | None:
|
||||
"""The opened ``req``, or ``None`` when it was not sealed or not ours."""
|
||||
if not isinstance(raw, dict) or not isinstance(raw.get(SEALED), str):
|
||||
return None
|
||||
req = self.rtdb.opened(f"requests/{rid}/req", raw)
|
||||
return req if isinstance(req, dict) and SEALED not in req else None
|
||||
|
||||
async def answer(self, rid: str, node: dict[str, Any]) -> dict[str, Any] | None:
|
||||
"""Serve one pending request node; ``None`` when not ours to answer."""
|
||||
if not valid_rid(rid) or "result" in node or "req" not in node:
|
||||
return None
|
||||
req = self.request_of(rid, node["req"])
|
||||
kind = req.get("kind") if req else None
|
||||
lease = Lease(self.rtdb, rid, self.worker, self._now)
|
||||
if not await lease.take():
|
||||
return None
|
||||
if req is None:
|
||||
result = failure(kind, UNSEALED, "request was not sealed")
|
||||
elif not isinstance(kind, str) or kind not in self.handlers:
|
||||
result = failure(kind, UNKNOWN_KIND, f"no handler for {kind!r}")
|
||||
else:
|
||||
result = await self._run(lease, kind, req)
|
||||
await self.rtdb.put(f"requests/{rid}/result", result)
|
||||
return result
|
||||
|
||||
async def _run(
|
||||
self, lease: Lease, kind: str, req: dict[str, Any]
|
||||
) -> dict[str, Any]:
|
||||
body = req.get("body")
|
||||
keeper = asyncio.create_task(self._keep(lease))
|
||||
try:
|
||||
data = await self.handlers[kind](body if isinstance(body, dict) else {})
|
||||
except RequestError as err:
|
||||
return failure(kind, err.code, err.message, retry=err.retry)
|
||||
except Exception as err:
|
||||
_logger.exception("handler %s failed", kind)
|
||||
return failure(kind, INTERNAL, type(err).__name__, retry=True)
|
||||
finally:
|
||||
keeper.cancel()
|
||||
with contextlib.suppress(asyncio.CancelledError):
|
||||
await keeper
|
||||
return {**data, "ok": True, "kind": kind, "at": SERVER_TIME}
|
||||
|
||||
async def _keep(self, lease: Lease) -> None:
|
||||
while True:
|
||||
await asyncio.sleep(RENEW_S)
|
||||
await lease.renew("running")
|
||||
|
||||
def dispatch(self, tree: dict[str, Any]) -> None:
|
||||
"""Start a task for every pending request not already in flight."""
|
||||
for rid, node in tree.items():
|
||||
if rid in self._busy or not isinstance(node, dict) or "result" in node:
|
||||
continue
|
||||
self._busy.add(rid)
|
||||
task = asyncio.create_task(self._serve(rid, node))
|
||||
self.tasks.add(task)
|
||||
task.add_done_callback(self.tasks.discard)
|
||||
|
||||
async def _serve(self, rid: str, node: dict[str, Any]) -> None:
|
||||
try:
|
||||
await self.answer(rid, node)
|
||||
except Exception:
|
||||
_logger.exception("request %s failed", rid)
|
||||
finally:
|
||||
self._busy.discard(rid)
|
||||
|
||||
async def run(self) -> None:
|
||||
"""Watch ``requests`` (raw, so sealing is checked) until cancelled."""
|
||||
tree: dict[str, Any] = {}
|
||||
async for event in sse.listen(self._raw, "requests"):
|
||||
tree = mirror.apply(tree, event)
|
||||
self.dispatch(tree)
|
||||
115
rtdb_relay/rtdb_relay/sse.py
Normal file
115
rtdb_relay/rtdb_relay/sse.py
Normal file
@ -0,0 +1,115 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""RTDB streaming: Server-Sent Events, reconnected forever.
|
||||
|
||||
Each (re)connect starts with a ``put`` of the whole node at path ``/``, so a
|
||||
consumer that mirrors ``put``/``patch`` events (``mirror.apply``) is always
|
||||
correct after a drop. ``auth_revoked`` (the id token expired) reconnects at
|
||||
once with a fresh token; network errors back off; ``cancel`` (the rules now
|
||||
refuse the path) is fatal.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from dataclasses import dataclass
|
||||
import json
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Any, Final
|
||||
|
||||
from rtdb_relay.rtdb import RtdbError
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncGenerator, AsyncIterator, Awaitable, Callable
|
||||
|
||||
from rtdb_relay.rtdb import Rtdb
|
||||
|
||||
BACKOFF_S: Final = (1.0, 2.0, 5.0, 10.0, 30.0)
|
||||
_logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Event:
|
||||
"""One ``put``/``patch`` event: where it applies and the new data."""
|
||||
|
||||
kind: str
|
||||
path: str
|
||||
data: Any
|
||||
|
||||
|
||||
class StreamCancelledError(RtdbError):
|
||||
"""RTDB cancelled the stream (rules refuse the path): not retried."""
|
||||
|
||||
|
||||
def parse(lines: list[str]) -> tuple[str, str]:
|
||||
"""``(event, data)`` of one SSE block (its lines, blank line excluded)."""
|
||||
event, data = "message", []
|
||||
for line in lines:
|
||||
field, _, value = line.partition(":")
|
||||
value = value.removeprefix(" ")
|
||||
if field == "event":
|
||||
event = value
|
||||
elif field == "data":
|
||||
data.append(value)
|
||||
return event, "\n".join(data)
|
||||
|
||||
|
||||
async def _blocks(source: AsyncIterator[str]) -> AsyncIterator[tuple[str, str]]:
|
||||
block: list[str] = []
|
||||
async for line in source:
|
||||
if line:
|
||||
block.append(line)
|
||||
elif block:
|
||||
yield parse(block)
|
||||
block = []
|
||||
|
||||
|
||||
def opened(rtdb: Rtdb, path: str, event: Event) -> Event:
|
||||
"""``event`` with its sealed units opened (``path``: the listened node).
|
||||
|
||||
A ``put``/``patch`` never lands below a unit: units are written whole.
|
||||
"""
|
||||
where = f"{path.strip('/')}/{event.path.strip('/')}".strip("/")
|
||||
if event.kind == "put":
|
||||
return Event(event.kind, event.path, rtdb.opened(where, event.data))
|
||||
data = event.data if isinstance(event.data, dict) else {}
|
||||
body = {k: rtdb.opened(f"{where}/{k}", v) for k, v in data.items()}
|
||||
return Event(event.kind, event.path, body)
|
||||
|
||||
|
||||
async def _one_connection(
|
||||
rtdb: Rtdb, path: str, params: dict[str, str] | None
|
||||
) -> AsyncIterator[Event]:
|
||||
"""Events of a single connection; returns on ``auth_revoked`` or EOF."""
|
||||
async for kind, raw in _blocks(rtdb.open_stream(path, params)):
|
||||
if kind in {"put", "patch"}:
|
||||
payload = json.loads(raw)
|
||||
event = Event(kind, str(payload.get("path", "/")), payload.get("data"))
|
||||
yield opened(rtdb, path, event)
|
||||
elif kind == "auth_revoked":
|
||||
_logger.info("stream %s: token expired, reconnecting", path)
|
||||
return
|
||||
elif kind == "cancel":
|
||||
msg = f"stream {path} cancelled by the server"
|
||||
raise StreamCancelledError(msg)
|
||||
|
||||
|
||||
async def listen(
|
||||
rtdb: Rtdb,
|
||||
path: str,
|
||||
params: dict[str, str] | None = None,
|
||||
sleep: Callable[[float], Awaitable[None]] = asyncio.sleep,
|
||||
) -> AsyncGenerator[Event]:
|
||||
"""Every event at ``path`` across reconnects; runs until cancelled."""
|
||||
failures = 0
|
||||
while True:
|
||||
try:
|
||||
async for event in _one_connection(rtdb, path, params):
|
||||
failures = 0
|
||||
yield event
|
||||
except StreamCancelledError:
|
||||
raise
|
||||
except (RtdbError, ValueError) as err: # ValueError: a bad JSON payload
|
||||
delay = BACKOFF_S[min(failures, len(BACKOFF_S) - 1)]
|
||||
failures += 1
|
||||
_logger.warning("stream %s dropped (%s); retry in %.0fs", path, err, delay)
|
||||
await sleep(delay)
|
||||
2
rtdb_relay/rtdb_relay/tests/__init__.py
Normal file
2
rtdb_relay/rtdb_relay/tests/__init__.py
Normal file
@ -0,0 +1,2 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Tests of rtdb_relay; the fake RTDB and the tutor layout are reusable."""
|
||||
235
rtdb_relay/rtdb_relay/tests/_fake_rtdb.py
Normal file
235
rtdb_relay/rtdb_relay/tests/_fake_rtdb.py
Normal file
@ -0,0 +1,235 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""An in-memory Firebase RTDB behind ``httpx.MockTransport``.
|
||||
|
||||
REST semantics the relay relies on: GET (shallow, orderBy/startAt), PUT,
|
||||
PATCH, DELETE, ``X-Firebase-ETag`` / ``if-match`` with 412 + current value,
|
||||
``{".sv": "timestamp"}``, ``?auth=`` checking, and SSE streams (initial put,
|
||||
put/patch events, keep-alive, auth_revoked, cancel, drop).
|
||||
|
||||
The stored tree (``raw``) is what RTDB would hold: under the tutor layout
|
||||
every unit is sealed with :data:`TEST_KEY`, as the relay writes it.
|
||||
``get``/``set`` are the test's clear view (``set`` writes like a client
|
||||
would: sealed); ``sealed=False`` stores and serves plaintext.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from crdt_sync import E2eCipher
|
||||
import httpx
|
||||
|
||||
from rtdb_relay.layout import E2eLayoutError
|
||||
from rtdb_relay.rtdb import Rtdb
|
||||
from rtdb_relay.seal import Sealer
|
||||
from rtdb_relay.tests._fake_tree import (
|
||||
_assign,
|
||||
_event,
|
||||
_parts,
|
||||
_prune,
|
||||
_reply,
|
||||
etag,
|
||||
)
|
||||
from rtdb_relay.tests._layouts import TUTOR
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
AUTH = "test-token"
|
||||
DB = "https://db.test"
|
||||
TEST_KEY = bytes(range(32))
|
||||
ROOT = TUTOR.root
|
||||
|
||||
|
||||
def sealer() -> Sealer:
|
||||
"""The sealer of :data:`TEST_KEY`."""
|
||||
return Sealer(E2eCipher(TEST_KEY), TUTOR)
|
||||
|
||||
|
||||
class FakeRtdb:
|
||||
"""The database; ``now_ms`` resolves server timestamps."""
|
||||
|
||||
def __init__(self, now_ms: int = 1_800_000_000_000, *, sealed: bool = True) -> None:
|
||||
self.raw: dict[str, Any] = {}
|
||||
self.sealer = sealer() if sealed else None
|
||||
self.now_ms = now_ms
|
||||
self.streams: list[tuple[list[str], asyncio.Queue[str | None]]] = []
|
||||
self.fail: dict[tuple[str, str], int] = {} # (method, rel path) -> status
|
||||
self.raise_on: set[tuple[str, str]] = set() # -> httpx.ConnectError
|
||||
self.log: list[tuple[str, str]] = []
|
||||
self.token = AUTH
|
||||
|
||||
# -- the test's clear view -------------------------------------------------
|
||||
def _rel(self, path: str) -> list[str] | None:
|
||||
parts = _parts(path)
|
||||
root = _parts(ROOT)
|
||||
return parts[len(root) :] if parts[: len(root)] == root else None
|
||||
|
||||
def get(self, path: str) -> Any:
|
||||
"""The value at ``path``, sealed units opened."""
|
||||
rel = self._rel(path)
|
||||
if self.sealer is None or rel is None:
|
||||
return self.raw_get(path)
|
||||
opened = self.sealer.open("", self.raw_get(ROOT))
|
||||
node: Any = opened
|
||||
for part in rel:
|
||||
if not isinstance(node, dict) or part not in node:
|
||||
return None
|
||||
node = node[part]
|
||||
return node
|
||||
|
||||
def set(self, path: str, value: Any) -> None:
|
||||
"""Write ``value`` as a client would (sealed; inside a unit: resealed)."""
|
||||
rel = self._rel(path)
|
||||
if self.sealer is None or rel is None:
|
||||
self.raw_set(path, value)
|
||||
return
|
||||
unit = TUTOR.unit_path("/".join(rel))
|
||||
if unit is None:
|
||||
try:
|
||||
sealed = self.sealer.seal("/".join(rel), self._sv(value))
|
||||
except E2eLayoutError as err:
|
||||
# A mis-shaped node no sealer writes: stored as a hostile client would.
|
||||
logging.getLogger(__name__).warning("stored unsealed: %s", err)
|
||||
sealed = value
|
||||
self.raw_set(path, sealed)
|
||||
return
|
||||
node = self.get(f"{ROOT}/{unit}")
|
||||
node = node if isinstance(node, dict) else {}
|
||||
_assign(node, rel[len(_parts(unit)) :], value)
|
||||
self.raw_set(f"{ROOT}/{unit}", self.sealer.seal(unit, self._sv(node)))
|
||||
|
||||
# -- tree -----------------------------------------------------------------
|
||||
def raw_get(self, path: str) -> Any:
|
||||
"""The stored value at ``path`` (ciphertext under the root)."""
|
||||
node: Any = self.raw
|
||||
for part in _parts(path):
|
||||
if not isinstance(node, dict) or part not in node:
|
||||
return None
|
||||
node = node[part]
|
||||
return node
|
||||
|
||||
def _sv(self, value: Any) -> Any:
|
||||
if value == {".sv": "timestamp"}:
|
||||
return self.now_ms
|
||||
if isinstance(value, dict):
|
||||
out = {k: self._sv(v) for k, v in value.items()}
|
||||
return {k: v for k, v in out.items() if v is not None} or None
|
||||
return value
|
||||
|
||||
def raw_set(self, path: str, value: Any) -> None:
|
||||
"""Store ``value`` at ``path`` as given (server timestamps resolved)."""
|
||||
parts = _parts(path)
|
||||
value = self._sv(value)
|
||||
if not parts:
|
||||
self.raw = value if isinstance(value, dict) else {}
|
||||
return
|
||||
node = self.raw
|
||||
for part in parts[:-1]:
|
||||
if not isinstance(node.get(part), dict):
|
||||
node[part] = {}
|
||||
node = node[part]
|
||||
if value is None:
|
||||
node.pop(parts[-1], None)
|
||||
else:
|
||||
node[parts[-1]] = value
|
||||
self.raw = _prune(self.raw) or {}
|
||||
|
||||
# -- streams --------------------------------------------------------------
|
||||
def _notify(self, kind: str, path: str, data: Any) -> None:
|
||||
where = _parts(path)
|
||||
for listen, queue in self.streams:
|
||||
if where[: len(listen)] == listen:
|
||||
rel = "/" + "/".join(where[len(listen) :])
|
||||
queue.put_nowait(_event(kind, rel, data))
|
||||
elif listen[: len(where)] == where:
|
||||
queue.put_nowait(_event("put", "/", self.raw_get("/".join(listen))))
|
||||
|
||||
def push(self, raw: str | None) -> None:
|
||||
"""Send a raw SSE block (or ``None``: close) to every stream."""
|
||||
for _, queue in self.streams:
|
||||
queue.put_nowait(raw)
|
||||
|
||||
# -- HTTP -----------------------------------------------------------------
|
||||
def client(self) -> httpx.AsyncClient:
|
||||
return httpx.AsyncClient(transport=httpx.MockTransport(self.handle))
|
||||
|
||||
def rtdb(self) -> Rtdb:
|
||||
async def token() -> str:
|
||||
return self.token
|
||||
|
||||
return Rtdb(DB, token, client=self.client(), root=ROOT, sealer=self.sealer)
|
||||
|
||||
async def handle(self, request: httpx.Request) -> httpx.Response:
|
||||
full = request.url.path.removesuffix(".json").strip("/")
|
||||
path = (
|
||||
"/" + full.removeprefix(ROOT).strip("/")
|
||||
if full.startswith(ROOT)
|
||||
else "/" + full
|
||||
)
|
||||
method = request.method
|
||||
self.log.append((method, path))
|
||||
if (method, path) in self.raise_on:
|
||||
msg = "boom"
|
||||
raise httpx.ConnectError(msg, request=request)
|
||||
if request.url.params.get("auth") != AUTH:
|
||||
return _reply(401, {"error": "Permission denied"})
|
||||
if (method, path) in self.fail:
|
||||
return _reply(self.fail[(method, path)], {"error": "x"})
|
||||
if request.headers.get("accept") == "text/event-stream":
|
||||
return self._stream(full)
|
||||
current = self.raw_get(full)
|
||||
match = request.headers.get("if-match")
|
||||
if match is not None and match != etag(current):
|
||||
return _reply(412, current, headers={"ETag": etag(current)})
|
||||
if method == "GET":
|
||||
return self._get(request, full, current)
|
||||
return self._write(method, full, request)
|
||||
|
||||
def _write(self, method: str, full: str, request: httpx.Request) -> httpx.Response:
|
||||
if method == "PUT":
|
||||
self.raw_set(full, json.loads(request.content))
|
||||
self._notify("put", full, self.raw_get(full))
|
||||
stored = self.raw_get(full)
|
||||
return _reply(200, stored, headers={"ETag": etag(stored)})
|
||||
if method == "PATCH":
|
||||
body = json.loads(request.content)
|
||||
for key, value in body.items():
|
||||
self.raw_set(f"{full}/{key}", value)
|
||||
self._notify("patch", full, self._sv(body))
|
||||
return _reply(200, body)
|
||||
self.raw_set(full, None) # DELETE
|
||||
self._notify("put", full, None)
|
||||
return _reply(200, None)
|
||||
|
||||
def _get(self, request: httpx.Request, full: str, value: Any) -> httpx.Response:
|
||||
params = request.url.params
|
||||
if params.get("shallow") == "true" and isinstance(value, dict):
|
||||
value = dict.fromkeys(value, True)
|
||||
start = params.get("startAt")
|
||||
if start and isinstance(value, dict):
|
||||
value = {k: v for k, v in value.items() if k >= json.loads(start)}
|
||||
headers = (
|
||||
{"ETag": etag(value)} if request.headers.get("x-firebase-etag") else {}
|
||||
)
|
||||
return _reply(200, value, headers=headers)
|
||||
|
||||
def _stream(self, full: str) -> httpx.Response:
|
||||
queue: asyncio.Queue[str | None] = asyncio.Queue()
|
||||
entry = (_parts(full), queue)
|
||||
self.streams.append(entry)
|
||||
queue.put_nowait(_event("put", "/", self.raw_get(full)))
|
||||
|
||||
async def body() -> AsyncIterator[bytes]:
|
||||
try:
|
||||
while (block := await queue.get()) is not None:
|
||||
yield block.encode()
|
||||
finally:
|
||||
self.streams.remove(entry)
|
||||
|
||||
return httpx.Response(
|
||||
200, content=body(), headers={"content-type": "text/event-stream"}
|
||||
)
|
||||
49
rtdb_relay/rtdb_relay/tests/_fake_tree.py
Normal file
49
rtdb_relay/rtdb_relay/tests/_fake_tree.py
Normal file
@ -0,0 +1,49 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Tree and wire helpers of the fake RTDB (``_fake_rtdb``)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
def _parts(path: str) -> list[str]:
|
||||
return [p for p in path.strip("/").split("/") if p]
|
||||
|
||||
|
||||
def etag(value: object) -> str:
|
||||
if value is None:
|
||||
return "null_etag"
|
||||
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()
|
||||
|
||||
|
||||
def _event(kind: str, path: str, data: Any) -> str:
|
||||
return f"event: {kind}\ndata: {json.dumps({'path': path, 'data': data})}\n\n"
|
||||
|
||||
|
||||
def _assign(node: dict[str, Any], parts: list[str], value: Any) -> None:
|
||||
for part in parts[:-1]:
|
||||
if not isinstance(node.get(part), dict):
|
||||
node[part] = {}
|
||||
node = node[part]
|
||||
node[parts[-1]] = value
|
||||
|
||||
|
||||
def _prune(node: Any) -> Any:
|
||||
"""RTDB keeps no empty objects."""
|
||||
if not isinstance(node, dict):
|
||||
return node
|
||||
out = {k: _prune(v) for k, v in node.items()}
|
||||
return {k: v for k, v in out.items() if v is not None and v != {}} or None
|
||||
|
||||
|
||||
def _reply(
|
||||
status: int, value: Any, headers: dict[str, str] | None = None
|
||||
) -> httpx.Response:
|
||||
"""A JSON response; ``httpx.Response(json=None)`` would send no body at all."""
|
||||
return httpx.Response(
|
||||
status, content=json.dumps(value).encode(), headers=headers or {}
|
||||
)
|
||||
40
rtdb_relay/rtdb_relay/tests/_layouts.py
Normal file
40
rtdb_relay/rtdb_relay/tests/_layouts.py
Normal file
@ -0,0 +1,40 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""The tutor's layout (its DOCS-contract §12): the reference the vectors pin."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from rtdb_relay.layout import Layout
|
||||
|
||||
VECTORS: dict[str, Any] = json.loads(
|
||||
(Path(__file__).parent / "data" / "e2e_vectors.json").read_text()
|
||||
)
|
||||
TUTOR = Layout(
|
||||
root="tutor/v1-e2e",
|
||||
clear=("meta", "rev", "migration", "requests/*/claim", "sessions/*/progress/rid"),
|
||||
units={
|
||||
"live": frozenset(),
|
||||
"prefs/*": frozenset(),
|
||||
"stats/*": frozenset(),
|
||||
"presence/workers/*": frozenset({"beat"}),
|
||||
"presence/clients/*": frozenset({"seen"}),
|
||||
"uploads/*/*": frozenset(),
|
||||
"requests/*/req": frozenset({"created"}),
|
||||
"requests/*/result": frozenset({"at"}),
|
||||
"sessions/*/meta/*": frozenset(),
|
||||
"sessions/*/msgs/*": frozenset(),
|
||||
"sessions/*/receipts/*": frozenset(),
|
||||
"sessions/*/user/*": frozenset(),
|
||||
"sessions/*/progress/steps/*": frozenset(),
|
||||
"session_index/*": frozenset(),
|
||||
"subjects/*/meta": frozenset(),
|
||||
"subjects/*/plan": frozenset(),
|
||||
"subjects/*/cards": frozenset(),
|
||||
"subjects/*/concepts": frozenset(),
|
||||
"log/*/*": frozenset(),
|
||||
"feedback/*": frozenset(),
|
||||
},
|
||||
)
|
||||
266
rtdb_relay/rtdb_relay/tests/data/e2e_vectors.json
Normal file
266
rtdb_relay/rtdb_relay/tests/data/e2e_vectors.json
Normal file
@ -0,0 +1,266 @@
|
||||
{
|
||||
"key_hex": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f",
|
||||
"root": "tutor/v1-e2e",
|
||||
"sealed_key": "_e2e",
|
||||
"rules": [
|
||||
{
|
||||
"path": "",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "meta",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "meta/min_client",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "rev",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "rev/catalog",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "live",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "prefs",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "prefs/current_subject",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "stats",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "stats/today",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "stats/today/new_cap",
|
||||
"kind": "below"
|
||||
},
|
||||
{
|
||||
"path": "presence/workers/d1",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "presence/clients/d1",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "uploads/u1/c0001",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "requests",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "requests/r1",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "requests/r1/req",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "requests/r1/claim",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "requests/r1/claim/by",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "requests/r1/result",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/meta",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/meta/busy",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/meta/busy/phase",
|
||||
"kind": "below"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/msgs/m0001",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/receipts/r0001",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/user",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/user/archived",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/progress",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/progress/rid",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/progress/steps/s0001",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "session_index/s1",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "subjects/car/meta",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "subjects/car/meta/rev",
|
||||
"kind": "below"
|
||||
},
|
||||
{
|
||||
"path": "subjects/car/plan",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "subjects/car/cards",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "subjects/car/concepts",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "log/d1",
|
||||
"kind": "struct"
|
||||
},
|
||||
{
|
||||
"path": "log/d1/e1",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "feedback/f1",
|
||||
"kind": "unit"
|
||||
},
|
||||
{
|
||||
"path": "migration/v1_copied_at",
|
||||
"kind": "clear"
|
||||
},
|
||||
{
|
||||
"path": "secrets",
|
||||
"kind": "unknown"
|
||||
},
|
||||
{
|
||||
"path": "sessions/s1/other",
|
||||
"kind": "unknown"
|
||||
}
|
||||
],
|
||||
"units": [
|
||||
{
|
||||
"path": "requests/01J0000000000000000000000R/req",
|
||||
"value": {
|
||||
"kind": "message",
|
||||
"dev": "d1",
|
||||
"created": 1800000000000,
|
||||
"sent_at": 1799999999000,
|
||||
"body": {
|
||||
"session": "20261012-170448-5b71",
|
||||
"text": "zażółć gęślą jaźń"
|
||||
}
|
||||
},
|
||||
"stored": {
|
||||
"_e2e": "AVnxpT1g95cKDT1ORKsCeyjnBNJGfjyto/ezodv1ysIc2zHOmj77unCOLXQqYgQjvbfbNub710heso/QkMAUS5Mq+M5+/yIW56HevylofoiyPapWi7obYbu9otQwuOUeiQzutdJpQNw9x7dfUPLZAu1BRQoboIyCG0TrNH3X9LkBIRgWLHOpf3lxhmXGJht3muR4lS4zwDOxj7pB1lRluLM/mnJcXqzDdmgHHw==",
|
||||
"created": 1800000000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "requests/01J0000000000000000000000R/result",
|
||||
"value": {
|
||||
"ok": true,
|
||||
"at": 1800000000500,
|
||||
"data": {
|
||||
"msg": "m0002"
|
||||
}
|
||||
},
|
||||
"stored": {
|
||||
"_e2e": "AREDDVq3hgmR9xsL5pUrhSJbK+fBCm823Hx8oKUo+ExwPT342IvIEqu55Rog+u4wWXxON6Vy5xxJZwd4+gXfxkQ49j6cb+H0tyRO",
|
||||
"at": 1800000000500
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "presence/clients/d1",
|
||||
"value": {
|
||||
"platform": "android",
|
||||
"app": "1.0.0",
|
||||
"seen": 1800000000000
|
||||
},
|
||||
"stored": {
|
||||
"_e2e": "AYhbgKJJ4tsHmFB0THKTxQoCzwKOpa/3ioNcYmtj2UhmhRZyavA7614RT6y3aPAmBWf5uNhQgkKKOO+Qavw4XAad+7xttA23RB7Xhb0=",
|
||||
"seen": 1800000000000
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "prefs/current_subject",
|
||||
"value": "car",
|
||||
"stored": {
|
||||
"_e2e": "AWEy8u/msdxYHsp7PQ0jkzrYpdNP6SEqbJ0SUyqzT2DzoEpXRf+1vKAQVh1hUQ=="
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "sessions/20261012-170448-5b71/user/archived",
|
||||
"value": true,
|
||||
"stored": {
|
||||
"_e2e": "AbS7aTNu+meRs1jvcw/alGgKKgbhV83VPihFMd8AmqW6QhwgCQMg46hnrJls"
|
||||
}
|
||||
},
|
||||
{
|
||||
"path": "log/d1/01J0000000000000000000000E",
|
||||
"value": {
|
||||
"type": "review",
|
||||
"card": "ld-x",
|
||||
"r": 3,
|
||||
"t": 1800000000000
|
||||
},
|
||||
"stored": {
|
||||
"_e2e": "ASqR1ybCH4+WKx7P0nihgr8IOcGMsw1fXXyigH7kgrAZHru4DX0lXW2cZF0DPwQxeay3dS36BYXPN6OgO8xnXjcTz3hUwjjlVLANzZvG0zICImA8rBDy5bInd/INDmc5"
|
||||
}
|
||||
}
|
||||
],
|
||||
"header": {
|
||||
"path": "tutor/e2e",
|
||||
"passphrase": "correct horse",
|
||||
"text": "{\"v\":1,\"kdf\":\"argon2id\",\"salt\":\"67H6FMZ+zlSFuBRuMIy2IQ==\",\"m\":8,\"t\":1,\"p\":1,\"check\":\"JQdQLQiowhtNkztQDPtOOUARTzUStpsKZmPHGZXEJ/8=\"}",
|
||||
"unit": {
|
||||
"path": "log/0b6f2c1e-8f2a-4c1d-9e3b-7a5d4c3b2a10/01J0000000000000000000000E",
|
||||
"value": {
|
||||
"type": "review",
|
||||
"card": "ld-x",
|
||||
"r": 3
|
||||
},
|
||||
"stored": {
|
||||
"_e2e": "ATvPPA3qI5F70kbmI3XDguYjUsmQZQD2j8fY85sGGOg5IcxeYQn4mbWichyVU82pQQ7WA1YDngEs0K17HImxMyJSUf2H622XzinCPSqN"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
60
rtdb_relay/rtdb_relay/tests/test_lease.py
Normal file
60
rtdb_relay/rtdb_relay/tests/test_lease.py
Normal file
@ -0,0 +1,60 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""The claim lease: take, lose a race, renew, lose the lease."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
import pytest
|
||||
|
||||
from rtdb_relay.lease import Lease, claimable, ms, valid_rid
|
||||
from rtdb_relay.rtdb import ConflictError
|
||||
from rtdb_relay.tests._fake_rtdb import ROOT, FakeRtdb
|
||||
|
||||
RID = "01J0000000000000000000000R"
|
||||
|
||||
|
||||
def test_helpers() -> None:
|
||||
assert valid_rid(RID)
|
||||
assert not valid_rid("../x")
|
||||
assert ms(1.5) == 1500
|
||||
assert claimable(None, 0)
|
||||
assert claimable({"lease_until": 999}, 1.0)
|
||||
assert not claimable({"lease_until": 2000, "by": "x"}, 1.0)
|
||||
assert claimable({"lease_until": 2000, "by": "me"}, 1.0, "me")
|
||||
assert claimable({"lease_until": "?"}, 1.0)
|
||||
|
||||
|
||||
def test_take_renew_and_races(
|
||||
caplog: pytest.LogCaptureFixture, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
fake = FakeRtdb()
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
lease = Lease(db, RID, "pc", lambda: 1.0)
|
||||
fake.set(f"{ROOT}/{lease.path}", {"attempt": 2, "lease_until": 0})
|
||||
assert await lease.take()
|
||||
assert lease.attempt == 3
|
||||
await lease.renew("running")
|
||||
assert fake.get(f"{ROOT}/{lease.path}")["phase"] == "running"
|
||||
fake.set(f"{ROOT}/{lease.path}", {"by": "other", "lease_until": 9e15})
|
||||
await lease.renew() # lost: only logged
|
||||
assert fake.get(f"{ROOT}/{lease.path}")["by"] == "other"
|
||||
assert not await Lease(db, RID, "pc", lambda: 1.0).take()
|
||||
fake.set(f"{ROOT}/{lease.path}", {"attempt": "x"})
|
||||
|
||||
async def lose(*args: object) -> tuple[object, str]:
|
||||
raise ConflictError("p", None, "")
|
||||
|
||||
monkeypatch.setattr(db, "put_etag", lose)
|
||||
racer = Lease(db, RID, "pc", lambda: 1.0)
|
||||
assert not await racer.take()
|
||||
assert racer.attempt == 1
|
||||
await db.aclose()
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
asyncio.run(go())
|
||||
assert "lost to another writer" in caplog.text
|
||||
assert "claimed" in caplog.text
|
||||
35
rtdb_relay/rtdb_relay/tests/test_presence.py
Normal file
35
rtdb_relay/rtdb_relay/tests/test_presence.py
Normal file
@ -0,0 +1,35 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""The worker heartbeat."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from rtdb_relay.presence import beat_forever, node
|
||||
from rtdb_relay.tests._fake_rtdb import ROOT, FakeRtdb
|
||||
|
||||
|
||||
def test_beats_until_cancelled() -> None:
|
||||
fake = FakeRtdb()
|
||||
sleeps: list[float] = []
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
|
||||
async def sleep(s: float) -> None:
|
||||
sleeps.append(s)
|
||||
if len(sleeps) == 2:
|
||||
raise asyncio.CancelledError
|
||||
|
||||
with pytest.raises(asyncio.CancelledError):
|
||||
await beat_forever(db, "pc", "1.0", sleep)
|
||||
await db.aclose()
|
||||
|
||||
asyncio.run(go())
|
||||
stored = fake.get(f"{ROOT}/presence/workers/pc")
|
||||
assert stored["beat"] == fake.now_ms
|
||||
assert stored["version"] == "1.0"
|
||||
assert sleeps == [30.0, 30.0]
|
||||
assert set(node("v")) == {"beat", "host", "version"}
|
||||
163
rtdb_relay/rtdb_relay/tests/test_rtdb.py
Normal file
163
rtdb_relay/rtdb_relay/tests/test_rtdb.py
Normal file
@ -0,0 +1,163 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""The RTDB REST client against the in-memory fake: ETags, 412, errors."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
from rtdb_relay import rtdb as rtdb_mod
|
||||
from rtdb_relay.rtdb import ConflictError, Rtdb, RtdbError
|
||||
from rtdb_relay.tests._fake_rtdb import DB, ROOT, FakeRtdb, etag
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator, Coroutine
|
||||
|
||||
|
||||
async def _drain(db: Rtdb) -> list[str]:
|
||||
return [line async for line in db.open_stream("s")]
|
||||
|
||||
|
||||
def run(coro: Coroutine[Any, Any, None]) -> None:
|
||||
asyncio.run(coro)
|
||||
|
||||
|
||||
def test_url_quotes_segments_and_root() -> None:
|
||||
fake = FakeRtdb(sealed=False)
|
||||
db = fake.rtdb()
|
||||
assert db.url("requests/a b") == f"{DB}/tutor/v1-e2e/requests/a%20b.json"
|
||||
assert db.url("/") == f"{DB}/tutor/v1-e2e.json"
|
||||
assert db.url("subjects/pojęcia-z-łodzi") == (
|
||||
f"{DB}/tutor/v1-e2e/subjects/poj%C4%99cia-z-%C5%82odzi.json"
|
||||
)
|
||||
run(db.aclose())
|
||||
|
||||
|
||||
def test_put_get_patch_delete_round_trip() -> None:
|
||||
fake = FakeRtdb(now_ms=42, sealed=False)
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
stored = await db.put("a/b", {"x": 1, "at": rtdb_mod.SERVER_TIME})
|
||||
assert stored == {"x": 1, "at": 42}
|
||||
await db.patch("a/b", {"y": 2, "x": None})
|
||||
assert await db.get("a/b") == {"at": 42, "y": 2}
|
||||
assert await db.get("a", {"shallow": "true"}) == {"b": True}
|
||||
await db.delete("a/b")
|
||||
assert await db.get("a/b") is None
|
||||
await db.aclose()
|
||||
|
||||
run(go())
|
||||
|
||||
|
||||
def test_conditional_put_and_lost_race() -> None:
|
||||
fake = FakeRtdb(sealed=False)
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
value, tag = await db.get_etag("c")
|
||||
assert value is None
|
||||
assert tag == "null_etag"
|
||||
_, tag2 = await db.put_etag("c", {"v": 1}, tag)
|
||||
assert tag2 == etag({"v": 1})
|
||||
with pytest.raises(ConflictError) as lost:
|
||||
await db.put_etag("c", {"v": 2}, tag)
|
||||
assert lost.value.value == {"v": 1}
|
||||
assert lost.value.etag == tag2
|
||||
await db.delete("c", if_match=tag2)
|
||||
await db.aclose()
|
||||
|
||||
run(go())
|
||||
|
||||
|
||||
def test_http_errors_never_show_the_url_or_token(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
fake = FakeRtdb(sealed=False)
|
||||
fake.fail[("GET", "/x")] = 500
|
||||
fake.raise_on.add(("PUT", "/y"))
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
with pytest.raises(RtdbError, match=r"^GET x: HTTP 500$"):
|
||||
await db.get("x")
|
||||
with pytest.raises(RtdbError, match=r"^PUT y: ConnectError$"):
|
||||
await db.put("y", 1)
|
||||
await db.aclose()
|
||||
|
||||
with caplog.at_level(logging.DEBUG):
|
||||
run(go())
|
||||
assert "test-token" not in caplog.text
|
||||
|
||||
|
||||
def test_wrong_token_is_an_error() -> None:
|
||||
fake = FakeRtdb(sealed=False)
|
||||
fake.token = fake.token[::-1]
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
with pytest.raises(RtdbError, match="HTTP 401"):
|
||||
await db.get("x")
|
||||
await db.aclose()
|
||||
|
||||
run(go())
|
||||
|
||||
|
||||
def test_default_client_is_created() -> None:
|
||||
async def token() -> str:
|
||||
return "t"
|
||||
|
||||
async def go() -> None:
|
||||
db = Rtdb(DB + "/", token, root=ROOT)
|
||||
assert isinstance(db.client, httpx.AsyncClient)
|
||||
assert db.url("x") == f"{DB}/tutor/v1-e2e/x.json"
|
||||
with pytest.raises(RtdbError, match="GET x: "):
|
||||
await db.get("x") # the test sandbox blocks real network
|
||||
await db.aclose()
|
||||
|
||||
run(go())
|
||||
|
||||
|
||||
def test_open_stream_errors() -> None:
|
||||
fake = FakeRtdb(sealed=False)
|
||||
fake.fail[("GET", "/s")] = 401
|
||||
fake.raise_on.add(("GET", "/t"))
|
||||
|
||||
async def drain(db: Rtdb, path: str) -> list[str]:
|
||||
return [line async for line in db.open_stream(path)]
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
with pytest.raises(RtdbError, match="STREAM s: HTTP 401"):
|
||||
await drain(db, "s")
|
||||
with pytest.raises(RtdbError, match="STREAM t: ConnectError"):
|
||||
await drain(db, "t")
|
||||
await db.aclose()
|
||||
|
||||
run(go())
|
||||
|
||||
|
||||
def test_open_stream_read_error_mid_stream() -> None:
|
||||
async def handler(request: httpx.Request) -> httpx.Response:
|
||||
async def body() -> AsyncIterator[bytes]:
|
||||
yield b"event: keep-alive\n"
|
||||
msg = "cut"
|
||||
raise httpx.ReadError(msg, request=request)
|
||||
|
||||
return httpx.Response(200, content=body())
|
||||
|
||||
async def token() -> str:
|
||||
return "t"
|
||||
|
||||
async def go() -> None:
|
||||
client = httpx.AsyncClient(transport=httpx.MockTransport(handler))
|
||||
db = Rtdb(DB, token, client=client, root=ROOT)
|
||||
with pytest.raises(RtdbError, match="STREAM s: ReadError"):
|
||||
await _drain(db)
|
||||
await db.aclose()
|
||||
|
||||
run(go())
|
||||
112
rtdb_relay/rtdb_relay/tests/test_seal.py
Normal file
112
rtdb_relay/rtdb_relay/tests/test_seal.py
Normal file
@ -0,0 +1,112 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Sealing: the tutor's contract vectors still open and classify the same."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from crdt_sync import E2eCipher
|
||||
import pytest
|
||||
|
||||
from rtdb_relay import sse
|
||||
from rtdb_relay.layout import E2eLayoutError, Layout, parts_of
|
||||
from rtdb_relay.seal import SEALED, Sealer, dumps
|
||||
from rtdb_relay.tests._fake_rtdb import ROOT, FakeRtdb
|
||||
from rtdb_relay.tests._layouts import TUTOR, VECTORS
|
||||
|
||||
|
||||
def _sealer() -> Sealer:
|
||||
return Sealer(E2eCipher(bytes.fromhex(VECTORS["key_hex"])), TUTOR)
|
||||
|
||||
|
||||
def test_vectors_pin_the_rules_and_open() -> None:
|
||||
assert VECTORS["root"] == TUTOR.root == "tutor/v1-e2e"
|
||||
assert VECTORS["sealed_key"] == SEALED
|
||||
for rule in VECTORS["rules"]:
|
||||
assert TUTOR.classify(parts_of(rule["path"])).kind == rule["kind"], rule
|
||||
s = _sealer()
|
||||
for unit in VECTORS["units"]:
|
||||
assert s.open(unit["path"], unit["stored"]) == unit["value"]
|
||||
assert set(unit["stored"]) - {SEALED} <= {"created", "at", "seen", "beat"}
|
||||
|
||||
|
||||
def test_sealed_bytes_are_the_cipher_over_compact_json_at_the_full_path() -> None:
|
||||
"""Differential: our seal == crdt_sync's cipher on the vectors' format."""
|
||||
cipher = E2eCipher(bytes.fromhex(VECTORS["key_hex"]))
|
||||
s = Sealer(cipher, TUTOR)
|
||||
for unit in VECTORS["units"]:
|
||||
ours = s.seal(unit["path"], unit["value"])
|
||||
assert isinstance(ours, dict)
|
||||
assert isinstance(unit["stored"], dict)
|
||||
clear = {k: v for k, v in unit["stored"].items() if k != SEALED}
|
||||
assert {k: v for k, v in ours.items() if k != SEALED} == clear
|
||||
aad = f"{TUTOR.root}/{unit['path']}"
|
||||
text = cipher.open(ours[SEALED], path=aad)
|
||||
value = unit["value"]
|
||||
rest = (
|
||||
{k: v for k, v in value.items() if k not in clear}
|
||||
if isinstance(value, dict)
|
||||
else value
|
||||
)
|
||||
assert text == dumps(rest)
|
||||
# And the vector's own blob opens to the same plaintext bytes.
|
||||
assert cipher.open(unit["stored"][SEALED], path=aad) == text
|
||||
|
||||
|
||||
def test_seal_refuses_unknown_and_inside_unit_writes() -> None:
|
||||
s = _sealer()
|
||||
with pytest.raises(E2eLayoutError):
|
||||
s.seal("secrets", {"a": 1})
|
||||
with pytest.raises(E2eLayoutError):
|
||||
s.seal("sessions", "flat") # a scalar where a tree belongs
|
||||
with pytest.raises(E2eLayoutError):
|
||||
s.seal_patch("sessions/s/msgs/m0001", {"text": "x"})
|
||||
with pytest.raises(E2eLayoutError):
|
||||
s.open("stats/today/x", 1)
|
||||
assert s.unit_path("stats/today/x") == "stats/today"
|
||||
assert s.unit_path("stats/today") is None
|
||||
assert s.seal("live", {"a": None}) is None # pruned to nothing
|
||||
assert s.seal("meta/x", 1) == 1
|
||||
assert s.seal_patch("prefs", {"a": None}) == {"a": None}
|
||||
|
||||
|
||||
def test_open_drops_a_moved_or_tampered_unit_and_keeps_plaintext() -> None:
|
||||
s = _sealer()
|
||||
sealed = s.seal("prefs/current_subject", "car")
|
||||
assert s.open("prefs", {"current_subject": sealed}) == {"current_subject": "car"}
|
||||
assert s.open("prefs", {"other": sealed}) == {}
|
||||
assert s.open("prefs", {"current_subject": {SEALED: "AQ=="}}) == {}
|
||||
assert s.open("prefs/current_subject", "car") == "car"
|
||||
assert s.open("sessions", [{"x": 1}]) == [{"x": 1}]
|
||||
assert s.open("meta", {"a": 1}) == {"a": 1}
|
||||
|
||||
|
||||
def test_root_override_changes_the_aad() -> None:
|
||||
layout = Layout(root="app/v1-e2e", clear=(), units={"x": frozenset()})
|
||||
cipher = E2eCipher(bytes(range(32)))
|
||||
blob = Sealer(cipher, layout).seal("x", 1)
|
||||
assert Sealer(cipher, layout, root="other").open("x", blob) is None
|
||||
assert Sealer(cipher, layout).open("x", blob) == 1
|
||||
|
||||
|
||||
def test_rtdb_reads_below_a_unit_and_streams_open() -> None:
|
||||
fake = FakeRtdb()
|
||||
fake.set(f"{ROOT}/stats/today", {"new_cap": 25})
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
assert await db.get("stats/today/new_cap") == 25
|
||||
assert await db.get("stats/today/nope/deeper") is None
|
||||
stream = sse.listen(db, "stats")
|
||||
first = await anext(stream)
|
||||
assert first.data == {"today": {"new_cap": 25}}
|
||||
await db.patch("stats", {"days": {"2026-11-12": {"m": 3}}})
|
||||
second = await anext(stream)
|
||||
assert second.kind == "patch"
|
||||
assert second.data == {"days": {"2026-11-12": {"m": 3}}}
|
||||
await stream.aclose()
|
||||
await db.aclose()
|
||||
|
||||
asyncio.run(go())
|
||||
assert "new_cap" not in json.dumps(fake.raw)
|
||||
147
rtdb_relay/rtdb_relay/tests/test_server.py
Normal file
147
rtdb_relay/rtdb_relay/tests/test_server.py
Normal file
@ -0,0 +1,147 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""The request server: claim, run the kind's handler, write the result."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from rtdb_relay import server as server_mod
|
||||
from rtdb_relay.server import RequestError, RequestServer
|
||||
from rtdb_relay.tests._fake_rtdb import ROOT, FakeRtdb
|
||||
|
||||
RID = "01J0000000000000000000000R"
|
||||
NOW = 1_800_000_000.0
|
||||
|
||||
|
||||
async def _echo(body: dict[str, Any]) -> dict[str, Any]:
|
||||
return {"echo": body.get("x")}
|
||||
|
||||
|
||||
async def _refuse(body: dict[str, Any]) -> dict[str, Any]:
|
||||
raise RequestError("not_now", "cannot start", retry=True)
|
||||
|
||||
|
||||
async def _bug(body: dict[str, Any]) -> dict[str, Any]:
|
||||
raise KeyError(body["missing"])
|
||||
|
||||
|
||||
HANDLERS = {"echo": _echo, "refuse": _refuse, "bug": _bug}
|
||||
|
||||
|
||||
def _req(kind: str, body: object = None) -> dict[str, Any]:
|
||||
return {"kind": kind, "dev": "d1", "created": 1, "sent_at": 1, "body": body}
|
||||
|
||||
|
||||
def _answer(fake: FakeRtdb, rid: str = RID) -> dict[str, Any] | None:
|
||||
async def go() -> dict[str, Any] | None:
|
||||
db = fake.rtdb()
|
||||
srv = RequestServer(db, "pc", HANDLERS, now=lambda: NOW)
|
||||
node = await srv._raw.get(f"requests/{rid}")
|
||||
out = await srv.answer(rid, node if isinstance(node, dict) else {})
|
||||
await db.aclose()
|
||||
return out
|
||||
|
||||
return asyncio.run(go())
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("kind", "body", "want"),
|
||||
[
|
||||
("echo", {"x": 7}, {"ok": True, "kind": "echo", "echo": 7}),
|
||||
("echo", "not a dict", {"ok": True, "kind": "echo"}),
|
||||
("nope", None, {"ok": False, "kind": "nope", "code": "unknown_kind"}),
|
||||
("refuse", {}, {"ok": False, "code": "not_now", "retry": True}),
|
||||
("bug", {}, {"ok": False, "code": "internal", "error": "KeyError"}),
|
||||
],
|
||||
)
|
||||
def test_answers_by_kind(kind: str, body: object, want: dict[str, Any]) -> None:
|
||||
fake = FakeRtdb()
|
||||
fake.set(f"{ROOT}/requests/{RID}/req", _req(kind, body))
|
||||
assert _answer(fake) is not None
|
||||
result = fake.get(f"{ROOT}/requests/{RID}/result")
|
||||
assert want.items() <= result.items()
|
||||
assert result["at"] == fake.now_ms
|
||||
claim = fake.get(f"{ROOT}/requests/{RID}/claim")
|
||||
assert claim["by"] == "pc"
|
||||
assert claim["attempt"] == 1
|
||||
|
||||
|
||||
def test_refuses_an_unsealed_request_and_skips_what_is_not_pending() -> None:
|
||||
fake = FakeRtdb()
|
||||
fake.raw_set(f"{ROOT}/requests/{RID}/req", _req("echo", {"x": 1}))
|
||||
assert _answer(fake) is not None
|
||||
assert fake.get(f"{ROOT}/requests/{RID}/result")["code"] == "unsealed"
|
||||
assert _answer(fake) is None # already has a result
|
||||
assert _answer(fake, "not-a-ulid") is None
|
||||
fake2 = FakeRtdb()
|
||||
fake2.set(f"{ROOT}/requests/{RID}/claim", {"by": "other", "lease_until": 9e15})
|
||||
assert _answer(fake2) is None # no req
|
||||
fake2.set(f"{ROOT}/requests/{RID}/req", _req("echo"))
|
||||
assert _answer(fake2) is None # another worker holds the lease
|
||||
|
||||
|
||||
def test_wrong_key_request_reads_as_unsealed() -> None:
|
||||
fake = FakeRtdb()
|
||||
fake.raw_set(f"{ROOT}/requests/{RID}/req", {"_e2e": "AQ==", "created": 1})
|
||||
_answer(fake)
|
||||
assert fake.get(f"{ROOT}/requests/{RID}/result")["code"] == "unsealed"
|
||||
|
||||
|
||||
def test_run_serves_new_requests_and_renews_long_leases(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
monkeypatch.setattr(server_mod, "RENEW_S", 0.01)
|
||||
fake = FakeRtdb()
|
||||
seen: list[str] = []
|
||||
|
||||
async def slow(body: dict[str, Any]) -> dict[str, Any]:
|
||||
await asyncio.sleep(0.05)
|
||||
claim = fake.get(f"{ROOT}/requests/{RID}/claim")
|
||||
seen.append(claim["phase"])
|
||||
return {}
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
srv = RequestServer(db, "pc", {"slow": slow})
|
||||
task = asyncio.create_task(srv.run())
|
||||
await asyncio.sleep(0.01)
|
||||
await db.put(f"requests/{RID}/req", _req("slow"))
|
||||
for _ in range(200):
|
||||
if fake.get(f"{ROOT}/requests/{RID}/result"):
|
||||
break
|
||||
await asyncio.sleep(0.01)
|
||||
task.cancel()
|
||||
await asyncio.gather(task, return_exceptions=True)
|
||||
await db.aclose()
|
||||
|
||||
asyncio.run(go())
|
||||
assert fake.get(f"{ROOT}/requests/{RID}/result")["ok"] is True
|
||||
assert seen == ["running"]
|
||||
|
||||
|
||||
def test_dispatch_skips_busy_and_logs_a_failing_request(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
fake = FakeRtdb()
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
srv = RequestServer(db, "pc", HANDLERS)
|
||||
|
||||
async def boom(rid: str, node: dict[str, Any]) -> None:
|
||||
raise RuntimeError(rid)
|
||||
|
||||
srv.answer = boom # type: ignore[method-assign]
|
||||
srv.dispatch({RID: {"req": {}}, "x": "flat", "y": {"result": {}}})
|
||||
srv.dispatch({RID: {"req": {}}}) # in flight: not started twice
|
||||
assert len(srv.tasks) == 1
|
||||
await asyncio.gather(*srv.tasks)
|
||||
await db.aclose()
|
||||
|
||||
with caplog.at_level(logging.ERROR):
|
||||
asyncio.run(go())
|
||||
assert f"request {RID} failed" in caplog.text
|
||||
145
rtdb_relay/rtdb_relay/tests/test_sse.py
Normal file
145
rtdb_relay/rtdb_relay/tests/test_sse.py
Normal file
@ -0,0 +1,145 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""SSE parsing, reconnects (drop, auth_revoked, errors), cancel; the mirror."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from rtdb_relay import mirror, sse
|
||||
from rtdb_relay.sse import Event, StreamCancelledError, parse
|
||||
from rtdb_relay.tests._fake_rtdb import FakeRtdb
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
|
||||
|
||||
def test_parse_block() -> None:
|
||||
assert parse(["event: put", 'data: {"a":1}', "id: 3"]) == ("put", '{"a":1}')
|
||||
assert parse(["data: x", "data: y"]) == ("message", "x\ny")
|
||||
|
||||
|
||||
async def _take(agen: AsyncIterator[Event], n: int) -> list[Event]:
|
||||
out = []
|
||||
async for event in agen:
|
||||
out.append(event)
|
||||
if len(out) == n:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def test_listen_snapshot_then_writes_and_reconnects() -> None:
|
||||
fake = FakeRtdb()
|
||||
fake.set("tutor/v1-e2e/requests/a", {"req": 1})
|
||||
sleeps: list[float] = []
|
||||
|
||||
async def sleep(delay: float) -> None:
|
||||
sleeps.append(delay)
|
||||
|
||||
async def go() -> list[Event]:
|
||||
db = fake.rtdb()
|
||||
stream = sse.listen(db, "requests", sleep=sleep)
|
||||
got = await _take(stream, 1)
|
||||
await db.put("requests/b/req", 2)
|
||||
got += await _take(stream, 1)
|
||||
fake.push("event: keep-alive\ndata: null\n\n")
|
||||
fake.push("event: auth_revoked\ndata: credential is no longer valid\n\n")
|
||||
got += await _take(stream, 1) # reconnected: snapshot again
|
||||
fake.push(None) # server closes: reconnect at once
|
||||
got += await _take(stream, 1)
|
||||
fake.raise_on.add(("GET", "/requests"))
|
||||
fake.push(None)
|
||||
|
||||
async def heal(delay: float) -> None:
|
||||
sleeps.append(delay)
|
||||
fake.raise_on.clear()
|
||||
|
||||
stream2 = sse.listen(db, "requests", sleep=heal)
|
||||
fake.raise_on.add(("GET", "/requests"))
|
||||
got += await _take(stream2, 1)
|
||||
fake.push('event: put\ndata: {"path": "/x", "data": 1}\n\n')
|
||||
got += await _take(stream2, 1)
|
||||
await stream.aclose()
|
||||
await stream2.aclose()
|
||||
await db.aclose()
|
||||
return got
|
||||
|
||||
got = asyncio.run(go())
|
||||
assert got[0] == Event("put", "/", {"a": {"req": 1}})
|
||||
assert got[1] == Event("put", "/b/req", 2)
|
||||
assert got[2].path == "/"
|
||||
assert got[2].data == {"a": {"req": 1}, "b": {"req": 2}}
|
||||
assert got[3].path == "/"
|
||||
assert got[4].path == "/"
|
||||
assert got[5] == Event("put", "/x", 1)
|
||||
assert sleeps == [1.0]
|
||||
|
||||
|
||||
def test_listen_backoff_grows_and_bad_json_retries() -> None:
|
||||
fake = FakeRtdb()
|
||||
fake.raise_on.add(("GET", "/n"))
|
||||
sleeps: list[float] = []
|
||||
|
||||
async def sleep(delay: float) -> None:
|
||||
sleeps.append(delay)
|
||||
if len(sleeps) == 6:
|
||||
fake.raise_on.clear()
|
||||
|
||||
async def go() -> list[Event]:
|
||||
db = fake.rtdb()
|
||||
stream = sse.listen(db, "n", sleep=sleep)
|
||||
got = await _take(stream, 1)
|
||||
fake.push("event: patch\ndata: {not json\n\n")
|
||||
got += await _take(stream, 1) # dropped, backed off, reconnected
|
||||
await stream.aclose()
|
||||
await db.aclose()
|
||||
return got
|
||||
|
||||
got = asyncio.run(go())
|
||||
assert [e.path for e in got] == ["/", "/"]
|
||||
assert sleeps == [1.0, 2.0, 5.0, 10.0, 30.0, 30.0, 1.0]
|
||||
|
||||
|
||||
def test_cancel_is_fatal() -> None:
|
||||
fake = FakeRtdb()
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
stream = sse.listen(db, "c")
|
||||
await _take(stream, 1)
|
||||
fake.push("event: cancel\ndata: permission denied\n\n")
|
||||
with pytest.raises(StreamCancelledError):
|
||||
await _take(stream, 1)
|
||||
await db.aclose()
|
||||
|
||||
asyncio.run(go())
|
||||
|
||||
|
||||
def test_mirror_applies_put_and_patch() -> None:
|
||||
tree = mirror.apply({}, Event("put", "/", {"a": {"req": 1}}))
|
||||
tree = mirror.apply(tree, Event("put", "/b/req", 2))
|
||||
tree = mirror.apply(tree, Event("patch", "/a", {"claim": {"x": 1}, "req": None}))
|
||||
assert tree == {"a": {"claim": {"x": 1}}, "b": {"req": 2}}
|
||||
tree = mirror.apply(tree, Event("put", "/a/claim/x", None))
|
||||
assert tree == {"b": {"req": 2}}
|
||||
tree = mirror.apply(tree, Event("put", "/b/req/deep", 3)) # replaces a leaf
|
||||
assert tree == {"b": {"req": {"deep": 3}}}
|
||||
tree = mirror.apply(tree, Event("patch", "/", {"c/d": 4}))
|
||||
assert tree["c"] == {"d": 4}
|
||||
tree = mirror.apply(tree, Event("patch", "/", None))
|
||||
assert mirror.apply(tree, Event("put", "/", None)) == {}
|
||||
assert mirror.first_key("/a/b") == "a"
|
||||
assert mirror.first_key("/") is None
|
||||
|
||||
|
||||
def test_blank_lines_without_a_block_are_skipped() -> None:
|
||||
async def lines() -> AsyncIterator[str]:
|
||||
for line in ("", "event: put", "data: {}", "", ""):
|
||||
yield line
|
||||
|
||||
async def go() -> list[tuple[str, str]]:
|
||||
return [b async for b in sse._blocks(lines())]
|
||||
|
||||
assert asyncio.run(go()) == [("put", "{}")]
|
||||
53
rtdb_relay/rtdb_relay/tests/test_unlock.py
Normal file
53
rtdb_relay/rtdb_relay/tests/test_unlock.py
Normal file
@ -0,0 +1,53 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""Unlocking: header made once, wrong or missing passphrase refused."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
from crdt_sync import Argon2Params, E2eLayout, E2eLockedError
|
||||
import pytest
|
||||
|
||||
from rtdb_relay.tests._fake_rtdb import FakeRtdb
|
||||
from rtdb_relay.unlock import HeaderStore, unlock
|
||||
|
||||
FAST = Argon2Params(memory_kib=8, iterations=1, parallelism=1)
|
||||
LAYOUT = E2eLayout.of("tutor/v1")
|
||||
|
||||
|
||||
def test_unlock_creates_the_header_then_refuses_a_wrong_passphrase() -> None:
|
||||
fake = FakeRtdb(sealed=False)
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
key = await unlock(db, "pw", LAYOUT, FAST)
|
||||
header = fake.raw_get(LAYOUT.e2e_path)
|
||||
assert isinstance(header, str)
|
||||
assert "argon2id" in header
|
||||
again = await unlock(db, "pw", LAYOUT, FAST)
|
||||
assert again.master == key.master
|
||||
with pytest.raises(E2eLockedError):
|
||||
await unlock(db, "other", LAYOUT, FAST)
|
||||
with pytest.raises(E2eLockedError):
|
||||
await unlock(db, "", LAYOUT, FAST)
|
||||
fake.raw_set(LAYOUT.e2e_path, {"not": "a header"})
|
||||
with pytest.raises(E2eLockedError):
|
||||
await unlock(db, "pw", LAYOUT, FAST)
|
||||
await db.aclose()
|
||||
|
||||
asyncio.run(go())
|
||||
|
||||
|
||||
def test_header_store_lists_nothing_and_never_deletes() -> None:
|
||||
fake = FakeRtdb(sealed=False)
|
||||
|
||||
async def go() -> None:
|
||||
db = fake.rtdb()
|
||||
store = HeaderStore(db.at(""), asyncio.get_running_loop())
|
||||
assert store.list_directory("x") == []
|
||||
assert store.can_access_remote()
|
||||
with pytest.raises(NotImplementedError):
|
||||
store.delete_file("x")
|
||||
await db.aclose()
|
||||
|
||||
asyncio.run(go())
|
||||
97
rtdb_relay/rtdb_relay/unlock.py
Normal file
97
rtdb_relay/rtdb_relay/unlock.py
Normal file
@ -0,0 +1,97 @@
|
||||
# Copyright (c) 2026 Krzysztof Rudnicki. MIT License.
|
||||
"""The E2E key from the shared crdt-sync passphrase (header made on first use)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from crdt_sync import (
|
||||
Argon2Params,
|
||||
E2eHeaderError,
|
||||
E2eKey,
|
||||
E2eLayout,
|
||||
E2eLockedError,
|
||||
E2eWrongPassphraseError,
|
||||
unlock_e2e,
|
||||
)
|
||||
|
||||
from rtdb_relay.seal import dumps
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Coroutine
|
||||
|
||||
from rtdb_relay.rtdb import Rtdb
|
||||
|
||||
|
||||
class HeaderStore:
|
||||
"""crdt_sync's sync ``RemoteStore``, just enough for ``unlock_e2e``.
|
||||
|
||||
``unlock_e2e`` runs in a worker thread (Argon2id must not block the
|
||||
loop); each call hops back to the loop that owns the async client.
|
||||
The header is one JSON string leaf, as the Flutter side writes it.
|
||||
"""
|
||||
|
||||
def __init__(self, raw: Rtdb, loop: asyncio.AbstractEventLoop) -> None:
|
||||
"""Read and write through ``raw`` (an unsealed client on ``""``)."""
|
||||
self._raw = raw
|
||||
self._loop = loop
|
||||
|
||||
def _run[T](self, coro: Coroutine[Any, Any, T]) -> T:
|
||||
return asyncio.run_coroutine_threadsafe(coro, self._loop).result()
|
||||
|
||||
def list_directory(self, path: str) -> list[str]:
|
||||
"""Nothing: only the header is read through here."""
|
||||
del path
|
||||
return []
|
||||
|
||||
def get_file_text(self, path: str) -> str | None:
|
||||
"""The header text, or ``None``; a non-string value comes back as JSON."""
|
||||
value = self._run(self._raw.get(path))
|
||||
if value is None or isinstance(value, str):
|
||||
return value
|
||||
return dumps(value) # not ours: E2eHeaderError, never overwritten
|
||||
|
||||
def put_file_text(self, path: str, text: str, *, message: str) -> None:
|
||||
"""Write the header as one string leaf."""
|
||||
del message
|
||||
self._run(self._raw.put(path, text))
|
||||
|
||||
def delete_file(self, path: str, *, message: str = "") -> None:
|
||||
"""Refused: the header is never deleted."""
|
||||
del path, message
|
||||
msg = "the relay never deletes the e2e header"
|
||||
raise NotImplementedError(msg)
|
||||
|
||||
def can_access_remote(self) -> bool:
|
||||
"""Always: a failing call raises on its own."""
|
||||
return True
|
||||
|
||||
|
||||
async def unlock(
|
||||
rtdb: Rtdb,
|
||||
passphrase: str,
|
||||
layout: E2eLayout,
|
||||
params: Argon2Params | None = None,
|
||||
) -> E2eKey:
|
||||
"""The key of ``layout`` (``E2eLayout.of("<app>/v1")``) from ``passphrase``.
|
||||
|
||||
Raises:
|
||||
E2eLockedError: No passphrase, a wrong one, or an unreadable header.
|
||||
|
||||
"""
|
||||
if not passphrase:
|
||||
msg = "no crdt-sync passphrase on this PC"
|
||||
raise E2eLockedError(msg)
|
||||
store = HeaderStore(rtdb.at(""), asyncio.get_running_loop())
|
||||
try:
|
||||
return await asyncio.to_thread(
|
||||
unlock_e2e,
|
||||
store,
|
||||
path_prefix=layout.prefix,
|
||||
passphrase=passphrase,
|
||||
params=params,
|
||||
e2e_path=layout.e2e_path,
|
||||
)
|
||||
except (E2eWrongPassphraseError, E2eHeaderError) as exc:
|
||||
raise E2eLockedError(str(exc) or type(exc).__name__) from exc
|
||||
Loading…
Reference in New Issue
Block a user