mirror of
https://github.com/kuhyx/utils.git
synced 2026-11-14 21:50:57 +01:00
Krzysztof Rudnicki
1c49abe8c2
Extracted and generalized from diet_guard's existing tombstone-merge scheme and GitHub Contents API client, extended with per-field HLC LWW so the same library can also cover todo's mutable notes and wake_alarm's single-value sync (Workstream D scoping decision, 2026-07-04). 100% branch coverage, clean ruff/mypy/pylint/bandit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017McuA1nAi6jBT1EaS5BrVf
101 lines
4.4 KiB
Markdown
101 lines
4.4 KiB
Markdown
# crdt-sync
|
|
|
|
Shared CRDT merge scheme + GitHub-Contents-API sync transport, extracted for
|
|
reuse across four personal apps that each need cross-device sync:
|
|
[todo](https://github.com/kuhyx/todo-app),
|
|
[diet_guard](https://github.com/kuhyx/testsAndMisc/tree/main/python_pkg/diet_guard),
|
|
[wake_alarm](https://github.com/kuhyx/testsAndMisc/tree/main/python_pkg/wake_alarm),
|
|
and [screen-locker](https://github.com/kuhyx/screen-locker).
|
|
|
|
## Why
|
|
|
|
Two of those apps (diet_guard, and independently todo) had already converged
|
|
on the same shape of solution: treat a private GitHub repo as dumb file
|
|
storage via the REST Contents API (not a git clone), have each device push
|
|
its own file, and merge on read. diet_guard's merge logic
|
|
(entry-id-keyed, tombstone-wins) and todo's merge logic (`sqlite_crdt`'s
|
|
per-field HLC last-writer-wins) solve the same underlying problem —
|
|
convergent merge without a central server — for two different data shapes
|
|
(append-only logs vs. mutable per-field records). `crdt-sync` generalizes
|
|
both into one scheme so all four apps can share it, at the cost of todo's
|
|
notes and wake_alarm's single wake-time value both running through
|
|
CRDT machinery their own data doesn't strictly need — an explicit
|
|
consistency-over-leanness tradeoff.
|
|
|
|
## The merge scheme
|
|
|
|
A generic **LWW-map-with-sticky-remove**, built from three pieces:
|
|
|
|
- **`Hlc`** — a Hybrid Logical Clock: `(wall_time_ms, counter, node_id)`,
|
|
totally ordered by tuple comparison (`node_id` breaks ties between two
|
|
clocks with identical wall time and counter, so no two distinct devices'
|
|
clocks ever compare equal). `Hlc.new_tick(node_id, previous)` advances off
|
|
wall-clock time, or bumps `counter` when the wall clock hasn't moved past
|
|
`previous` — the standard HLC construction, guaranteeing monotonicity even
|
|
across clock skew or two ticks in the same millisecond.
|
|
- **`Record`** — `id`, a `fields` map of `name -> (value, Hlc)`, and a sticky
|
|
`deleted` flag. `merge_field` keeps whichever side's `Hlc` is greater,
|
|
per field — this is what lets two devices concurrently edit *different*
|
|
fields of the same record (or the same field) and converge deterministically
|
|
without a central authority. `merge_record`'s `deleted` is **not** itself
|
|
per-field LWW: it's `a.deleted or b.deleted`, monotonic and one-directional.
|
|
This preserves a guarantee diet_guard's original tombstone scheme already
|
|
relied on — a delete can never be silently undone by merging in an older
|
|
non-deleted copy pulled from a device that hasn't seen the delete yet.
|
|
- **`Log`** = `dict[id, Record]`. `merge_logs` takes the union of ids and
|
|
`merge_record`s the ones present on both sides.
|
|
|
|
All three merges are commutative and idempotent (provable directly from
|
|
`max`-by-`Hlc` and boolean-`or`), so pull order between devices never matters
|
|
and a repeated sync tick is a no-op — see `crdt_sync/tests/test_log.py`'s
|
|
property-style cases for the concrete proof.
|
|
|
|
**Per-app adapters are out of scope for this library.** Translating a `Note`,
|
|
a `FoodEntry`, a workout entry, or wake_alarm's `{hour, minute}` value to and
|
|
from `Record` is each app's own job — keeping this library domain-agnostic is
|
|
what makes it actually shared, rather than shaped around one app's schema.
|
|
|
|
## Install
|
|
|
|
```bash
|
|
pip install "crdt-sync @ git+https://github.com/kuhyx/crdt-sync@v0.1.0"
|
|
```
|
|
|
|
## Usage
|
|
|
|
```python
|
|
from crdt_sync import GitHubSyncClient, Hlc, Record, sync_log
|
|
|
|
node_id = "pc"
|
|
clock = Hlc.new_tick(node_id)
|
|
record = Record(id="abc123", fields={"text": ("buy milk", clock)}, deleted=False)
|
|
|
|
client = GitHubSyncClient(owner="kuhyx", repo="my-app-sync", token=token)
|
|
result = sync_log(
|
|
client=client,
|
|
device_id=node_id,
|
|
path_prefix="devices",
|
|
local_log={record.id: record},
|
|
encode=lambda log: json.dumps({k: r.to_dict() for k, r in log.items()}),
|
|
decode=lambda text: {
|
|
k: Record.from_dict(v) for k, v in json.loads(text).items()
|
|
},
|
|
)
|
|
```
|
|
|
|
`sync_log` pulls every other device's last-pushed log from
|
|
`<path_prefix>/<other-device-id>/...`, merges each into the local log with
|
|
`merge_logs`, then pushes this device's own merged result back up — the same
|
|
pull-all/merge/push-own pattern diet_guard's `_sync.py` already used, made
|
|
domain-agnostic via the `encode`/`decode` callbacks so callers keep their own
|
|
on-disk JSON shape instead of `crdt-sync` dictating one.
|
|
|
|
## Development
|
|
|
|
```bash
|
|
python -m venv .venv && source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
pytest
|
|
pre-commit install && pre-commit install --hook-type pre-push
|
|
```
|