mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 18:07:06 +01:00
Remove SyncYomi and TachiyomiSY traces
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
ee78ae44cc
commit
57a070e7d9
@ -73,10 +73,6 @@ copies were the only ones on the machine.
|
||||
detached worktree at HEAD. Live-system state, not a regression.
|
||||
- **`workout_locker` reports error** in `check_and_enable_services.sh`. That is
|
||||
the separate `screen-locker` repo.
|
||||
- **`syncyomi-guard` is red** since 00:00 on 2026-08-23, working as designed:
|
||||
the library collapsed from 2185 manga to 8. See the
|
||||
`syncyomi-restore-silent-partial` memory; do NOT checkpoint the WAL before
|
||||
recovering. Newest snapshot: `~/services/syncyomi/snapshots/`, 2026-08-15.
|
||||
|
||||
## Traps worth keeping
|
||||
|
||||
|
||||
@ -45,7 +45,7 @@ which your own hook (`track_reads_pretool.sh`, since disabled) was already tryin
|
||||
### Docker MCP Gateway (mcp, sandbox, cli, 1.5k★)
|
||||
|
||||
You run 23+ Dockerfiles for self-hosted MCP server containers (`~/src/mcp-servers/servers/*`) per the repo survey, on top
|
||||
of an already-Docker-heavy self-hosted stack (Gitea, SearXNG, Syncyomi, Joplin Server, all `docker-compose.yml`-based).
|
||||
of an already-Docker-heavy self-hosted stack (Gitea, SearXNG, Joplin Server, all `docker-compose.yml`-based).
|
||||
A container-aware MCP gateway is a direct fit for consolidating discovery/auth across a fleet this size, rather than
|
||||
each server being wired in ad hoc. Speculative on the _specific_ tool, high-confidence on the _need_ — you already
|
||||
have the infrastructure pattern this tool assumes.
|
||||
@ -159,7 +159,7 @@ with the browser tooling you already have that a swap would fix.
|
||||
|
||||
### OpenClaw, Khoj, nanobot, CowAgent (self-hosted "personal agent" runtimes)
|
||||
|
||||
Surface-level strong fit — you unambiguously self-host (Gitea, SearXNG, Syncyomi, Joplin Server + DuckDNS, Ollama on
|
||||
Surface-level strong fit — you unambiguously self-host (Gitea, SearXNG, Joplin Server + DuckDNS, Ollama on
|
||||
an RTX 3090) and run 39 systemd user units as personal automation. But these products are chat/webhook-fronted
|
||||
always-on daemons (Telegram/Discord/WhatsApp bots, "second brain" Q&A) — a different shape of automation than what you
|
||||
actually build, which is narrow, single-purpose guard daemons with typed configs and 100%-coverage test suites
|
||||
|
||||
@ -27,9 +27,6 @@ its timer next fires.
|
||||
/home/kuhy/.config/systemd/user/morning-routine.service
|
||||
13:Environment=PYTHONPATH=%h/src/testsAndMisc
|
||||
16:WorkingDirectory=%h/src/testsAndMisc
|
||||
/home/kuhy/.config/systemd/user/syncyomi-guard.service
|
||||
8:WorkingDirectory=/home/kuhy/src/testsAndMisc
|
||||
9:Environment=PYTHONPATH=/home/kuhy/src/testsAndMisc
|
||||
|
||||
## /etc references
|
||||
/etc/guard-lib/targets/nsswitch.conf
|
||||
|
||||
@ -1,196 +0,0 @@
|
||||
# SyncYomi / TachiyomiSY restore runbook
|
||||
|
||||
Written after the 2026-08-09 incident, in which a restore that reported success
|
||||
replaced a 2182-manga library with a 267 KB stub.
|
||||
|
||||
## The bug
|
||||
|
||||
TachiyomiSY **1.13.2** (current release as of 2026-08-09) fails a backup restore
|
||||
with:
|
||||
|
||||
```
|
||||
android.database.SQLException: Error code: 5, message: database is locked
|
||||
```
|
||||
|
||||
The locked database is **the app's own `data.db` on the phone**, not the
|
||||
SyncYomi server. The stack runs through Room's `InvalidationTracker` — UI
|
||||
screens hold observer queries while the restore wants an exclusive write, and
|
||||
the bundled SQLite returns `SQLITE_BUSY` instead of waiting.
|
||||
|
||||
Upstream, open and unfixed: [#1634] and [#1638].
|
||||
|
||||
[#1634]: https://github.com/jobobby04/TachiyomiSY/issues/1634
|
||||
[#1638]: https://github.com/jobobby04/TachiyomiSY/issues/1638
|
||||
|
||||
## Fixed on this phone (2026-08-28)
|
||||
|
||||
The phone no longer runs the upstream APK. It runs
|
||||
[kuhyx/TachiyomiSY](https://github.com/kuhyx/TachiyomiSY) (`~/vendor/tachiyomisy`,
|
||||
commit "Serialize backup restore and refuse to sync a collapsed library"),
|
||||
which carries three changes upstream does not have:
|
||||
|
||||
- `BackupRestorer` awaits the six restorers in order instead of `launch`ing
|
||||
them into one scope, so they stop contending for SQLite's single writer.
|
||||
That is the code-5 bug below.
|
||||
- `SyncService.mergeSyncData` throws `SyncCollapseException` when more than
|
||||
10% of the server's entries are missing locally. A row that is simply gone
|
||||
is a damaged database, not a deletion -- deleting in the app keeps the row
|
||||
and clears `favorite`.
|
||||
- `SyncYomiSyncService` additionally compares the outgoing payload against the
|
||||
entry count this device last pushed, which covers the 304 path where there
|
||||
is no remote data to merge against.
|
||||
|
||||
Both guards fail closed: the sync aborts and the message reaches the user
|
||||
through `SyncDataJob`'s existing error notification. Verified on the phone on
|
||||
2026-08-28 against a real 2163-entry payload -- guard A refused at a faked
|
||||
baseline of 99999, guard B refused with 1300 of 2163 entries removed, and the
|
||||
server row was byte-identical afterwards in both runs.
|
||||
|
||||
The build is `eu.kanade.tachiyomi.sy` version `1.13.2-kuhy`, i.e. it replaces
|
||||
the official APK rather than sitting beside it, and is signed with
|
||||
`~/.android/release/key.properties`. Rebuild and upgrade in place with:
|
||||
|
||||
```bash
|
||||
JAVA_HOME=/usr/lib/jvm/java-21-openjdk \
|
||||
~/vendor/tachiyomisy/gradlew -p ~/vendor/tachiyomisy assembleFoss -PsyReplaceUpstream
|
||||
adb install -r ~/vendor/tachiyomisy/app/build/outputs/apk/foss/app-arm64-v8a-foss.apk
|
||||
```
|
||||
|
||||
Never install an official TachiyomiSY APK over it: the signatures differ, so
|
||||
the install fails, and taking the official build back means an uninstall and a
|
||||
restore. The fork's `foss` build type has the in-app updater compiled out for
|
||||
exactly that reason.
|
||||
|
||||
Rebasing on a new upstream release means replaying those commits; the fork is
|
||||
not upstreamed.
|
||||
|
||||
## Why it is dangerous
|
||||
|
||||
The loud failure is survivable. The dangerous case is the quiet one:
|
||||
|
||||
1. A restore fails with code 5. The library is now empty or partial.
|
||||
2. A retry reports `Worker result SUCCESS` — but writes almost nothing.
|
||||
3. SyncYomi pushes that near-empty library to the server.
|
||||
4. The good server copy is gone.
|
||||
|
||||
On 2026-08-09 the whole sequence took 51 seconds, and the real library survived
|
||||
only as stale pages in the server's SQLite write-ahead log.
|
||||
|
||||
**A success toast is not evidence.** Verify by entry count, always.
|
||||
|
||||
## Restoring safely
|
||||
|
||||
1. **Turn sync OFF in SY first.** Settings → Data and storage → Sync.
|
||||
Skipping this lets a background sync overwrite the server mid-recovery.
|
||||
2. **Force-stop the app** — `adb shell am force-stop eu.kanade.tachiyomi.sy`,
|
||||
or Settings → Apps → Force stop.
|
||||
3. **Cold-start straight into the restore.** Do not open the library, do not
|
||||
let it sit on a tab loading covers. Every open screen is another observer
|
||||
holding a connection, which is what the restore loses the race to.
|
||||
4. **Put the phone down while it runs.** Navigating during a restore adds
|
||||
exactly the contention that triggers the bug.
|
||||
5. **Verify by count**, not by the toast. Compare against the guard's baseline:
|
||||
`cat ~/.local/share/syncyomi_guard/last_known_good.json`
|
||||
6. **Only then re-enable sync**, so the verified library propagates up.
|
||||
|
||||
If it fails, force-stop and retry from step 3. Upstream reporters needed up to
|
||||
five attempts. Retrying is expected, not a sign of a new problem.
|
||||
|
||||
If it will not restore at all, the upstream workaround is a downgrade:
|
||||
install 1.12.0 → restore → upgrade back to 1.13.2. This wipes `data.db`, so do
|
||||
it **only** with a verified-good `.tachibk` in hand.
|
||||
|
||||
## Known collateral: categories
|
||||
|
||||
A restore that recovers every manga can still drop **all categories and all
|
||||
category memberships**. Observed 2026-08-09: 2182/2182 manga and 57 389/57 389
|
||||
distinct chapters restored, while 18 categories and 1724 manga-to-category
|
||||
assignments were lost. Matches the upstream report in [#1634].
|
||||
|
||||
Categories are not recoverable by re-syncing — the server copy is whatever the
|
||||
phone last pushed. They come back only from a `.tachibk` that still has them.
|
||||
|
||||
## Chapter counts shrink legitimately
|
||||
|
||||
A restore **deduplicates** chapter entries. On 2026-08-09 the total fell from
|
||||
75 376 to 68 418 (−9.2 %) while the distinct chapter count was unchanged at
|
||||
57 389: 6958 duplicates were collapsed, and nothing was lost.
|
||||
|
||||
Do not read a falling chapter count as data loss without comparing _distinct_
|
||||
chapter URLs. The guard's 10 % threshold is set above this specific event for
|
||||
exactly that reason.
|
||||
|
||||
## The guard
|
||||
|
||||
`python_pkg/syncyomi_guard/` checks the server payload every 30 minutes, fails
|
||||
loudly on a collapse, and keeps 14 restorable `.tachibk` snapshots.
|
||||
|
||||
```bash
|
||||
# check now
|
||||
python3 -m python_pkg.syncyomi_guard # exit 0 healthy, 1 collapsed
|
||||
|
||||
# install / remove the timer
|
||||
./linux_configuration/features/setup_syncyomi_guard.sh
|
||||
./linux_configuration/features/setup_syncyomi_guard.sh --uninstall
|
||||
|
||||
systemctl --user list-timers syncyomi-guard.timer
|
||||
journalctl --user -u syncyomi-guard.service -n 20
|
||||
```
|
||||
|
||||
Snapshots live in `~/services/syncyomi/snapshots/` and are ordinary `.tachibk` files —
|
||||
`adb push` one to the phone and restore it directly.
|
||||
|
||||
**Retention: 14 snapshots at 30-minute intervals is about 7 hours of history.**
|
||||
That covers a fast incident — the 2026-08-09 window was 51 seconds — but not a
|
||||
slow drift noticed days later. For that, the reference copy is
|
||||
`~/services/syncyomi/recovery/`, which is never pruned.
|
||||
|
||||
After a **deliberate** library purge the guard will report a collapse; accept
|
||||
the new state with `--accept` to re-baseline.
|
||||
|
||||
### Outcome of the 2026-08-09 incident
|
||||
|
||||
Manga and chapters were fully recovered. **The 18 categories were not, and the
|
||||
loss was accepted deliberately** — the baseline now records 1 category, so the
|
||||
guard runs green against reality.
|
||||
|
||||
Three attempts failed to get categories back onto the phone:
|
||||
|
||||
1. Restore the recovered `.tachibk` → app briefly showed categories, but the
|
||||
next sync pushed **1** category up.
|
||||
2. `PUT` the full 18-category payload straight to the server → server held 18,
|
||||
phone synced and flattened it back to 1.
|
||||
3. Repeat of 2 with the same result.
|
||||
|
||||
Round 2 is the informative one: the server had **18** and the phone had **1**,
|
||||
and 1 won. The phone was not losing a merge — it was honestly reporting what it
|
||||
had. The restore writes manga and chapters but silently drops the category
|
||||
tables, which is the narrow form of the same 1.13.2 bug.
|
||||
|
||||
The categories still exist in `~/services/syncyomi/recovery/syncyomi_recovered_2026-08-09.tachibk`
|
||||
(18 definitions, 1724 memberships). Getting them into the app needs the
|
||||
downgrade path — 1.12.0 → restore → verify → upgrade — which was not attempted.
|
||||
If a future SY release fixes #1634, retry the restore before the downgrade.
|
||||
|
||||
## Recovering from the server WAL (last resort)
|
||||
|
||||
If the server payload is already degraded and no snapshot predates it, the old
|
||||
payload may still exist as stale pages in `syncyomi.db-wal`.
|
||||
|
||||
**Do not restart the container and do not checkpoint** — a clean close deletes
|
||||
the WAL and the only remaining copy with it. Snapshot all three files first:
|
||||
|
||||
```bash
|
||||
sudo cp -a ~/services/syncyomi/config/syncyomi.db{,-shm,-wal} /some/safe/dir/
|
||||
```
|
||||
|
||||
Then rebuild by applying every WAL frame over the base database, last writer
|
||||
wins, and extract `sync_data.data`. The 2026-08-09 reconstruction and its
|
||||
verification are recorded in
|
||||
`docs/superpowers/evidence/syncyomi-library-recovery-2026-08-09.json`.
|
||||
|
||||
**`~/services/syncyomi/recovery/recovered.db` is a forensic artifact, not a database.**
|
||||
It was assembled by splicing pages from several checkpoint generations, so it is
|
||||
not any single committed SQLite state — `quick_check` passes and the payload
|
||||
blob decodes, but do **not** copy it over `config/syncyomi.db`. The deliverable
|
||||
from that recovery is the `.tachibk` beside it.
|
||||
@ -19,7 +19,7 @@
|
||||
<p>
|
||||
This home server hosts several things at once: a self-hosted
|
||||
<strong>Gitea</strong> (a GitHub-like git host), a <strong>dufs</strong>
|
||||
file cloud, a <strong>SyncYomi</strong> manga-sync service — and now a
|
||||
file cloud — and now a
|
||||
personal website. They all share a single public IP and a single pair of
|
||||
open ports (<code>80</code>/<code>443</code>).
|
||||
</p>
|
||||
@ -44,7 +44,6 @@
|
||||
<div class="backcol">
|
||||
<div class="back">gitea → <code>127.0.0.1:3000</code></div>
|
||||
<div class="back">dufs → <code>127.0.0.1:5000</code></div>
|
||||
<div class="back">syncyomi → <code>127.0.0.1:8282</code></div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@ -56,7 +55,6 @@
|
||||
|
||||
<div class="code"><pre>kuhy.duckdns.org { reverse_proxy 127.0.0.1:3000 } # Gitea (the root!)
|
||||
kuhy-cloud.duckdns.org{ reverse_proxy 127.0.0.1:5000 } # dufs
|
||||
kuhy-sync.duckdns.org { reverse_proxy 127.0.0.1:8282 } # SyncYomi</pre></div>
|
||||
|
||||
<p>
|
||||
Two details matter for this change. First, the bare root
|
||||
@ -71,7 +69,7 @@ kuhy-sync.duckdns.org { reverse_proxy 127.0.0.1:8282 } # SyncYomi</pre></div>
|
||||
|
||||
<p>
|
||||
Finally, that Caddyfile isn't hand-written — it's <em>generated</em> by
|
||||
<code>setup_gitea.sh</code>. SyncYomi and dufs each <em>appended</em> their
|
||||
<code>setup_gitea.sh</code>. dufs <em>appended</em> its
|
||||
block to the same file from their own setup scripts. Hold that thought.
|
||||
</p>
|
||||
</section>
|
||||
@ -84,8 +82,8 @@ kuhy-sync.duckdns.org { reverse_proxy 127.0.0.1:8282 } # SyncYomi</pre></div>
|
||||
<code>setup_gitea.sh</code> is meant to be safe to re-run. But its way of
|
||||
writing the Caddyfile was <code>cat > Caddyfile</code> — it
|
||||
<strong>regenerated the whole file from scratch, containing only Gitea's
|
||||
block.</strong> dufs and SyncYomi had appended <em>their</em> blocks later.
|
||||
So every time you re-ran the Gitea setup, those blocks silently vanished and
|
||||
block.</strong> dufs had appended <em>its</em> block later.
|
||||
So every time you re-ran the Gitea setup, that block silently vanished and
|
||||
two live services went dark. A latent trap, waiting for the next re-run.
|
||||
</p>
|
||||
|
||||
@ -108,7 +106,7 @@ kuhy-sync.duckdns.org { reverse_proxy 127.0.0.1:8282 } # SyncYomi</pre></div>
|
||||
<button id="mw-rerun">▶ Re-run setup_gitea.sh</button>
|
||||
<button id="mw-reset" class="secondary">↺ Reset</button>
|
||||
</div>
|
||||
<p class="mw-log" id="mw-log">All four services are up. Now re-run the Gitea setup…</p>
|
||||
<p class="mw-log" id="mw-log">All three services are up. Now re-run the Gitea setup…</p>
|
||||
</div>
|
||||
|
||||
<h3>Problem B — two Caddies, one admin port</h3>
|
||||
@ -181,10 +179,10 @@ kuhy-sync.duckdns.org { reverse_proxy 127.0.0.1:8282 } # SyncYomi</pre></div>
|
||||
<code>setup_personal_website.sh</code> orchestrates the whole cutover. The
|
||||
<strong>order</strong> is the safety property: existing blocks are migrated
|
||||
into snippets <em>before</em> the root file is switched to the import line —
|
||||
otherwise dufs and SyncYomi would blink out during the migration itself.
|
||||
otherwise dufs would blink out during the migration itself.
|
||||
</p>
|
||||
<div class="code"><pre>preflight # deps, docker up, gitea-caddy present
|
||||
migrate_caddy_snippets # split TODAY's Caddyfile → sites/*.caddy (keep dufs, sync)
|
||||
migrate_caddy_snippets # split TODAY's Caddyfile → sites/*.caddy (keep dufs)
|
||||
relocate_gitea # re-run setup_gitea.sh (now writes import + gitea.caddy)
|
||||
build_website # pnpm build → ~/src/personal-website/dist
|
||||
write_website_stack # loopback caddy compose + inner Caddyfile
|
||||
@ -336,7 +334,6 @@ ensure_firewall # 80/443 already open — idempotent</pre></div>
|
||||
const SERVICES = [
|
||||
{ key: "gitea", name: "Gitea", host: "gitea.kuhy…:3000" },
|
||||
{ key: "dufs", name: "dufs", host: "kuhy-cloud…:5000" },
|
||||
{ key: "sync", name: "SyncYomi", host: "kuhy-sync…:8282" },
|
||||
{ key: "web", name: "Website", host: "kuhy…:8088" },
|
||||
];
|
||||
let mode = "mono";
|
||||
@ -347,7 +344,7 @@ ensure_firewall # 80/443 already open — idempotent</pre></div>
|
||||
function resetState() {
|
||||
SERVICES.forEach((s) => (state[s.key] = true));
|
||||
render();
|
||||
elLog.textContent = "All four services are up. Now re-run the Gitea setup…";
|
||||
elLog.textContent = "All three services are up. Now re-run the Gitea setup…";
|
||||
}
|
||||
function render() {
|
||||
elServices.innerHTML = "";
|
||||
@ -368,7 +365,7 @@ ensure_firewall # 80/443 already open — idempotent</pre></div>
|
||||
SERVICES.forEach((s) => (state[s.key] = s.key === "gitea"));
|
||||
render();
|
||||
elLog.innerHTML =
|
||||
"<strong style='color:var(--bad)'>Clobbered.</strong> The whole Caddyfile was rewritten with only Gitea's block — dufs, SyncYomi and the website disappeared.";
|
||||
"<strong style='color:var(--bad)'>Clobbered.</strong> The whole Caddyfile was rewritten with only Gitea's block — dufs and the website disappeared.";
|
||||
} else {
|
||||
// Only touches gitea.caddy; imports leave the rest intact.
|
||||
SERVICES.forEach((s) => (state[s.key] = true));
|
||||
@ -392,7 +389,7 @@ ensure_firewall # 80/443 already open — idempotent</pre></div>
|
||||
// ---- Quiz (debiased: correct never the longest; positions spread) ----
|
||||
const QUIZ = [
|
||||
{
|
||||
stem: "Before this change, why did re-running setup_gitea.sh drop the SyncYomi and dufs blocks?",
|
||||
stem: "Before this change, why did re-running setup_gitea.sh drop the dufs block?",
|
||||
options: [
|
||||
"Docker restarted the Caddy container, which cleared its in-memory config.",
|
||||
"It rewrote the entire Caddyfile from scratch, keeping only Gitea's block.",
|
||||
@ -433,7 +430,7 @@ ensure_firewall # 80/443 already open — idempotent</pre></div>
|
||||
"The DuckDNS wildcard has to resolve before Caddy can request certificates.",
|
||||
],
|
||||
correct: 2,
|
||||
why: "The migration must copy dufs/SyncYomi out of the old monolith before it's overwritten.",
|
||||
why: "The migration must copy dufs out of the old monolith before it's overwritten.",
|
||||
},
|
||||
{
|
||||
stem: "Why is the site served by a separate loopback container fronted by gitea-caddy, not binding 80/443 itself?",
|
||||
|
||||
@ -1,402 +0,0 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Adding SyncYomi to a one-edge home server</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #ffffff; --fg: #1b1f24; --muted: #5b6470; --card: #f6f8fa;
|
||||
--border: #d8dee4; --accent: #4f46e5; --accent-soft: #eef0ff;
|
||||
--ok: #1a7f37; --ok-soft: #e6f4ea; --bad: #b42318; --bad-soft: #fdecea;
|
||||
--code: #0b1020; --code-fg: #e6e6e6; --pill: #eaeef2;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
:root {
|
||||
--bg: #0d1117; --fg: #e6edf3; --muted: #9aa5b1; --card: #161b22;
|
||||
--border: #2a323c; --accent: #a5b4fc; --accent-soft: #1c2030;
|
||||
--ok: #4ac776; --ok-soft: #12261a; --bad: #ff7b72; --bad-soft: #2a1615;
|
||||
--code: #010409; --code-fg: #e6edf3; --pill: #1c2430;
|
||||
}
|
||||
}
|
||||
:root[data-theme="dark"] {
|
||||
--bg: #0d1117; --fg: #e6edf3; --muted: #9aa5b1; --card: #161b22;
|
||||
--border: #2a323c; --accent: #a5b4fc; --accent-soft: #1c2030;
|
||||
--ok: #4ac776; --ok-soft: #12261a; --bad: #ff7b72; --bad-soft: #2a1615;
|
||||
--code: #010409; --code-fg: #e6edf3; --pill: #1c2430;
|
||||
}
|
||||
:root[data-theme="light"] {
|
||||
--bg: #ffffff; --fg: #1b1f24; --muted: #5b6470; --card: #f6f8fa;
|
||||
--border: #d8dee4; --accent: #4f46e5; --accent-soft: #eef0ff;
|
||||
--ok: #1a7f37; --ok-soft: #e6f4ea; --bad: #b42318; --bad-soft: #fdecea;
|
||||
--code: #0b1020; --code-fg: #e6e6e6; --pill: #eaeef2;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body {
|
||||
margin: 0; background: var(--bg); color: var(--fg);
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
|
||||
line-height: 1.65; font-size: 17px;
|
||||
}
|
||||
.wrap { max-width: min(48rem, 92vw); margin: 0 auto; padding: 2.2rem 0 5rem; }
|
||||
h1 { font-size: 1.85rem; line-height: 1.2; margin: 0 0 .3rem; }
|
||||
h2 { font-size: 1.35rem; margin: 2.6rem 0 .7rem; padding-top: .4rem; }
|
||||
h3 { font-size: 1.05rem; margin: 1.6rem 0 .4rem; }
|
||||
p { margin: .7rem 0; }
|
||||
a { color: var(--accent); }
|
||||
.sub { color: var(--muted); font-size: 1.02rem; margin-top: 0; }
|
||||
code { font-family: "SF Mono", ui-monospace, Menlo, Consolas, monospace; font-size: .88em;
|
||||
background: var(--pill); padding: .1em .38em; border-radius: 5px; }
|
||||
pre { background: var(--code); color: var(--code-fg); border-radius: 10px;
|
||||
padding: 1rem 1.1rem; overflow-x: auto; white-space: pre-wrap; word-break: break-word;
|
||||
font-size: .82rem; line-height: 1.55; border: 1px solid var(--border); }
|
||||
pre code { background: none; padding: 0; font-size: inherit; color: inherit; }
|
||||
.toc { background: var(--card); border: 1px solid var(--border); border-radius: 12px;
|
||||
padding: 1rem 1.2rem; margin: 1.6rem 0; }
|
||||
.toc ol { margin: .3rem 0 0; padding-left: 1.2rem; }
|
||||
.toc li { margin: .2rem 0; }
|
||||
.card { background: var(--card); border: 1px solid var(--border); border-radius: 12px;
|
||||
padding: 1.1rem 1.2rem; margin: 1.1rem 0; }
|
||||
.note { border-left: 4px solid var(--accent); background: var(--accent-soft);
|
||||
border-radius: 0 10px 10px 0; padding: .7rem 1rem; margin: 1.1rem 0; }
|
||||
.pill { display: inline-block; background: var(--pill); border: 1px solid var(--border);
|
||||
border-radius: 999px; padding: .12em .7em; font-size: .8rem; color: var(--muted);
|
||||
margin: .15rem .2rem .15rem 0; font-family: ui-monospace, monospace; }
|
||||
/* diagram */
|
||||
.diagram { display: flex; flex-direction: column; gap: .55rem; align-items: stretch;
|
||||
margin: 1rem 0; }
|
||||
.row { display: flex; align-items: center; gap: .6rem; flex-wrap: wrap; }
|
||||
.box { border: 1px solid var(--border); border-radius: 10px; padding: .55rem .8rem;
|
||||
background: var(--bg); font-size: .86rem; }
|
||||
.box b { display: block; font-size: .8rem; }
|
||||
.box small { color: var(--muted); font-family: ui-monospace, monospace; }
|
||||
.edge { border-color: var(--accent); background: var(--accent-soft); }
|
||||
.new { border-color: var(--ok); background: var(--ok-soft); }
|
||||
.arrow { color: var(--muted); font-weight: 700; }
|
||||
/* router widget */
|
||||
.router { background: var(--card); border: 1px solid var(--border); border-radius: 12px;
|
||||
padding: 1.1rem 1.2rem; margin: 1.2rem 0; }
|
||||
.router label { font-size: .85rem; color: var(--muted); display: block; margin-bottom: .3rem; }
|
||||
.router select, .router button { font-size: .95rem; padding: .5rem .7rem; border-radius: 8px;
|
||||
border: 1px solid var(--border); background: var(--bg); color: var(--fg); }
|
||||
.router button { background: var(--accent); color: #fff; border: none; cursor: pointer; font-weight: 600; }
|
||||
.router button:hover { filter: brightness(1.08); }
|
||||
.trace { margin-top: 1rem; display: flex; flex-direction: column; gap: .4rem; }
|
||||
.step { opacity: 0; transform: translateY(4px); transition: opacity .35s, transform .35s;
|
||||
border: 1px solid var(--border); border-radius: 9px; padding: .5rem .75rem; font-size: .87rem;
|
||||
background: var(--bg); }
|
||||
.step.show { opacity: 1; transform: none; }
|
||||
.step.hit { border-color: var(--ok); background: var(--ok-soft); }
|
||||
.step.miss { border-color: var(--bad); background: var(--bad-soft); }
|
||||
.step small { color: var(--muted); font-family: ui-monospace, monospace; }
|
||||
/* quiz */
|
||||
.q { border: 1px solid var(--border); border-radius: 12px; padding: 1rem 1.2rem; margin: 1rem 0;
|
||||
background: var(--card); }
|
||||
.q .stem { font-weight: 600; margin-bottom: .6rem; }
|
||||
.opt { display: block; width: 100%; text-align: left; border: 1px solid var(--border);
|
||||
background: var(--bg); color: var(--fg); border-radius: 9px; padding: .6rem .8rem;
|
||||
margin: .4rem 0; cursor: pointer; font-size: .93rem; line-height: 1.4; }
|
||||
.opt:hover { border-color: var(--accent); }
|
||||
.opt.correct { border-color: var(--ok); background: var(--ok-soft); }
|
||||
.opt.wrong { border-color: var(--bad); background: var(--bad-soft); }
|
||||
.opt:disabled { cursor: default; }
|
||||
.why { margin-top: .5rem; font-size: .87rem; color: var(--muted); display: none; }
|
||||
.why.show { display: block; }
|
||||
.score { position: sticky; bottom: 0; background: var(--bg); border-top: 1px solid var(--border);
|
||||
padding: .6rem 0; font-weight: 600; }
|
||||
.foot { color: var(--muted); font-size: .85rem; margin-top: 3rem; border-top: 1px solid var(--border);
|
||||
padding-top: 1rem; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
|
||||
<h1>Adding SyncYomi to a one-edge home server</h1>
|
||||
<p class="sub">How <code>setup_syncyomi.sh</code> stands up a manga-sync server that a phone can reach over the public internet — by slotting into infrastructure that already existed, rather than building a new stack.</p>
|
||||
|
||||
<div class="toc">
|
||||
<strong>Contents</strong>
|
||||
<ol>
|
||||
<li><a href="#background">Background — the pieces already on the box</a></li>
|
||||
<li><a href="#intuition">Intuition — one door, three rooms</a></li>
|
||||
<li><a href="#code">Code — a walkthrough by concern</a></li>
|
||||
<li><a href="#quiz">Quiz — check your understanding</a></li>
|
||||
</ol>
|
||||
</div>
|
||||
|
||||
<h2 id="background">1 · Background</h2>
|
||||
|
||||
<p><strong>SyncYomi</strong> is a small self-hosted server. The manga reader app <strong>TachiyomiSY</strong> on a phone talks to it to keep its library and reading progress in sync across devices — the phone uploads a blob of its library, the server stores it, and other devices download it. The phone authenticates with a single <em>API key</em>, and it needs to reach the server from anywhere, including mobile data.</p>
|
||||
|
||||
<p>The interesting part isn't SyncYomi itself — it's that this PC was <em>already</em> a public server for two other things, and the new script had to fit into that without disturbing them. Three existing facts shaped every decision:</p>
|
||||
|
||||
<div class="card">
|
||||
<h3>a. There is exactly one front door</h3>
|
||||
<p>A single <strong>Caddy</strong> container (named <code>gitea-caddy</code>) owns the public ports <code>80</code> and <code>443</code>. It terminates TLS and forwards each request to a small backend that listens only on <code>127.0.0.1</code> (loopback — unreachable from the network). Two tenants already live behind it:</p>
|
||||
<p>
|
||||
<span class="pill">kuhy.duckdns.org → 127.0.0.1:3000 (Gitea)</span>
|
||||
<span class="pill">kuhy-cloud.duckdns.org → 127.0.0.1:5000 (dufs)</span>
|
||||
</p>
|
||||
<p>Caddy tells them apart by the <em>hostname</em> in the request. It also fetches and renews a free Let's Encrypt TLS certificate for each hostname automatically.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>b. DuckDNS gives every service the same public IP</h3>
|
||||
<p>The home connection has one changing public IP. A cron job (<code>~/.joplin-server/duckdns-update.sh</code>) tells DuckDNS "these labels all point at my current IP". Adding a service means adding its label to that one updater — not creating a second updater.</p>
|
||||
</div>
|
||||
|
||||
<div class="card">
|
||||
<h3>c. The firewall silently kills bridged containers</h3>
|
||||
<p>This host's <code>nftables</code> has a default-drop <em>FORWARD</em> chain. A normal Docker bridge network routes through FORWARD, so a bridged container's traffic gets dropped with no error. The existing services dodge this by using <code>network_mode: host</code> and binding to <code>127.0.0.1</code>. The new service must do the same.</p>
|
||||
</div>
|
||||
|
||||
<p>So the change is not "install a server." It's "add a third tenant to a shared, hostname-routed edge, reusing its TLS, its DNS updater, and its firewall posture."</p>
|
||||
|
||||
<h2 id="intuition">2 · Intuition — one door, three rooms</h2>
|
||||
|
||||
<p>Picture the whole public surface as a single reception desk. Everything from the outside world arrives at that one desk (port 443 on <code>gitea-caddy</code>). The desk reads the name on the envelope — the hostname — and walks it to the right internal room. The rooms themselves have no external doors at all.</p>
|
||||
|
||||
<div class="diagram">
|
||||
<div class="row">
|
||||
<div class="box"><b>📱 Phone</b><small>https://kuhy-sync.duckdns.org</small></div>
|
||||
<span class="arrow">→</span>
|
||||
<div class="box edge"><b>gitea-caddy</b><small>:443 · TLS · host-net</small></div>
|
||||
</div>
|
||||
<div class="row" style="padding-left:1rem">
|
||||
<span class="arrow">routes by hostname ↓</span>
|
||||
</div>
|
||||
<div class="row" style="padding-left:1rem">
|
||||
<div class="box"><b>Gitea</b><small>127.0.0.1:3000</small></div>
|
||||
<div class="box"><b>dufs</b><small>127.0.0.1:5000</small></div>
|
||||
<div class="box new"><b>SyncYomi ⭐ new</b><small>127.0.0.1:8282</small></div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<p>Adding SyncYomi is therefore four small edits, not a new stack:</p>
|
||||
<ul>
|
||||
<li><strong>A new room:</strong> a SyncYomi container on <code>127.0.0.1:8282</code>.</li>
|
||||
<li><strong>A new name plate at the desk:</strong> one <code>kuhy-sync.duckdns.org { … }</code> block appended to the shared Caddyfile.</li>
|
||||
<li><strong>A new label on the DNS updater</strong> so the name resolves to home.</li>
|
||||
<li><strong>An API key,</strong> minted automatically so you can hand it to the phone.</li>
|
||||
</ul>
|
||||
|
||||
<div class="note">
|
||||
<strong>Why loopback matters.</strong> If SyncYomi listened on <code>0.0.0.0</code>, it would have an external door of its own — bypassing the TLS desk entirely and serving your library in plaintext. Binding <code>127.0.0.1</code> forces every request to go through Caddy. SyncYomi actually defaults to <code>0.0.0.0</code>, so the script <em>rewrites</em> that after first launch.</p>
|
||||
</div>
|
||||
|
||||
<h3>Try it: how the desk routes a request</h3>
|
||||
<p>Pick a hostname and watch Caddy decide where it goes. This mirrors the real site blocks in <code>~/services/gitea/Caddyfile</code>.</p>
|
||||
|
||||
<div class="router">
|
||||
<label for="host">Incoming request hostname</label>
|
||||
<div class="row">
|
||||
<select id="host">
|
||||
<option value="kuhy-sync.duckdns.org">kuhy-sync.duckdns.org</option>
|
||||
<option value="kuhy.duckdns.org">kuhy.duckdns.org</option>
|
||||
<option value="kuhy-cloud.duckdns.org">kuhy-cloud.duckdns.org</option>
|
||||
<option value="kuhy-unknown.duckdns.org">kuhy-unknown.duckdns.org</option>
|
||||
</select>
|
||||
<button id="routeBtn" type="button">Route it →</button>
|
||||
</div>
|
||||
<div class="trace" id="trace"></div>
|
||||
</div>
|
||||
|
||||
<h2 id="code">3 · Code — a walkthrough by concern</h2>
|
||||
<p>The script is one file, <code>setup_syncyomi.sh</code>, with <code>setup</code> / <code>status</code> / <code>help</code> subcommands. Here are the parts that carry the design, grouped by what they achieve.</p>
|
||||
|
||||
<h3>Reuse, don't rebuild: preflight</h3>
|
||||
<p>It refuses to invent infrastructure. It hard-requires the existing edge to be present and bails with a pointer if not — so it can never accidentally spin up a competing proxy.</p>
|
||||
<pre><code>has_cmd docker || die "docker not found — run setup_gitea.sh first."
|
||||
docker ps --format '{{.Names}}' | grep -qx "${CADDY_CONTAINER}" ||
|
||||
die "Caddy container '${CADDY_CONTAINER}' not running — run setup_gitea.sh first."
|
||||
[[ -f ${CADDYFILE} ]] || die "Caddyfile ${CADDYFILE} missing — run setup_gitea.sh first."
|
||||
[[ -x ${DUCKDNS_UPDATER} ]] || die "DuckDNS updater missing — run install_joplin.sh first."</code></pre>
|
||||
|
||||
<h3>The container: host networking, loopback bind</h3>
|
||||
<p><code>network_mode: host</code> sidesteps the FORWARD-chain trap; the loopback bind (enforced just below) keeps it private.</p>
|
||||
<pre><code>services:
|
||||
syncyomi:
|
||||
image: ghcr.io/syncyomi/syncyomi:v1.1.11 # pinned, not :latest
|
||||
restart: unless-stopped # ← survives reboot
|
||||
network_mode: host
|
||||
volumes: [ ./config:/config ]</code></pre>
|
||||
<p><code>restart: unless-stopped</code> plus the enabled Docker service is the entire "auto-start after reboot" story — no bespoke systemd unit needed.</p>
|
||||
|
||||
<h3>Readiness + the loopback rewrite</h3>
|
||||
<p>SyncYomi v1.1.x has <em>no</em> public health route (<code>/api/healthz</code> is a 404). The script instead treats <code>GET /api/auth/onboard</code> as "up" — it answers <code>204</code> when no account exists yet, or <code>403</code> once one does. Either means the API is serving.</p>
|
||||
<pre><code>syncyomi_ready() {
|
||||
code="$(curl -sS -o /dev/null -w '%{http_code}' "${base}/api/auth/onboard" ...)"
|
||||
[[ ${code} == "204" || ${code} == "403" ]]
|
||||
}</code></pre>
|
||||
<p>Once it's up, the script patches the generated config from the insecure default and restarts:</p>
|
||||
<pre><code>sed -i -E 's|^\s*host\s*=.*|host = "127.0.0.1"|' "${CONFIG_TOML}" # was 0.0.0.0</code></pre>
|
||||
|
||||
<h3>Minting the API key — the deliverable</h3>
|
||||
<p>This is the part that lets the script hand you a key instead of making you click through a web UI. It onboards the first account (only if none exists), logs in for a session cookie, then reuses or creates a key. Two gotchas discovered live are baked in:</p>
|
||||
<ul>
|
||||
<li>It <strong>reuses</strong> an existing <code>tachiyomisy</code> key so repeated runs don't pile up keys.</li>
|
||||
<li>The create call <strong>must send a non-empty <code>scopes</code></strong> array — SyncYomi returns <code>HTTP 500</code> on a null.</li>
|
||||
</ul>
|
||||
<pre><code>key="$(curl -sS -b "${jar}" "${SYNCYOMI_LOCAL}/api/keys" \
|
||||
| jq -r '[.[] | select(.name=="tachiyomisy")][0].key // empty')"
|
||||
if [[ -z ${key} ]]; then
|
||||
key="$(curl -sS -b "${jar}" \
|
||||
-d '{"name":"tachiyomisy","scopes":["read","write"]}' \
|
||||
"${SYNCYOMI_LOCAL}/api/keys" | jq -r '.key // empty')"
|
||||
fi</code></pre>
|
||||
|
||||
<h3>Joining the shared edge: Caddy + DuckDNS</h3>
|
||||
<p>One site block appended to the shared Caddyfile, then an in-container reload. Caddy issues the TLS cert on its own once the name resolves.</p>
|
||||
<pre><code>printf '\n%s {\n\treverse_proxy 127.0.0.1:%s\n}\n' \
|
||||
"${SYNCYOMI_SUBDOMAIN}" "${SYNCYOMI_PORT}" >>"${CADDYFILE}"
|
||||
docker exec "${CADDY_CONTAINER}" caddy reload --config "${CADDYFILE_IN_CONTAINER}"</code></pre>
|
||||
<p>And one label spliced into the existing updater's <code>domains=</code> list — no second cron job:</p>
|
||||
<pre><code>sed -i -E "s|(domains=)([^&\"']*)|\1\2,${label}|" "${DUCKDNS_UPDATER}"</code></pre>
|
||||
|
||||
<div class="note">
|
||||
<strong>The one fragility.</strong> That Caddy block lives in a file <em>owned</em> by <code>setup_gitea.sh</code>, which rewrites it from scratch when re-run. If that happens, SyncYomi's block vanishes — the fix is simply to re-run <code>setup_syncyomi.sh setup</code>, which idempotently re-appends it. This is a known, documented trade-off inherited from the sibling <code>setup_dufs_cloud.sh</code>.
|
||||
</div>
|
||||
|
||||
<h2 id="quiz">4 · Quiz</h2>
|
||||
<p>Five questions on the <em>reasoning</em> behind the change. Click an option to check it.</p>
|
||||
<div id="quiz"></div>
|
||||
<div class="score" id="score">Score: 0 / 0</div>
|
||||
|
||||
<div class="foot">
|
||||
Generated by <code>/explain-diff</code> for commit <code>59430f9</code> ·
|
||||
<code>linux_configuration/features/setup_syncyomi.sh</code>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// ---------- router micro-world ----------
|
||||
const ROUTES = {
|
||||
"kuhy.duckdns.org": { port: 3000, name: "Gitea" },
|
||||
"kuhy-cloud.duckdns.org": { port: 5000, name: "dufs" },
|
||||
"kuhy-sync.duckdns.org": { port: 8282, name: "SyncYomi" },
|
||||
};
|
||||
const traceEl = document.getElementById("trace");
|
||||
function step(html, cls) {
|
||||
const d = document.createElement("div");
|
||||
d.className = "step " + (cls || "");
|
||||
d.innerHTML = html;
|
||||
traceEl.appendChild(d);
|
||||
requestAnimationFrame(() => setTimeout(() => d.classList.add("show"), 10));
|
||||
return d;
|
||||
}
|
||||
document.getElementById("routeBtn").addEventListener("click", () => {
|
||||
traceEl.innerHTML = "";
|
||||
const host = document.getElementById("host").value;
|
||||
const seq = [
|
||||
() => step(`DNS: <b>${host}</b> resolves to the home public IP <small>(one IP for all labels)</small>`),
|
||||
() => step(`Arrives at <b>gitea-caddy</b> on <small>:443</small>, TLS terminated`, "edge"),
|
||||
() => {
|
||||
const r = ROUTES[host];
|
||||
if (r) return step(`Host matches a site block → forward to <b>127.0.0.1:${r.port}</b> <small>(${r.name})</small>`, "hit");
|
||||
return step(`No site block matches <b>${host}</b> → Caddy has no cert/route, TLS alert <small>(request refused)</small>`, "miss");
|
||||
},
|
||||
];
|
||||
seq.forEach((fn, i) => setTimeout(fn, i * 480));
|
||||
});
|
||||
|
||||
// ---------- quiz ----------
|
||||
// correct positions chosen to span A/B/C/D: [B, A, C, D, B]
|
||||
const QUIZ = [
|
||||
{
|
||||
stem: "Why does the SyncYomi container bind to 127.0.0.1:8282 rather than 0.0.0.0?",
|
||||
options: [
|
||||
"Because Docker host networking is unable to bind to the 0.0.0.0 wildcard address.",
|
||||
"So every request must pass through Caddy, which terminates TLS for the service.",
|
||||
"Because the Gitea container is already holding on to TCP port 8282 on this host.",
|
||||
"To make SyncYomi skip writing its generated config.toml file on first boot.",
|
||||
],
|
||||
correct: 1,
|
||||
why: "A 0.0.0.0 bind would expose an external door with no TLS, bypassing Caddy and serving the library in plaintext.",
|
||||
},
|
||||
{
|
||||
stem: "Why must the create-key request send a non-empty scopes array?",
|
||||
options: [
|
||||
"SyncYomi's store rejects a null scopes value and answers with HTTP 500.",
|
||||
"TachiyomiSY on the phone will not accept a key that grants no permissions.",
|
||||
"Caddy strips any scopes field out of the body before SyncYomi receives it.",
|
||||
"The onboarding call consumes the default scopes, leaving none for the key.",
|
||||
],
|
||||
correct: 0,
|
||||
why: "Discovered live: POST /api/keys with no scopes returned 500; adding [\"read\",\"write\"] returned 201 with the key.",
|
||||
},
|
||||
{
|
||||
stem: "Why doesn't the script start its own Caddy container for SyncYomi?",
|
||||
options: [
|
||||
"Because DuckDNS permits only a single reverse proxy per user account.",
|
||||
"Because a Caddy instance can serve at most one TLS certificate at a time.",
|
||||
"Because Gitea's Caddy already owns ports 80 and 443; a second would collide.",
|
||||
"Because SyncYomi already bundles its own reverse proxy that Caddy would duplicate.",
|
||||
],
|
||||
correct: 2,
|
||||
why: "One host has one :443. The design is a single shared edge that routes to many loopback backends by hostname.",
|
||||
},
|
||||
{
|
||||
stem: "What does the script poll to decide SyncYomi is 'up', and why that?",
|
||||
options: [
|
||||
"The container's exit code, which turns non-zero only after startup fails.",
|
||||
"GET /api/healthz, the documented health endpoint that returns 200 when ready.",
|
||||
"A TLS handshake against the public subdomain to confirm the cert is live.",
|
||||
"GET /api/auth/onboard, because v1.1.x exposes no public health route.",
|
||||
],
|
||||
correct: 3,
|
||||
why: "/api/healthz is a 404 in v1.1.x; onboard answers 204 (no user) or 403 (user exists), both meaning the API serves.",
|
||||
},
|
||||
{
|
||||
stem: "If setup_gitea.sh is re-run later, what happens to SyncYomi's exposure?",
|
||||
options: [
|
||||
"Nothing: each service keeps its exposure in a separate, private Caddyfile.",
|
||||
"Its Caddy block is wiped, so re-running setup_syncyomi.sh restores it.",
|
||||
"The kuhy-sync DuckDNS label is deleted and must be registered again by hand.",
|
||||
"SyncYomi's container is stopped until the Gitea setup finishes running.",
|
||||
],
|
||||
correct: 1,
|
||||
why: "setup_gitea.sh regenerates the shared Caddyfile from scratch; the appended block is lost until setup_syncyomi.sh re-adds it idempotently.",
|
||||
},
|
||||
];
|
||||
|
||||
const quizEl = document.getElementById("quiz");
|
||||
const scoreEl = document.getElementById("score");
|
||||
let answered = 0, correctCount = 0;
|
||||
|
||||
QUIZ.forEach((q, qi) => {
|
||||
const card = document.createElement("div");
|
||||
card.className = "q";
|
||||
const stem = document.createElement("div");
|
||||
stem.className = "stem";
|
||||
stem.textContent = (qi + 1) + ". " + q.stem;
|
||||
card.appendChild(stem);
|
||||
const why = document.createElement("div");
|
||||
why.className = "why";
|
||||
why.textContent = "Why: " + q.why;
|
||||
let done = false;
|
||||
q.options.forEach((opt, oi) => {
|
||||
const b = document.createElement("button");
|
||||
b.className = "opt";
|
||||
b.type = "button";
|
||||
b.textContent = opt;
|
||||
b.addEventListener("click", () => {
|
||||
if (done) return;
|
||||
done = true;
|
||||
answered++;
|
||||
const isRight = oi === q.correct;
|
||||
if (isRight) correctCount++;
|
||||
Array.from(card.querySelectorAll(".opt")).forEach((el, ei) => {
|
||||
el.disabled = true;
|
||||
if (ei === q.correct) el.classList.add("correct");
|
||||
else if (ei === oi) el.classList.add("wrong");
|
||||
});
|
||||
why.classList.add("show");
|
||||
scoreEl.textContent = `Score: ${correctCount} / ${answered}`;
|
||||
});
|
||||
card.appendChild(b);
|
||||
});
|
||||
card.appendChild(why);
|
||||
quizEl.appendChild(card);
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@ -130,7 +130,7 @@
|
||||
|
||||
<h2 id="background">1. Background</h2>
|
||||
|
||||
<p>This machine leads a double life. By day it is a desktop. Around the clock it is a small server: <strong>Gitea</strong>, a public <strong>Caddy</strong> TLS edge on ports 80/443, <strong>SyncYomi</strong>, a personal website, <strong>Open WebUI</strong>, <strong>Joplin</strong>, <strong>dufs</strong>, plus <code>ollama</code>, <code>dnsmasq</code>, <code>wg-quick@wg0</code>, <code>nftables</code> and <code>sshd</code>.</p>
|
||||
<p>This machine leads a double life. By day it is a desktop. Around the clock it is a small server: <strong>Gitea</strong>, a public <strong>Caddy</strong> TLS edge on ports 80/443, a personal website, <strong>Open WebUI</strong>, <strong>Joplin</strong>, <strong>dufs</strong>, plus <code>ollama</code>, <code>dnsmasq</code>, <code>wg-quick@wg0</code>, <code>nftables</code> and <code>sshd</code>.</p>
|
||||
|
||||
<p>It also runs a <em>digital-wellbeing curfew</em>: a deliberately hard-to-defeat system that stopped you using the PC late at night. It is built as a fortress, and the distinction between its parts is the key to this whole change:</p>
|
||||
|
||||
@ -184,7 +184,7 @@ esac</code></pre>
|
||||
<div class="flow">
|
||||
<div class="box kill"><b>Torn down at 21:00</b>lightdm → X → i3<br>TTY logins (<code>getty@</code> masked)<br>console (blanked)<br>RGB (black)<br>audio (muted)</div>
|
||||
<div class="arrow">|</div>
|
||||
<div class="box live"><b>Untouched, all night</b>Docker: gitea, caddy, syncyomi, website, open-webui, joplin, dufs<br><code>ollama</code>, <code>dnsmasq</code>, <code>wg-quick@wg0</code>, <code>nftables</code>, <code>sshd</code></div>
|
||||
<div class="box live"><b>Untouched, all night</b>Docker: gitea, caddy, website, open-webui, joplin, dufs<br><code>ollama</code>, <code>dnsmasq</code>, <code>wg-quick@wg0</code>, <code>nftables</code>, <code>sshd</code></div>
|
||||
</div>
|
||||
<figcaption>The dividing line: everything on the left needs a graphical session; nothing on the right does. That is why the split is clean rather than a compromise.</figcaption>
|
||||
</figure>
|
||||
|
||||
@ -80,10 +80,6 @@
|
||||
<span class="date">2026-07-13</span>
|
||||
<span class="title">Explaining: the hotspot / tethering block</span>
|
||||
</a>
|
||||
<a class="card" href="2026-07-13-explanation-setup-syncyomi.html">
|
||||
<span class="date">2026-07-13</span>
|
||||
<span class="title">Adding SyncYomi to a one-edge home server</span>
|
||||
</a>
|
||||
<a class="card" href="2026-07-13-explanation-personal-website-edge.html">
|
||||
<span class="date">2026-07-13</span>
|
||||
<span class="title">Personal website edge</span>
|
||||
|
||||
25
docs/superpowers/evidence/remove-syncyomi-2026-11-01.json
Normal file
25
docs/superpowers/evidence/remove-syncyomi-2026-11-01.json
Normal file
@ -0,0 +1,25 @@
|
||||
{
|
||||
"intent": "Remove every SyncYomi and TachiyomiSY trace; kuhy replaced the app with kartka.",
|
||||
"scope": [
|
||||
"python_pkg/syncyomi_guard, setup_syncyomi*.sh, lib/syncyomi_*.sh, syncyomi docs",
|
||||
"Non-goal: Suwayomi and eu.kanade.tachiyomi.extension mentions stay"
|
||||
],
|
||||
"changes": [
|
||||
"Deleted the syncyomi_guard package, its setup scripts, libs and runbook",
|
||||
"Dropped syncyomi lines from shared docs, comments and the coverage allowlist"
|
||||
],
|
||||
"verification": [
|
||||
{
|
||||
"command": "grep -rIliE 'syncyomi|tachiyomisy|tachiyomi\\.sy' . (excluding .git, caches)",
|
||||
"result": "pass",
|
||||
"evidence": "no matches in tracked files"
|
||||
},
|
||||
{
|
||||
"command": "shellcheck setup_gitea.sh setup_personal_website.sh",
|
||||
"result": "pass",
|
||||
"evidence": "no findings"
|
||||
}
|
||||
],
|
||||
"risks": ["Stale references in untracked local settings"],
|
||||
"rollback": ["git revert the commit", "Restore services from the 2026-11-01 backup tarball"]
|
||||
}
|
||||
@ -1,161 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Caddy vhost, DuckDNS record and firewall rules for exposing SyncYomi.
|
||||
#
|
||||
# Sourced by setup_syncyomi.sh; split out to keep syncyomi_stack.sh under
|
||||
# the 250-line cap. Sourced rather than run, so it inherits the caller's
|
||||
# strict mode and the variables defined above the source line.
|
||||
|
||||
# --- Step 6: Caddy reverse-proxy block (append + reload) ---------------------
|
||||
configure_caddy() {
|
||||
local snippet="${CADDY_SITES_DIR}/${SYNCYOMI_SUBDOMAIN}.caddy"
|
||||
local desired
|
||||
desired="$(printf '# Managed by setup_syncyomi.sh.\n%s {\n\treverse_proxy 127.0.0.1:%s\n}' \
|
||||
"${SYNCYOMI_SUBDOMAIN}" "${SYNCYOMI_PORT}")"
|
||||
|
||||
grep -qE '^[[:space:]]*import[[:space:]]+/etc/caddy/sites' "${CADDYFILE}" ||
|
||||
die "Root Caddyfile does not import sites/ — run setup_personal_website.sh first."
|
||||
|
||||
# Every service on this edge owns one snippet under sites/. Writing into the
|
||||
# shared Caddyfile instead is what produced "ambiguous site definition" on
|
||||
# 2026-08-28: the site was already defined in its snippet, the block was
|
||||
# appended a second time, and the reload failed while the edge kept serving
|
||||
# the last config it had accepted — so the damage only showed up later.
|
||||
ensure_dir "${CADDY_SITES_DIR}"
|
||||
if [[ -f ${snippet} && "$(cat "${snippet}")" == "${desired}" ]]; then
|
||||
log_ok "Caddy snippet already current: ${snippet}"
|
||||
else
|
||||
log_info "Writing Caddy snippet ${snippet}"
|
||||
printf '%s\n' "${desired}" >"${snippet}"
|
||||
fi
|
||||
|
||||
if grep -qE "^${SYNCYOMI_SUBDOMAIN} \{" "${CADDYFILE}"; then
|
||||
log_info "Removing the legacy inline ${SYNCYOMI_SUBDOMAIN} block from ${CADDYFILE}"
|
||||
sed -i -E "/^${SYNCYOMI_SUBDOMAIN} \{/,/^\}/d" "${CADDYFILE}"
|
||||
fi
|
||||
|
||||
docker exec "${CADDY_CONTAINER}" caddy validate --config "${CADDYFILE_IN_CONTAINER}" >/dev/null 2>&1 ||
|
||||
die "Caddy config invalid — not reloading. Check ${CADDY_SITES_DIR}/*.caddy."
|
||||
docker exec "${CADDY_CONTAINER}" caddy reload --config "${CADDYFILE_IN_CONTAINER}" ||
|
||||
die "caddy reload failed — check 'docker logs ${CADDY_CONTAINER}'."
|
||||
log_ok "Caddy reloaded (auto-HTTPS will be issued once the subdomain resolves)"
|
||||
}
|
||||
|
||||
# --- Step 7: DuckDNS subdomain (extend existing updater; no new cron) --------
|
||||
configure_duckdns() {
|
||||
local label="${SYNCYOMI_SUBDOMAIN%.duckdns.org}"
|
||||
if grep -qE "domains=[^&\"']*${label}([,&\"']|$)" "${DUCKDNS_UPDATER}"; then
|
||||
log_ok "DuckDNS updater already includes '${label}'"
|
||||
else
|
||||
log_info "Adding '${label}' to the existing DuckDNS updater"
|
||||
sed -i -E "s|(domains=)([^&\"']*)|\1\2,${label}|" "${DUCKDNS_UPDATER}"
|
||||
fi
|
||||
# Refresh now (harmless if the subdomain isn't registered yet).
|
||||
bash "${DUCKDNS_UPDATER}" >/dev/null 2>&1 || log_warn "DuckDNS update returned non-zero (register the subdomain first)."
|
||||
if getent hosts "${SYNCYOMI_SUBDOMAIN}" >/dev/null 2>&1; then
|
||||
log_ok "${SYNCYOMI_SUBDOMAIN} resolves"
|
||||
else
|
||||
log_warn "${SYNCYOMI_SUBDOMAIN} does not resolve yet — create the '${label}' subdomain in the DuckDNS dashboard."
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Step 8: firewall (open 80/443 via the owning script, idempotent) -------
|
||||
ensure_firewall() {
|
||||
if sudo nft list ruleset 2>/dev/null | grep -qE 'dport \{[^}]*80[^}]*443'; then
|
||||
log_ok "Firewall already allows tcp/80 + tcp/443"
|
||||
elif [[ -f ${WG_SCRIPT} ]]; then
|
||||
log_info "Opening 80/443 via setup_wireguard_ssh.sh allow-web"
|
||||
sudo bash "${WG_SCRIPT}" allow-web
|
||||
else
|
||||
log_warn "setup_wireguard_ssh.sh not found — ensure tcp/80+443 are open."
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Step 9: report ----------------------------------------------------------
|
||||
print_report() {
|
||||
local url="https://${SYNCYOMI_SUBDOMAIN}"
|
||||
local key="(create one in the web UI)"
|
||||
[[ -f ${TOKEN_FILE} ]] && key="$(<"${TOKEN_FILE}")"
|
||||
cat <<EOF
|
||||
|
||||
============================================================================
|
||||
SyncYomi is up.
|
||||
Host address : ${url}
|
||||
API key : ${key}
|
||||
Web UI login : ${SYNCYOMI_USER} / (password in ${CONF_FILE})
|
||||
|
||||
TachiyomiSY setup (on the phone):
|
||||
Settings → Data & storage → Sync → SyncYomi
|
||||
Host : ${url}
|
||||
API key : ${key}
|
||||
Use the SAME API key on every device you want to sync.
|
||||
|
||||
If the subdomain is new, create it once in the DuckDNS dashboard
|
||||
(label: ${SYNCYOMI_SUBDOMAIN%.duckdns.org}) so Caddy can issue TLS.
|
||||
|
||||
Acceptance test: on your phone with Wi-Fi OFF (cellular), configure the
|
||||
sync above and confirm the library round-trips.
|
||||
============================================================================
|
||||
EOF
|
||||
}
|
||||
|
||||
# --- status subcommand -------------------------------------------------------
|
||||
status_cmd() {
|
||||
print_setup_header "SyncYomi status"
|
||||
echo
|
||||
if [[ -f ${COMPOSE_FILE} ]]; then
|
||||
docker compose -f "${COMPOSE_FILE}" ps
|
||||
else
|
||||
log_warn "No compose file at ${COMPOSE_FILE}"
|
||||
fi
|
||||
echo
|
||||
|
||||
local code
|
||||
code="$(syncyomi_ready "${SYNCYOMI_LOCAL}" || true)"
|
||||
if [[ ${code} == "204" || ${code} == "403" ]]; then
|
||||
log_ok "Local API answering (${SYNCYOMI_LOCAL}, HTTP ${code})"
|
||||
else
|
||||
log_warn "Local API not answering (HTTP ${code:-none})"
|
||||
fi
|
||||
|
||||
local sub="${SYNCYOMI_SUBDOMAIN:-kuhy-sync.duckdns.org}"
|
||||
if [[ -f ${CONF_FILE} ]]; then
|
||||
# shellcheck source=/dev/null
|
||||
source "${CONF_FILE}"
|
||||
sub="${SYNCYOMI_SUBDOMAIN}"
|
||||
fi
|
||||
|
||||
if grep -qE "^${sub} \{" "${CADDYFILE}" 2>/dev/null; then
|
||||
log_ok "Caddy block present for ${sub}"
|
||||
else
|
||||
log_warn "No Caddy block for ${sub}"
|
||||
fi
|
||||
|
||||
if getent hosts "${sub}" >/dev/null 2>&1; then
|
||||
log_ok "${sub} resolves"
|
||||
else
|
||||
log_warn "${sub} does not resolve"
|
||||
fi
|
||||
|
||||
code="$(syncyomi_ready "https://${sub}" || true)"
|
||||
if [[ ${code} == "204" || ${code} == "403" ]]; then
|
||||
log_ok "External HTTPS reachable (https://${sub}, HTTP ${code})"
|
||||
else
|
||||
log_warn "External HTTPS not reachable yet (HTTP ${code:-none})"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- main --------------------------------------------------------------------
|
||||
setup_cmd() {
|
||||
print_setup_header "Self-hosted SyncYomi manga sync"
|
||||
preflight
|
||||
load_config
|
||||
write_compose
|
||||
start_containers
|
||||
wait_for_syncyomi
|
||||
ensure_local_config
|
||||
mint_api_key
|
||||
configure_caddy
|
||||
configure_duckdns
|
||||
ensure_firewall
|
||||
print_report
|
||||
}
|
||||
@ -1,176 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Compose file, container startup and readiness polling for SyncYomi.
|
||||
#
|
||||
# Sourced by setup_syncyomi.sh; split out to keep it under the 250-line cap.
|
||||
# Sourced rather than run, so it inherits the caller's strict mode and
|
||||
# the helper functions and variables defined above the source line.
|
||||
|
||||
# --- Step 3: docker-compose stack (host networking, loopback bind) -----------
|
||||
write_compose() {
|
||||
ensure_dir "${DATA_DIR}/config"
|
||||
# network_mode: host + SyncYomi's host=127.0.0.1 keeps it loopback-only
|
||||
# (reached by the host-networked Caddy) and dodges this host's default-drop
|
||||
# nftables FORWARD chain that would kill a bridge-networked container.
|
||||
umask 077
|
||||
cat >"${COMPOSE_FILE}" <<EOF
|
||||
# Generated by setup_syncyomi.sh — do not edit by hand.
|
||||
services:
|
||||
syncyomi:
|
||||
container_name: syncyomi
|
||||
image: ${SYNCYOMI_IMAGE}
|
||||
restart: unless-stopped
|
||||
network_mode: host
|
||||
environment:
|
||||
- TZ=${TZ_VALUE}
|
||||
volumes:
|
||||
- ./config:/config
|
||||
EOF
|
||||
log_ok "Wrote ${COMPOSE_FILE}"
|
||||
}
|
||||
|
||||
# Build the local image the compose file asks for. A local tag exists in no
|
||||
# registry, so `docker compose up` would fail with a pull error rather than an
|
||||
# explanation -- and a fork whose changes are never rebuilt is worse than no
|
||||
# fork at all. Builds whatever is checked out in ${SYNCYOMI_SRC}; never moves
|
||||
# that working tree. Set SYNCYOMI_REBUILD=1 to rebuild after editing it.
|
||||
ensure_local_image() {
|
||||
if [[ ${SYNCYOMI_IMAGE} == */* ]]; then
|
||||
log_info "Using registry image ${SYNCYOMI_IMAGE}"
|
||||
return 0
|
||||
fi
|
||||
if [[ ${SYNCYOMI_REBUILD:-0} != "1" ]] && docker image inspect "${SYNCYOMI_IMAGE}" >/dev/null 2>&1; then
|
||||
log_ok "Local image ${SYNCYOMI_IMAGE} already built"
|
||||
return 0
|
||||
fi
|
||||
if [[ ! -d ${SYNCYOMI_SRC}/.git ]]; then
|
||||
log_info "Cloning ${SYNCYOMI_FORK_URL} into ${SYNCYOMI_SRC}"
|
||||
git clone "${SYNCYOMI_FORK_URL}" "${SYNCYOMI_SRC}"
|
||||
fi
|
||||
local version revision
|
||||
version="$(git -C "${SYNCYOMI_SRC}" describe --tags --always)"
|
||||
revision="$(git -C "${SYNCYOMI_SRC}" rev-parse HEAD)"
|
||||
log_info "Building ${SYNCYOMI_IMAGE} from ${SYNCYOMI_SRC} (${version})"
|
||||
# BuildKit is required: the Dockerfile reads $BUILDPLATFORM, which the
|
||||
# legacy builder leaves empty and then rejects as an invalid platform.
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
-t "${SYNCYOMI_IMAGE}" \
|
||||
--build-arg VERSION="${version}-kuhy" \
|
||||
--build-arg REVISION="${revision}" \
|
||||
--build-arg BUILDTIME="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
|
||||
"${SYNCYOMI_SRC}"
|
||||
log_ok "Built ${SYNCYOMI_IMAGE} from ${version}"
|
||||
}
|
||||
|
||||
start_containers() {
|
||||
ensure_local_image
|
||||
log_info "Starting SyncYomi container"
|
||||
docker compose -f "${COMPOSE_FILE}" up -d
|
||||
}
|
||||
|
||||
# Readiness probe. SyncYomi (v1.1.x) has no public /healthz, but GET
|
||||
# /api/auth/onboard answers 204 (no user yet) or 403 (a user exists) once the
|
||||
# API is serving — either means "up". Echoes the HTTP code; returns 0 if ready.
|
||||
# Usage: syncyomi_ready "http://127.0.0.1:8282"
|
||||
syncyomi_ready() {
|
||||
local base="$1" code
|
||||
code="$(curl -sS -o /dev/null -w '%{http_code}' "${base}/api/auth/onboard" 2>/dev/null || true)"
|
||||
echo "${code}"
|
||||
[[ ${code} == "204" || ${code} == "403" ]]
|
||||
}
|
||||
|
||||
# --- Step 4: wait until the local API answers --------------------------------
|
||||
wait_for_syncyomi() {
|
||||
log_info "Waiting for SyncYomi to come up on ${SYNCYOMI_LOCAL}"
|
||||
local i
|
||||
for ((i = 0; i < 60; i++)); do
|
||||
if syncyomi_ready "${SYNCYOMI_LOCAL}" >/dev/null; then
|
||||
log_ok "SyncYomi is up (API answering)"
|
||||
return 0
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
die "SyncYomi did not become healthy in 120s — check 'docker logs syncyomi'."
|
||||
}
|
||||
|
||||
# Defensive: SyncYomi generates config/config.toml on first run. Ensure it binds
|
||||
# loopback-only (never 0.0.0.0, which would bypass Caddy/TLS) and serves at root.
|
||||
ensure_local_config() {
|
||||
[[ -f ${CONFIG_TOML} ]] || return 0
|
||||
local changed=0
|
||||
if grep -qE '^\s*host\s*=' "${CONFIG_TOML}"; then
|
||||
if ! grep -qE '^\s*host\s*=\s*"127\.0\.0\.1"' "${CONFIG_TOML}"; then
|
||||
sed -i -E 's|^\s*host\s*=.*|host = "127.0.0.1"|' "${CONFIG_TOML}"
|
||||
changed=1
|
||||
fi
|
||||
fi
|
||||
# Force an empty baseUrl (dedicated subdomain, not a sub-path).
|
||||
if grep -qE '^\s*baseUrl\s*=' "${CONFIG_TOML}" &&
|
||||
! grep -qE '^\s*baseUrl\s*=\s*""' "${CONFIG_TOML}"; then
|
||||
sed -i -E 's|^\s*baseUrl\s*=.*|baseUrl = ""|' "${CONFIG_TOML}"
|
||||
changed=1
|
||||
fi
|
||||
if [[ ${changed} -eq 1 ]]; then
|
||||
log_info "Patched ${CONFIG_TOML} (host=127.0.0.1, baseUrl empty) — restarting"
|
||||
docker compose -f "${COMPOSE_FILE}" restart
|
||||
wait_for_syncyomi
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Step 5: create first account + mint API key over the local REST API -----
|
||||
mint_api_key() {
|
||||
if [[ -f ${TOKEN_FILE} ]]; then
|
||||
log_ok "API key already minted ($(<"${TOKEN_FILE}") stored in ${TOKEN_FILE})"
|
||||
return 0
|
||||
fi
|
||||
log_info "Creating SyncYomi account and minting API key"
|
||||
|
||||
local jar body http key
|
||||
jar="$(mktemp)"
|
||||
# shellcheck disable=SC2064
|
||||
trap "rm -f '${jar}'" RETURN
|
||||
|
||||
# JSON with proper escaping (password may contain +,/,= from base64).
|
||||
body="$(jq -n --arg u "${SYNCYOMI_USER}" --arg p "${SYNCYOMI_PASSWORD}" \
|
||||
'{username:$u,password:$p}')"
|
||||
|
||||
# Onboard the first user only if none exists yet (GET returns 204 when empty).
|
||||
http="$(curl -fsS -o /dev/null -w '%{http_code}' "${SYNCYOMI_LOCAL}/api/auth/onboard" 2>/dev/null || true)"
|
||||
if [[ ${http} == "204" ]]; then
|
||||
http="$(curl -sS -o /dev/null -w '%{http_code}' \
|
||||
-H 'Content-Type: application/json' -d "${body}" \
|
||||
"${SYNCYOMI_LOCAL}/api/auth/onboard" 2>/dev/null || true)"
|
||||
[[ ${http} =~ ^2 ]] || die "Onboard failed (HTTP ${http}) — check 'docker logs syncyomi'."
|
||||
log_ok "Created first account '${SYNCYOMI_USER}'"
|
||||
else
|
||||
log_warn "An account already exists (onboard HTTP ${http}); trying stored credentials"
|
||||
fi
|
||||
|
||||
# Log in to get a session cookie.
|
||||
http="$(curl -sS -c "${jar}" -o /dev/null -w '%{http_code}' \
|
||||
-H 'Content-Type: application/json' -d "${body}" \
|
||||
"${SYNCYOMI_LOCAL}/api/auth/login" 2>/dev/null || true)"
|
||||
if [[ ! ${http} =~ ^2 ]]; then
|
||||
log_warn "Login failed (HTTP ${http}). A different account may already own this server."
|
||||
log_warn "Create an API key manually at ${SYNCYOMI_LOCAL} → Settings → API Keys."
|
||||
return 0
|
||||
fi
|
||||
|
||||
# Reuse an existing 'tachiyomisy' key if one exists (idempotent); otherwise
|
||||
# create one. scopes must be a non-empty array — SyncYomi rejects a null.
|
||||
key="$(curl -sS -b "${jar}" "${SYNCYOMI_LOCAL}/api/keys" 2>/dev/null |
|
||||
jq -r '[.[] | select(.name=="tachiyomisy")][0].key // empty')"
|
||||
if [[ -n ${key} ]]; then
|
||||
log_ok "Reusing existing 'tachiyomisy' API key"
|
||||
else
|
||||
key="$(curl -sS -b "${jar}" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"name":"tachiyomisy","scopes":["read","write"]}' \
|
||||
"${SYNCYOMI_LOCAL}/api/keys" 2>/dev/null | jq -r '.key // empty')"
|
||||
[[ -n ${key} ]] || die "API key creation returned no key — check 'docker logs syncyomi'."
|
||||
fi
|
||||
|
||||
umask 077
|
||||
printf '%s\n' "${key}" >"${TOKEN_FILE}"
|
||||
chmod 600 "${TOKEN_FILE}"
|
||||
log_ok "Minted API key and stored it in ${TOKEN_FILE} (0600)"
|
||||
}
|
||||
@ -96,7 +96,7 @@ volumes:
|
||||
EOF
|
||||
|
||||
# The root Caddyfile only imports per-service snippets; every service
|
||||
# (gitea, dufs, syncyomi, the personal website, ...) owns its own file in
|
||||
# (gitea, dufs, the personal website, ...) owns its own file in
|
||||
# sites/ so re-running any one setup script never drops another's block.
|
||||
ensure_dir "$SITES_DIR"
|
||||
cat >"$CADDYFILE" <<'EOF'
|
||||
|
||||
@ -11,7 +11,7 @@
|
||||
# - Installs build/runtime deps (node, pnpm, docker, ...).
|
||||
# - Migrates the shared Caddyfile to a per-service snippet directory so no
|
||||
# service block ever clobbers another (fixes a latent bug where re-running
|
||||
# setup_gitea.sh dropped the dufs/syncyomi blocks).
|
||||
# setup_gitea.sh dropped the dufs block).
|
||||
# - Moves Gitea to gitea.kuhy.duckdns.org via setup_gitea.sh (single source).
|
||||
# - Builds the website and serves dist/ from a loopback caddy container on
|
||||
# 127.0.0.1:8088, fronted by a reverse-proxy snippet on the root domain.
|
||||
@ -104,7 +104,7 @@ preflight() {
|
||||
# Reads the CURRENT monolithic Caddyfile and writes every top-level block to
|
||||
# sites/<label>.caddy, EXCEPT the old root block (kuhy.duckdns.org, previously
|
||||
# gitea — now repurposed for the website) and the gitea block (setup_gitea.sh
|
||||
# rewrites its own). This preserves dufs (kuhy-cloud) and syncyomi (kuhy-sync)
|
||||
# rewrites its own). This preserves dufs (kuhy-cloud)
|
||||
# before setup_gitea.sh switches the root Caddyfile to `import`. Idempotent.
|
||||
migrate_caddy_snippets() {
|
||||
ensure_dir "$SITES_DIR"
|
||||
|
||||
@ -1,150 +0,0 @@
|
||||
#!/bin/bash
|
||||
# setup_syncyomi.sh — Self-hosted SyncYomi manga-sync server on this Arch PC.
|
||||
#
|
||||
# Stands up SyncYomi (github.com/SyncYomi/SyncYomi) in Docker, bound to
|
||||
# 127.0.0.1:8282, and fronts it with the EXISTING Caddy reverse proxy (from
|
||||
# setup_gitea.sh) for automatic public HTTPS at a new DuckDNS subdomain, so
|
||||
# TachiyomiSY on the phone can sync its library from a mobile network.
|
||||
#
|
||||
# It reuses the host's existing exposure stack — it does NOT add a second Caddy,
|
||||
# a second DuckDNS updater, or hand-edit /etc/nftables.conf. It also creates the
|
||||
# first SyncYomi account and mints an API key over the local REST API, then
|
||||
# prints the host address + API key ready to paste into TachiyomiSY.
|
||||
#
|
||||
# Run as your normal user (NOT root). It uses sudo only for pacman, the firewall
|
||||
# flag and enabling docker. Idempotent — safe to re-run.
|
||||
#
|
||||
# Usage:
|
||||
# ./setup_syncyomi.sh Install / update everything (default)
|
||||
# ./setup_syncyomi.sh setup Same as above
|
||||
# ./setup_syncyomi.sh status Show health / exposure diagnostics
|
||||
# ./setup_syncyomi.sh help Show this help
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# --- SyncYomi image ----------------------------------------------------------
|
||||
# Runs a local build of the personal fork by default, so server-side changes
|
||||
# actually reach the service instead of sitting unbuilt in a clone. The
|
||||
# published image stays one variable away:
|
||||
# SYNCYOMI_IMAGE=ghcr.io/syncyomi/syncyomi:v1.1.11 ./setup_syncyomi.sh
|
||||
# Any value containing a "/" is treated as a registry image and pulled as-is.
|
||||
readonly SYNCYOMI_IMAGE="${SYNCYOMI_IMAGE:-syncyomi-kuhy:local}"
|
||||
readonly SYNCYOMI_SRC="${SYNCYOMI_SRC:-${HOME}/vendor/syncyomi-src}"
|
||||
readonly SYNCYOMI_FORK_URL="https://github.com/kuhyx/syncyomi.git"
|
||||
readonly SYNCYOMI_PORT=8282
|
||||
readonly SYNCYOMI_LOCAL="http://127.0.0.1:${SYNCYOMI_PORT}"
|
||||
|
||||
# --- Data dir + secrets (outside the git repo, like ~/services/gitea) -----------------
|
||||
readonly DATA_DIR="${HOME}/services/syncyomi"
|
||||
readonly COMPOSE_FILE="${DATA_DIR}/docker-compose.yml"
|
||||
readonly CONFIG_TOML="${DATA_DIR}/config/config.toml"
|
||||
readonly CONF_FILE="${DATA_DIR}/.syncyomi.conf"
|
||||
readonly TOKEN_FILE="${DATA_DIR}/.api_token"
|
||||
|
||||
# --- Existing host infrastructure (from setup_gitea.sh / install_joplin.sh) --
|
||||
readonly CADDY_CONTAINER="gitea-caddy"
|
||||
readonly CADDYFILE="${HOME}/services/gitea/Caddyfile"
|
||||
readonly CADDY_SITES_DIR="${HOME}/services/gitea/sites"
|
||||
readonly CADDYFILE_IN_CONTAINER="/etc/caddy/Caddyfile"
|
||||
readonly DUCKDNS_UPDATER="${HOME}/.joplin-server/duckdns-update.sh"
|
||||
|
||||
SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
|
||||
readonly SCRIPT_DIR
|
||||
readonly WG_SCRIPT="${SCRIPT_DIR}/setup_wireguard_ssh.sh"
|
||||
|
||||
# shellcheck source=../lib/common.sh
|
||||
# shellcheck disable=SC1091
|
||||
source "${SCRIPT_DIR}/../lib/common.sh"
|
||||
|
||||
# Config values (populated from CONF_FILE / defaults).
|
||||
SYNCYOMI_USER=""
|
||||
SYNCYOMI_PASSWORD=""
|
||||
SYNCYOMI_SUBDOMAIN=""
|
||||
|
||||
# Local timezone for the container (falls back if timedatectl is unavailable).
|
||||
TZ_VALUE="$(timedatectl show -p Timezone --value 2>/dev/null || echo 'Europe/Warsaw')"
|
||||
readonly TZ_VALUE
|
||||
|
||||
die() {
|
||||
log_error "$1"
|
||||
exit 1
|
||||
}
|
||||
|
||||
usage() {
|
||||
grep -E '^#( |$)' "$0" | sed -E 's/^# ?//'
|
||||
exit 0
|
||||
}
|
||||
|
||||
# --- Step 1: preflight -------------------------------------------------------
|
||||
preflight() {
|
||||
log_info "Preflight checks"
|
||||
[[ ${EUID} -ne 0 ]] || die "Run as your normal user, not root (it will sudo where needed)."
|
||||
|
||||
# Runtime deps; install any that are missing from the official repos.
|
||||
# docker-buildx: the fork's Dockerfile uses $BUILDPLATFORM, which only the
|
||||
# BuildKit builder defines; the legacy builder rejects the empty value.
|
||||
install_missing_pacman_packages docker docker-compose jq curl openssl git docker-buildx
|
||||
|
||||
# Ensure the docker daemon is enabled+running (auto-start after reboot). Only
|
||||
# touch systemd via sudo when it is not already active, to stay a no-op on the
|
||||
# common case where docker is already up (Gitea/dufs run on it).
|
||||
has_cmd docker || die "docker not found — run setup_gitea.sh first."
|
||||
if ! is_service_active docker; then
|
||||
log_info "Enabling and starting docker.service"
|
||||
sudo systemctl enable --now docker
|
||||
fi
|
||||
|
||||
# The public-exposure stack must already exist (Caddy from setup_gitea.sh).
|
||||
docker ps --format '{{.Names}}' | grep -qx "${CADDY_CONTAINER}" ||
|
||||
die "Caddy container '${CADDY_CONTAINER}' not running — run setup_gitea.sh first."
|
||||
[[ -f ${CADDYFILE} ]] || die "Caddyfile ${CADDYFILE} missing — run setup_gitea.sh first."
|
||||
[[ -x ${DUCKDNS_UPDATER} ]] || die "DuckDNS updater ${DUCKDNS_UPDATER} missing — run install_joplin.sh first."
|
||||
log_ok "Host exposure stack present (docker + ${CADDY_CONTAINER} + DuckDNS updater)"
|
||||
}
|
||||
|
||||
# --- Step 2: config (auto-generate credentials, store 0600) ------------------
|
||||
load_config() {
|
||||
ensure_dir "${DATA_DIR}"
|
||||
chmod 700 "${DATA_DIR}"
|
||||
if [[ -f ${CONF_FILE} ]]; then
|
||||
# shellcheck source=/dev/null
|
||||
source "${CONF_FILE}"
|
||||
fi
|
||||
SYNCYOMI_USER="${SYNCYOMI_USER:-$(get_actual_user)}"
|
||||
SYNCYOMI_SUBDOMAIN="${SYNCYOMI_SUBDOMAIN:-kuhy-sync.duckdns.org}"
|
||||
# Generate the web/API password once; never regenerate on re-run.
|
||||
if [[ -z ${SYNCYOMI_PASSWORD:-} ]]; then
|
||||
SYNCYOMI_PASSWORD="$(openssl rand -base64 24)"
|
||||
fi
|
||||
save_config
|
||||
}
|
||||
|
||||
save_config() {
|
||||
# 0600 — holds the plaintext SyncYomi password so re-runs stay
|
||||
# non-interactive. Lives under ~/services/syncyomi (outside the git repo).
|
||||
umask 077
|
||||
{
|
||||
printf 'SYNCYOMI_USER=%q\n' "${SYNCYOMI_USER}"
|
||||
printf 'SYNCYOMI_PASSWORD=%q\n' "${SYNCYOMI_PASSWORD}"
|
||||
printf 'SYNCYOMI_SUBDOMAIN=%q\n' "${SYNCYOMI_SUBDOMAIN}"
|
||||
} >"${CONF_FILE}"
|
||||
chmod 600 "${CONF_FILE}"
|
||||
log_ok "Saved config to ${CONF_FILE} (0600)"
|
||||
}
|
||||
|
||||
# shellcheck source=lib/syncyomi_stack.sh
|
||||
source "$SCRIPT_DIR/lib/syncyomi_stack.sh"
|
||||
# shellcheck source=lib/syncyomi_expose.sh
|
||||
source "$SCRIPT_DIR/lib/syncyomi_expose.sh"
|
||||
|
||||
main() {
|
||||
local cmd="${1:-setup}"
|
||||
case "${cmd}" in
|
||||
setup) setup_cmd ;;
|
||||
status) status_cmd ;;
|
||||
help | -h | --help) usage ;;
|
||||
*) die "Unknown command '${cmd}' (use: setup | status | help)" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@ -1,123 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# ============================================================================
|
||||
# Install the SyncYomi library integrity guard as a user systemd timer.
|
||||
#
|
||||
# TachiyomiSY 1.13.2 can finish a backup restore, report success, and have
|
||||
# written almost nothing (upstream jobobby04/TachiyomiSY#1634, #1638). The app
|
||||
# then syncs that near-empty library up, replacing the good server copy. On
|
||||
# 2026-08-09 the real library survived only as stale pages in the SQLite WAL.
|
||||
#
|
||||
# The guard cannot fix the app. It makes the silent case loud, and keeps
|
||||
# restorable .tachibk snapshots so recovery never again depends on WAL forensics.
|
||||
# ============================================================================
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_NAME="$(basename "$0")"
|
||||
readonly SCRIPT_NAME
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
readonly REPO_ROOT
|
||||
readonly UNIT_DIR="${HOME}/.config/systemd/user"
|
||||
readonly SERVICE_NAME="syncyomi-guard.service"
|
||||
readonly TIMER_NAME="syncyomi-guard.timer"
|
||||
readonly SNAPSHOT_DIR="${HOME}/services/syncyomi/snapshots"
|
||||
readonly STATE_DIR="${HOME}/.local/share/syncyomi_guard"
|
||||
readonly DB_PATH="${HOME}/services/syncyomi/config/syncyomi.db"
|
||||
|
||||
log_info() { echo "[*] $*"; }
|
||||
log_ok() { echo "[+] $*"; }
|
||||
die() {
|
||||
echo "[!] $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
usage() {
|
||||
echo "Usage: ${SCRIPT_NAME} [--uninstall]"
|
||||
echo " Installs a user timer that verifies the SyncYomi payload every 30 min."
|
||||
exit 0
|
||||
}
|
||||
|
||||
require_python() {
|
||||
command -v python3 >/dev/null 2>&1 ||
|
||||
die "python3 not found — install it with: sudo pacman -S python"
|
||||
}
|
||||
|
||||
# The guard reads the database through sudo-less paths, so it must be able to
|
||||
# open it. A root-owned db (the container writes as root) is readable only if
|
||||
# the mode allows it; check now rather than failing silently on the first timer.
|
||||
check_readable() {
|
||||
[[ -f ${DB_PATH} ]] || die "no SyncYomi database at ${DB_PATH}"
|
||||
if ! head -c 1 "${DB_PATH}" >/dev/null 2>&1; then
|
||||
die "cannot read ${DB_PATH} as $(whoami) — run: sudo chmod o+r '${DB_PATH}'"
|
||||
fi
|
||||
}
|
||||
|
||||
write_units() {
|
||||
mkdir -p "${UNIT_DIR}" "${SNAPSHOT_DIR}" "${STATE_DIR}"
|
||||
|
||||
cat >"${UNIT_DIR}/${SERVICE_NAME}" <<EOF
|
||||
# Generated by ${SCRIPT_NAME} — do not edit by hand.
|
||||
[Unit]
|
||||
Description=Verify the SyncYomi library payload has not silently collapsed
|
||||
Documentation=https://github.com/jobobby04/TachiyomiSY/issues/1634
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
WorkingDirectory=${REPO_ROOT}
|
||||
Environment=PYTHONPATH=${REPO_ROOT}
|
||||
ExecStart=/usr/bin/python3 -m python_pkg.syncyomi_guard \\
|
||||
--db ${DB_PATH} \\
|
||||
--state-dir ${STATE_DIR} \\
|
||||
--snapshot-dir ${SNAPSHOT_DIR}
|
||||
EOF
|
||||
|
||||
cat >"${UNIT_DIR}/${TIMER_NAME}" <<EOF
|
||||
# Generated by ${SCRIPT_NAME} — do not edit by hand.
|
||||
[Unit]
|
||||
Description=Periodic SyncYomi library integrity check
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*:00/30
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
EOF
|
||||
|
||||
log_ok "Wrote ${UNIT_DIR}/${SERVICE_NAME} and ${TIMER_NAME}"
|
||||
}
|
||||
|
||||
install_timer() {
|
||||
require_python
|
||||
check_readable
|
||||
write_units
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable --now "${TIMER_NAME}"
|
||||
log_ok "Timer enabled (every 30 minutes)"
|
||||
|
||||
log_info "Running one check now"
|
||||
if systemctl --user start "${SERVICE_NAME}"; then
|
||||
systemctl --user status "${SERVICE_NAME}" --no-pager -n 10 || true
|
||||
else
|
||||
die "first run failed — inspect: journalctl --user -u ${SERVICE_NAME}"
|
||||
fi
|
||||
}
|
||||
|
||||
uninstall_timer() {
|
||||
systemctl --user disable --now "${TIMER_NAME}" 2>/dev/null || true
|
||||
rm -f "${UNIT_DIR}/${SERVICE_NAME}" "${UNIT_DIR}/${TIMER_NAME}"
|
||||
systemctl --user daemon-reload
|
||||
log_ok "Removed timer and service (snapshots in ${SNAPSHOT_DIR} kept)"
|
||||
}
|
||||
|
||||
main() {
|
||||
case "${1:-}" in
|
||||
--uninstall) uninstall_timer ;;
|
||||
-h | --help) usage ;;
|
||||
"") install_timer ;;
|
||||
*) die "Unknown option: $1" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@ -52,8 +52,6 @@ linux_configuration/features/lib/searxng_settings.sh
|
||||
linux_configuration/features/lib/searxng_stack.sh
|
||||
linux_configuration/features/lib/searxng_start.sh
|
||||
linux_configuration/features/lib/searxng_status.sh
|
||||
linux_configuration/features/lib/syncyomi_expose.sh
|
||||
linux_configuration/features/lib/syncyomi_stack.sh
|
||||
linux_configuration/features/lib/unreal_mcp_repo.sh
|
||||
linux_configuration/features/lib/website_build.sh
|
||||
linux_configuration/features/lib/wg_firewall.sh
|
||||
|
||||
@ -1,41 +0,0 @@
|
||||
"""Integrity guard for the SyncYomi server's library payload.
|
||||
|
||||
TachiyomiSY 1.13.2 has an open upstream bug (jobobby04/TachiyomiSY#1634, #1638)
|
||||
where a backup restore fails with ``SQLITE_BUSY`` on the app's *own* database.
|
||||
The loud failure is survivable. The dangerous case is the quiet one: a restore
|
||||
that reports ``Worker result SUCCESS`` while having written almost nothing, and
|
||||
then pushes that near-empty library up to the server, replacing the only good
|
||||
copy.
|
||||
|
||||
That happened on 2026-08-09: a 14 MB payload holding 2182 manga became a 267 KB
|
||||
stub holding only sources and settings, and the real library survived solely as
|
||||
stale pages in the SQLite write-ahead log.
|
||||
|
||||
This package exists so that never depends on luck again. It decodes the payload,
|
||||
counts what is actually in it, and refuses to accept a collapse as normal. It
|
||||
cannot prevent the upstream bug — that is Kotlin code in an APK — so it aims at
|
||||
the part that is actually fixable: never letting a degraded copy become the only
|
||||
copy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from python_pkg.syncyomi_guard.payload import (
|
||||
PayloadError,
|
||||
PayloadStats,
|
||||
decode_payload,
|
||||
)
|
||||
from python_pkg.syncyomi_guard.verdict import (
|
||||
Thresholds,
|
||||
Verdict,
|
||||
compare,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"PayloadError",
|
||||
"PayloadStats",
|
||||
"Thresholds",
|
||||
"Verdict",
|
||||
"compare",
|
||||
"decode_payload",
|
||||
]
|
||||
@ -1,168 +0,0 @@
|
||||
"""Command line entry point for the SyncYomi integrity guard.
|
||||
|
||||
Exit codes are the contract; a systemd timer and a human both read them:
|
||||
|
||||
* ``0`` — payload healthy (or a first-run baseline was recorded)
|
||||
* ``1`` — payload collapsed, or it could not be read or decoded
|
||||
|
||||
A read failure is deliberately *not* a pass. The whole point of the guard is to
|
||||
be believed on the day something is wrong, and "the database was missing so I
|
||||
said OK" is the fail-open shape that lets a bad copy become the only copy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from python_pkg.syncyomi_guard.payload import PayloadError, decode_payload
|
||||
from python_pkg.syncyomi_guard.store import (
|
||||
StoreError,
|
||||
load_baseline,
|
||||
read_payload,
|
||||
save_baseline,
|
||||
write_snapshot,
|
||||
)
|
||||
from python_pkg.syncyomi_guard.verdict import Status, Thresholds, Verdict, compare
|
||||
|
||||
_DEFAULT_DB = Path("/home/kuhy/services/syncyomi/config/syncyomi.db")
|
||||
_DEFAULT_STATE = Path.home() / ".local/share/syncyomi_guard"
|
||||
_DEFAULT_SNAPSHOTS = Path.home() / "services/syncyomi/snapshots"
|
||||
_DEFAULT_KEEP = 14
|
||||
|
||||
|
||||
def _notify(title: str, body: str) -> None:
|
||||
"""Best-effort desktop notification; never fails the run.
|
||||
|
||||
The exit code is the real signal. A notification is a convenience, so a
|
||||
machine without ``notify-send`` (or a run with no session bus) must not turn
|
||||
a healthy check into a failure.
|
||||
"""
|
||||
binary = shutil.which("notify-send")
|
||||
if binary is None:
|
||||
return
|
||||
try:
|
||||
subprocess.run(
|
||||
[binary, "--urgency=critical", title, body],
|
||||
check=False,
|
||||
timeout=10,
|
||||
)
|
||||
except OSError, subprocess.SubprocessError:
|
||||
return
|
||||
|
||||
|
||||
def _build_parser() -> argparse.ArgumentParser:
|
||||
"""Construct the argument parser."""
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="syncyomi-guard",
|
||||
description=(
|
||||
"Verify the SyncYomi library payload has not silently collapsed, "
|
||||
"and snapshot it as a restorable .tachibk when it is healthy."
|
||||
),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--db",
|
||||
type=Path,
|
||||
default=_DEFAULT_DB,
|
||||
help=f"SyncYomi SQLite database (default: {_DEFAULT_DB})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--state-dir",
|
||||
type=Path,
|
||||
default=_DEFAULT_STATE,
|
||||
help=f"where the baseline is stored (default: {_DEFAULT_STATE})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--snapshot-dir",
|
||||
type=Path,
|
||||
default=_DEFAULT_SNAPSHOTS,
|
||||
help=f"where .tachibk snapshots are written (default: {_DEFAULT_SNAPSHOTS})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--keep",
|
||||
type=int,
|
||||
default=_DEFAULT_KEEP,
|
||||
help=f"how many snapshots to retain (default: {_DEFAULT_KEEP})",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--max-drop",
|
||||
type=float,
|
||||
default=Thresholds().max_drop_ratio,
|
||||
help="fail if any count drops by more than this fraction (default: 0.10)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-snapshot",
|
||||
action="store_true",
|
||||
help="check only; do not write a snapshot or update the baseline",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--accept",
|
||||
action="store_true",
|
||||
help=(
|
||||
"record the current payload as the new baseline even if it looks "
|
||||
"collapsed (use after a deliberate library purge)"
|
||||
),
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def _report(verdict: Verdict) -> None:
|
||||
"""Write the verdict to the appropriate stream.
|
||||
|
||||
A silent guard is a useless one: the journal line is how a human finds out
|
||||
what the timer saw, so the counts are written on every run, not only on
|
||||
failures.
|
||||
"""
|
||||
if verdict.is_failure:
|
||||
sys.stderr.write(f"COLLAPSED: {verdict.reason}\n")
|
||||
if verdict.previous is not None:
|
||||
sys.stderr.write(f" previous: {verdict.previous.describe()}\n")
|
||||
sys.stderr.write(f" current: {verdict.current.describe()}\n")
|
||||
elif verdict.status is Status.FIRST_RUN:
|
||||
sys.stdout.write(f"baseline recorded: {verdict.current.describe()}\n")
|
||||
else:
|
||||
sys.stdout.write(f"ok: {verdict.reason}\n")
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
"""Run the guard, returning the process exit code."""
|
||||
args = _build_parser().parse_args(argv)
|
||||
|
||||
try:
|
||||
blob = read_payload(args.db)
|
||||
stats = decode_payload(blob)
|
||||
except (StoreError, PayloadError) as exc:
|
||||
sys.stderr.write(f"FAILED: {exc}\n")
|
||||
_notify("SyncYomi guard: cannot verify", str(exc))
|
||||
return 1
|
||||
|
||||
try:
|
||||
thresholds = Thresholds(max_drop_ratio=args.max_drop)
|
||||
except ValueError as exc:
|
||||
sys.stderr.write(f"FAILED: {exc}\n")
|
||||
return 1
|
||||
|
||||
verdict = compare(stats, load_baseline(args.state_dir), thresholds)
|
||||
_report(verdict)
|
||||
|
||||
if verdict.is_failure and not args.accept:
|
||||
_notify(
|
||||
"SyncYomi library collapsed",
|
||||
f"{verdict.reason}. Snapshot NOT overwritten; recover from "
|
||||
f"{args.snapshot_dir}.",
|
||||
)
|
||||
return 1
|
||||
|
||||
if not args.no_snapshot:
|
||||
snapshot = write_snapshot(args.snapshot_dir, blob, args.keep)
|
||||
save_baseline(args.state_dir, stats)
|
||||
sys.stdout.write(f"snapshot: {snapshot}\n")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@ -1,179 +0,0 @@
|
||||
"""Minimal protobuf reader for the Tachiyomi/Mihon ``Backup`` message.
|
||||
|
||||
Deliberately dependency-free. The alternative is vendoring Mihon's ``.proto``
|
||||
files and a protobuf runtime, which would couple this guard to the app's schema
|
||||
version — the guard has to keep working when upstream adds a field, because its
|
||||
whole job is to run unattended and be trustworthy on the day something breaks.
|
||||
|
||||
Only the top-level field numbers are interpreted, and every unknown field is
|
||||
skipped by wire type. Field numbers come from Mihon's ``BackupProto``:
|
||||
|
||||
===== ====================== =========================================
|
||||
Field Meaning Why the guard cares
|
||||
===== ====================== =========================================
|
||||
1 ``backupManga`` The library. A collapse here is the alarm.
|
||||
2 ``backupCategories`` Lost silently on 2026-08-09 while manga survived.
|
||||
16 ``chapters`` (nested) Reading progress; catches a shallow restore.
|
||||
101 ``backupSources`` Present even in a degraded payload, so it is
|
||||
*not* evidence of health on its own.
|
||||
===== ====================== =========================================
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
_WIRE_VARINT = 0
|
||||
_WIRE_64BIT = 1
|
||||
_WIRE_LEN = 2
|
||||
_WIRE_32BIT = 5
|
||||
|
||||
_FIELD_MANGA = 1
|
||||
_FIELD_CATEGORIES = 2
|
||||
_FIELD_CHAPTERS = 16
|
||||
_FIELD_SOURCES = 101
|
||||
|
||||
# A varint wider than this is corruption, not a large number. Guards the decode
|
||||
# loop against spinning on malformed input.
|
||||
_MAX_VARINT_BYTES = 10
|
||||
|
||||
|
||||
class PayloadError(ValueError):
|
||||
"""Raised when the payload is not a decodable ``Backup`` message."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PayloadStats:
|
||||
"""What a payload actually contains, as opposed to what it claims.
|
||||
|
||||
``size_bytes`` is kept because it is the cheapest smoke test, but it is
|
||||
never the deciding signal: the 2026-08-09 stub was a perfectly well-formed
|
||||
protobuf message, just an almost empty one.
|
||||
"""
|
||||
|
||||
size_bytes: int
|
||||
manga: int
|
||||
categories: int
|
||||
chapters: int
|
||||
sources: int
|
||||
|
||||
def describe(self) -> str:
|
||||
"""Render a one-line human summary for logs and notifications."""
|
||||
return (
|
||||
f"{self.manga} manga, {self.chapters} chapters, "
|
||||
f"{self.categories} categories, {self.sources} sources "
|
||||
f"({self.size_bytes} bytes)"
|
||||
)
|
||||
|
||||
|
||||
def _read_varint(buf: bytes, pos: int) -> tuple[int, int]:
|
||||
"""Read a base-128 varint, returning the value and the new offset."""
|
||||
result = 0
|
||||
shift = 0
|
||||
start = pos
|
||||
while pos < len(buf):
|
||||
if pos - start >= _MAX_VARINT_BYTES:
|
||||
msg = f"varint at offset {start} is unterminated"
|
||||
raise PayloadError(msg)
|
||||
byte = buf[pos]
|
||||
pos += 1
|
||||
result |= (byte & 0x7F) << shift
|
||||
if not byte & 0x80:
|
||||
return result, pos
|
||||
shift += 7
|
||||
msg = f"varint at offset {start} runs past end of payload"
|
||||
raise PayloadError(msg)
|
||||
|
||||
|
||||
def _skip_field(buf: bytes, pos: int, wire_type: int) -> int:
|
||||
"""Advance past one non-length-delimited field, returning the new offset.
|
||||
|
||||
Length-delimited fields never reach here: both callers must inspect their
|
||||
contents, so they handle that wire type inline.
|
||||
"""
|
||||
if wire_type == _WIRE_VARINT:
|
||||
_, pos = _read_varint(buf, pos)
|
||||
return pos
|
||||
if wire_type == _WIRE_64BIT:
|
||||
return pos + 8
|
||||
if wire_type == _WIRE_32BIT:
|
||||
return pos + 4
|
||||
msg = f"unsupported wire type {wire_type} at offset {pos}"
|
||||
raise PayloadError(msg)
|
||||
|
||||
|
||||
def _count_chapters(entry: bytes) -> int:
|
||||
"""Count ``chapters`` submessages inside one ``BackupManga`` entry.
|
||||
|
||||
Chapters are field 16, not field 3 — field 3 is the title. Getting this
|
||||
wrong yields exactly one "chapter" per manga, which looks plausible enough
|
||||
to pass a careless review, so the distinction is load-bearing.
|
||||
"""
|
||||
chapters = 0
|
||||
pos = 0
|
||||
while pos < len(entry):
|
||||
key, pos = _read_varint(entry, pos)
|
||||
field_number = key >> 3
|
||||
wire_type = key & 0x07
|
||||
if wire_type == _WIRE_LEN:
|
||||
length, pos = _read_varint(entry, pos)
|
||||
end = pos + length
|
||||
if end > len(entry):
|
||||
msg = "chapter field overruns manga entry"
|
||||
raise PayloadError(msg)
|
||||
if field_number == _FIELD_CHAPTERS:
|
||||
chapters += 1
|
||||
pos = end
|
||||
else:
|
||||
pos = _skip_field(entry, pos, wire_type)
|
||||
return chapters
|
||||
|
||||
|
||||
def decode_payload(blob: bytes) -> PayloadStats:
|
||||
"""Decode a ``Backup`` payload into verifiable counts.
|
||||
|
||||
The payload must be consumed exactly: trailing bytes mean the message was
|
||||
truncated or spliced, which is precisely the corruption a page-level
|
||||
recovery can produce, so it is treated as an error rather than ignored.
|
||||
|
||||
Raises:
|
||||
PayloadError: If the payload is empty, malformed, or not fully consumed.
|
||||
"""
|
||||
if not blob:
|
||||
msg = "payload is empty"
|
||||
raise PayloadError(msg)
|
||||
|
||||
manga = categories = chapters = sources = 0
|
||||
pos = 0
|
||||
while pos < len(blob):
|
||||
key, pos = _read_varint(blob, pos)
|
||||
field_number = key >> 3
|
||||
wire_type = key & 0x07
|
||||
if wire_type == _WIRE_LEN:
|
||||
length, pos = _read_varint(blob, pos)
|
||||
end = pos + length
|
||||
if end > len(blob):
|
||||
msg = f"field {field_number} overruns payload"
|
||||
raise PayloadError(msg)
|
||||
if field_number == _FIELD_MANGA:
|
||||
manga += 1
|
||||
chapters += _count_chapters(blob[pos:end])
|
||||
elif field_number == _FIELD_CATEGORIES:
|
||||
categories += 1
|
||||
elif field_number == _FIELD_SOURCES:
|
||||
sources += 1
|
||||
pos = end
|
||||
else:
|
||||
pos = _skip_field(blob, pos, wire_type)
|
||||
|
||||
if pos != len(blob):
|
||||
msg = f"payload not fully consumed ({pos} of {len(blob)} bytes)"
|
||||
raise PayloadError(msg)
|
||||
|
||||
return PayloadStats(
|
||||
size_bytes=len(blob),
|
||||
manga=manga,
|
||||
categories=categories,
|
||||
chapters=chapters,
|
||||
sources=sources,
|
||||
)
|
||||
@ -1,182 +0,0 @@
|
||||
"""Read the live payload and keep versioned snapshots of the good ones.
|
||||
|
||||
Two rules shape this module, both learned on 2026-08-09:
|
||||
|
||||
* The database is opened **read-only**. The guard runs on a timer against a
|
||||
database a container is actively writing; it must never be the reason
|
||||
SyncYomi sees a lock.
|
||||
* A snapshot is written only for a payload that passed. Snapshotting first and
|
||||
judging afterwards would have faithfully archived the 267 KB stub.
|
||||
|
||||
Snapshots are gzipped ``Backup`` messages, which is exactly the ``.tachibk``
|
||||
format the app restores from — so recovery is "copy this file to the phone",
|
||||
with no decoding step at the moment it is needed most.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict
|
||||
from datetime import UTC, datetime
|
||||
import gzip
|
||||
import json
|
||||
import sqlite3
|
||||
from typing import TYPE_CHECKING, Final
|
||||
|
||||
from python_pkg.syncyomi_guard.payload import PayloadError, PayloadStats
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
_STATE_FILENAME: Final = "last_known_good.json"
|
||||
|
||||
# The glob matches the timestamp layout *only*. A looser pattern such as
|
||||
# ``syncyomi_*.tachibk`` would also match a hand-named file like
|
||||
# ``syncyomi_recovered_2026-08-09.tachibk`` — the irreplaceable original
|
||||
# rescued from the WAL — and the pruner would delete it as a stale snapshot.
|
||||
_SNAPSHOT_GLOB: Final = "syncyomi_[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]T*Z.tachibk"
|
||||
_SNAPSHOT_STEM: Final = "syncyomi_%Y-%m-%dT%H-%M-%SZ"
|
||||
|
||||
|
||||
class StoreError(RuntimeError):
|
||||
"""Raised when the payload cannot be read from the SyncYomi database."""
|
||||
|
||||
|
||||
def read_payload(db_path: Path) -> bytes:
|
||||
"""Read the single ``sync_data`` blob from a SyncYomi SQLite database.
|
||||
|
||||
Opened through an immutable URI: no locking, no WAL recovery, no write to
|
||||
the directory. The database stays exactly as the container left it.
|
||||
|
||||
Raises:
|
||||
StoreError: If the database is unreadable or holds no sync row.
|
||||
"""
|
||||
if not db_path.is_file():
|
||||
msg = f"no SyncYomi database at {db_path}"
|
||||
raise StoreError(msg)
|
||||
|
||||
# ``sqlite3.connect`` as a context manager commits the transaction but does
|
||||
# NOT close the connection, so it is closed explicitly. A guard that leaks
|
||||
# handles against the live database would eventually become the thing
|
||||
# holding a file open on it.
|
||||
uri = f"file:{db_path}?immutable=1"
|
||||
try:
|
||||
conn = sqlite3.connect(uri, uri=True)
|
||||
except sqlite3.Error as exc:
|
||||
msg = f"cannot read {db_path}: {exc}"
|
||||
raise StoreError(msg) from exc
|
||||
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT data FROM sync_data ORDER BY id LIMIT 1",
|
||||
).fetchall()
|
||||
except sqlite3.Error as exc:
|
||||
msg = f"cannot read {db_path}: {exc}"
|
||||
raise StoreError(msg) from exc
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
if not rows:
|
||||
msg = f"{db_path} has no sync_data row"
|
||||
raise StoreError(msg)
|
||||
|
||||
blob = rows[0][0]
|
||||
if not isinstance(blob, bytes):
|
||||
msg = f"sync_data.data is {type(blob).__name__}, expected bytes"
|
||||
raise StoreError(msg)
|
||||
return blob
|
||||
|
||||
|
||||
def load_baseline(state_dir: Path) -> PayloadStats | None:
|
||||
"""Load the last known good stats, or ``None`` when there is no baseline.
|
||||
|
||||
A corrupt or partial state file is treated as absent rather than fatal: the
|
||||
guard then re-baselines loudly (``FIRST_RUN``) instead of refusing to run.
|
||||
"""
|
||||
path = state_dir / _STATE_FILENAME
|
||||
if not path.is_file():
|
||||
return None
|
||||
try:
|
||||
raw = json.loads(path.read_text(encoding="utf-8"))
|
||||
return PayloadStats(
|
||||
size_bytes=int(raw["size_bytes"]),
|
||||
manga=int(raw["manga"]),
|
||||
categories=int(raw["categories"]),
|
||||
chapters=int(raw["chapters"]),
|
||||
sources=int(raw["sources"]),
|
||||
)
|
||||
except OSError, ValueError, KeyError, TypeError:
|
||||
return None
|
||||
|
||||
|
||||
def save_baseline(state_dir: Path, stats: PayloadStats) -> Path:
|
||||
"""Record ``stats`` as the new last known good baseline."""
|
||||
state_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = state_dir / _STATE_FILENAME
|
||||
payload = dict(asdict(stats))
|
||||
payload["recorded_at"] = datetime.now(UTC).isoformat()
|
||||
path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def write_snapshot(
|
||||
snapshot_dir: Path,
|
||||
blob: bytes,
|
||||
keep: int,
|
||||
now: datetime | None = None,
|
||||
) -> Path:
|
||||
"""Write ``blob`` as a restorable ``.tachibk`` and prune old snapshots.
|
||||
|
||||
Raises:
|
||||
ValueError: If ``keep`` is not at least 1.
|
||||
"""
|
||||
if keep < 1:
|
||||
msg = f"keep must be >= 1, got {keep}"
|
||||
raise ValueError(msg)
|
||||
|
||||
snapshot_dir.mkdir(parents=True, exist_ok=True)
|
||||
stamp = (now or datetime.now(UTC)).strftime(_SNAPSHOT_STEM)
|
||||
|
||||
# Timestamps are second-granular, so two runs in the same second would map
|
||||
# to one filename and the second would silently overwrite the first —
|
||||
# quietly holding fewer than ``keep`` distinct snapshots, and in the worst
|
||||
# case replacing a good library with a stub written a fraction of a second
|
||||
# later. Suffix on collision rather than clobber.
|
||||
path = snapshot_dir / f"{stamp}.tachibk"
|
||||
suffix = 1
|
||||
while path.exists():
|
||||
path = snapshot_dir / f"{stamp}-{suffix}.tachibk"
|
||||
suffix += 1
|
||||
|
||||
path.write_bytes(gzip.compress(blob))
|
||||
_prune(snapshot_dir, keep)
|
||||
return path
|
||||
|
||||
|
||||
def _prune(snapshot_dir: Path, keep: int) -> None:
|
||||
"""Delete all but the ``keep`` newest snapshots, by filename order.
|
||||
|
||||
Filenames are UTC timestamps in a sortable layout, so lexical order is
|
||||
chronological order without stat-ing every file.
|
||||
"""
|
||||
snapshots = sorted(snapshot_dir.glob(_SNAPSHOT_GLOB))
|
||||
for stale in snapshots[:-keep]:
|
||||
stale.unlink()
|
||||
|
||||
|
||||
def latest_snapshot(snapshot_dir: Path) -> Path | None:
|
||||
"""Return the newest snapshot, or ``None`` when none exist."""
|
||||
snapshots = sorted(snapshot_dir.glob(_SNAPSHOT_GLOB))
|
||||
return snapshots[-1] if snapshots else None
|
||||
|
||||
|
||||
def read_snapshot(path: Path) -> bytes:
|
||||
"""Decompress a ``.tachibk`` snapshot back into a raw payload.
|
||||
|
||||
Raises:
|
||||
PayloadError: If the file is not readable gzip.
|
||||
"""
|
||||
try:
|
||||
return gzip.decompress(path.read_bytes())
|
||||
except (OSError, gzip.BadGzipFile) as exc:
|
||||
msg = f"cannot read snapshot {path}: {exc}"
|
||||
raise PayloadError(msg) from exc
|
||||
@ -1,93 +0,0 @@
|
||||
"""Shared protobuf builders for the guard's tests.
|
||||
|
||||
The tests construct real ``Backup`` wire format rather than mocking the decoder.
|
||||
A mock would happily agree with a decoder that counts the wrong field number,
|
||||
which is the specific bug this package had while being written (chapters are
|
||||
field 16; field 3 is the title, and counting it yields exactly one "chapter"
|
||||
per manga).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
_WIRE_VARINT = 0
|
||||
_WIRE_LEN = 2
|
||||
|
||||
|
||||
def encode_varint(value: int) -> bytes:
|
||||
"""Encode ``value`` as a base-128 varint."""
|
||||
out = bytearray()
|
||||
while True:
|
||||
byte = value & 0x7F
|
||||
value >>= 7
|
||||
if value:
|
||||
out.append(byte | 0x80)
|
||||
else:
|
||||
out.append(byte)
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def tag(field_number: int, wire_type: int) -> bytes:
|
||||
"""Encode a protobuf field tag."""
|
||||
return encode_varint((field_number << 3) | wire_type)
|
||||
|
||||
|
||||
def length_delimited(field_number: int, payload: bytes) -> bytes:
|
||||
"""Encode a length-delimited field."""
|
||||
return tag(field_number, _WIRE_LEN) + encode_varint(len(payload)) + payload
|
||||
|
||||
|
||||
def varint_field(field_number: int, value: int) -> bytes:
|
||||
"""Encode a varint field."""
|
||||
return tag(field_number, _WIRE_VARINT) + encode_varint(value)
|
||||
|
||||
|
||||
def chapter(url: str = "/chapter/x") -> bytes:
|
||||
"""Build one ``chapters`` submessage (field 16 of ``BackupManga``)."""
|
||||
return length_delimited(16, length_delimited(1, url.encode()))
|
||||
|
||||
|
||||
def manga(*, url: str = "/manga/x", chapters: int = 0, title: str = "T") -> bytes:
|
||||
"""Build one ``BackupManga`` entry (field 1 of ``Backup``)."""
|
||||
body = length_delimited(2, url.encode()) + length_delimited(3, title.encode())
|
||||
body += b"".join(chapter(f"{url}/c{i}") for i in range(chapters))
|
||||
return length_delimited(1, body)
|
||||
|
||||
|
||||
def category(name: str) -> bytes:
|
||||
"""Build one ``BackupCategory`` entry (field 2 of ``Backup``)."""
|
||||
return length_delimited(2, length_delimited(1, name.encode()))
|
||||
|
||||
|
||||
def source(name: str = "MangaDex") -> bytes:
|
||||
"""Build one ``BackupSource`` entry (field 101 of ``Backup``)."""
|
||||
return length_delimited(101, length_delimited(1, name.encode()))
|
||||
|
||||
|
||||
def backup(
|
||||
*,
|
||||
manga_count: int = 0,
|
||||
chapters_each: int = 0,
|
||||
categories: int = 0,
|
||||
sources: int = 0,
|
||||
) -> bytes:
|
||||
"""Build a whole ``Backup`` message with the requested shape."""
|
||||
parts = [
|
||||
manga(url=f"/manga/{i}", chapters=chapters_each) for i in range(manga_count)
|
||||
]
|
||||
parts += [category(f"cat{i}") for i in range(categories)]
|
||||
parts += [source(f"src{i}") for i in range(sources)]
|
||||
return b"".join(parts)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def healthy_payload() -> bytes:
|
||||
"""A payload standing in for a normal library."""
|
||||
return backup(manga_count=10, chapters_each=3, categories=4, sources=2)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stub_payload() -> bytes:
|
||||
"""The 2026-08-09 failure shape: sources survive, the library does not."""
|
||||
return backup(manga_count=0, categories=0, sources=2)
|
||||
@ -1,216 +0,0 @@
|
||||
"""Tests for the CLI, whose exit code is the contract systemd reads.
|
||||
|
||||
The rule under test throughout: the guard fails closed. Anything it cannot
|
||||
verify is a failure, because "I could not check, so everything is fine" is how
|
||||
a degraded copy becomes the only copy.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from python_pkg.syncyomi_guard.__main__ import main
|
||||
from python_pkg.syncyomi_guard.store import load_baseline
|
||||
from python_pkg.syncyomi_guard.tests.conftest import backup
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def _write_db(path: Path, blob: bytes) -> None:
|
||||
conn = sqlite3.connect(path)
|
||||
try:
|
||||
with conn:
|
||||
conn.execute("CREATE TABLE sync_data (id INTEGER PRIMARY KEY, data BLOB)")
|
||||
conn.execute("INSERT INTO sync_data (id, data) VALUES (1, ?)", (blob,))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def _args(tmp_path: Path, db: Path, *extra: str) -> Sequence[str]:
|
||||
return [
|
||||
"--db",
|
||||
str(db),
|
||||
"--state-dir",
|
||||
str(tmp_path / "state"),
|
||||
"--snapshot-dir",
|
||||
str(tmp_path / "snaps"),
|
||||
*extra,
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_notifications(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Never fire a real desktop notification from a test."""
|
||||
monkeypatch.setattr(
|
||||
"python_pkg.syncyomi_guard.__main__.shutil.which", lambda _: None
|
||||
)
|
||||
|
||||
|
||||
def test_first_run_records_a_baseline_and_snapshot(tmp_path: Path) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, backup(manga_count=10, chapters_each=2, categories=3))
|
||||
|
||||
assert main(_args(tmp_path, db)) == 0
|
||||
assert load_baseline(tmp_path / "state") is not None
|
||||
assert list((tmp_path / "snaps").glob("*.tachibk"))
|
||||
|
||||
|
||||
def test_healthy_second_run_passes(tmp_path: Path) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, backup(manga_count=10, chapters_each=2, categories=3))
|
||||
assert main(_args(tmp_path, db)) == 0
|
||||
assert main(_args(tmp_path, db)) == 0
|
||||
|
||||
|
||||
def test_collapse_fails_and_preserves_the_good_snapshot(tmp_path: Path) -> None:
|
||||
"""The 2026-08-09 scenario, end to end through the binary."""
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, backup(manga_count=100, chapters_each=5, categories=8))
|
||||
assert main(_args(tmp_path, db)) == 0
|
||||
|
||||
good = sorted((tmp_path / "snaps").glob("*.tachibk"))
|
||||
assert len(good) == 1
|
||||
good_bytes = good[0].read_bytes()
|
||||
|
||||
# The library collapses to a sources-only stub, exactly as it did that night.
|
||||
db.unlink()
|
||||
_write_db(db, backup(manga_count=0, sources=2))
|
||||
|
||||
assert main(_args(tmp_path, db)) == 1
|
||||
|
||||
after = sorted((tmp_path / "snaps").glob("*.tachibk"))
|
||||
assert len(after) == 1, "a collapsed payload must not be snapshotted"
|
||||
assert after[0].read_bytes() == good_bytes
|
||||
baseline = load_baseline(tmp_path / "state")
|
||||
assert baseline is not None
|
||||
assert baseline.manga == 100, "baseline must not absorb the collapse"
|
||||
|
||||
|
||||
def test_accept_overrides_a_deliberate_purge(tmp_path: Path) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, backup(manga_count=100, chapters_each=2, categories=4))
|
||||
assert main(_args(tmp_path, db)) == 0
|
||||
|
||||
db.unlink()
|
||||
_write_db(db, backup(manga_count=1, sources=1))
|
||||
|
||||
assert main(_args(tmp_path, db, "--accept")) == 0
|
||||
baseline = load_baseline(tmp_path / "state")
|
||||
assert baseline is not None
|
||||
assert baseline.manga == 1
|
||||
|
||||
|
||||
def test_no_snapshot_checks_without_writing(tmp_path: Path) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, backup(manga_count=5, chapters_each=1))
|
||||
assert main(_args(tmp_path, db, "--no-snapshot")) == 0
|
||||
assert not (tmp_path / "snaps").exists()
|
||||
assert load_baseline(tmp_path / "state") is None
|
||||
|
||||
|
||||
def test_missing_database_exits_nonzero(tmp_path: Path) -> None:
|
||||
"""Fail closed rather than reporting a pass for an unreadable database."""
|
||||
assert main(_args(tmp_path, tmp_path / "absent.db")) == 1
|
||||
|
||||
|
||||
def test_undecodable_payload_exits_nonzero(tmp_path: Path) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, b"\xff\xff\xff not protobuf")
|
||||
assert main(_args(tmp_path, db)) == 1
|
||||
|
||||
|
||||
def test_invalid_threshold_exits_nonzero(tmp_path: Path) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, backup(manga_count=1))
|
||||
assert main(_args(tmp_path, db, "--max-drop", "5")) == 1
|
||||
|
||||
|
||||
def test_collapse_message_names_the_loss(
|
||||
tmp_path: Path,
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_write_db(db, backup(manga_count=50, chapters_each=2, categories=6))
|
||||
main(_args(tmp_path, db))
|
||||
capsys.readouterr()
|
||||
|
||||
db.unlink()
|
||||
_write_db(db, backup(manga_count=50, chapters_each=2, categories=0))
|
||||
main(_args(tmp_path, db))
|
||||
|
||||
err = capsys.readouterr().err
|
||||
assert "COLLAPSED" in err
|
||||
assert "categories went from 6 to 0" in err
|
||||
|
||||
|
||||
def test_report_handles_a_collapse_without_a_baseline(
|
||||
capsys: pytest.CaptureFixture[str],
|
||||
) -> None:
|
||||
"""Defensive branch: ``compare`` never returns this, but ``_report`` must cope.
|
||||
|
||||
A COLLAPSED verdict always carries the baseline it was measured against, so
|
||||
this shape is unreachable through ``main``. It is covered directly so the
|
||||
guard cannot crash while reporting the one thing it exists to report.
|
||||
"""
|
||||
from python_pkg.syncyomi_guard.__main__ import _report
|
||||
from python_pkg.syncyomi_guard.payload import PayloadStats
|
||||
from python_pkg.syncyomi_guard.verdict import Status, Verdict
|
||||
|
||||
stats = PayloadStats(
|
||||
size_bytes=1,
|
||||
manga=0,
|
||||
categories=0,
|
||||
chapters=0,
|
||||
sources=0,
|
||||
)
|
||||
_report(
|
||||
Verdict(
|
||||
status=Status.COLLAPSED,
|
||||
reason="synthetic",
|
||||
current=stats,
|
||||
previous=None,
|
||||
),
|
||||
)
|
||||
err = capsys.readouterr().err
|
||||
assert "COLLAPSED" in err
|
||||
assert "previous:" not in err
|
||||
|
||||
|
||||
def test_notification_is_attempted_on_failure(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A failure should reach the desktop, but must not depend on it."""
|
||||
calls: list[list[str]] = []
|
||||
monkeypatch.setattr(
|
||||
"python_pkg.syncyomi_guard.__main__.shutil.which",
|
||||
lambda _: "/usr/bin/notify-send",
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"python_pkg.syncyomi_guard.__main__.subprocess.run",
|
||||
lambda cmd, **_: calls.append(cmd),
|
||||
)
|
||||
assert main(_args(tmp_path, tmp_path / "absent.db")) == 1
|
||||
assert calls
|
||||
assert "notify-send" in calls[0][0]
|
||||
|
||||
|
||||
def test_notification_failure_does_not_mask_the_verdict(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
def explode(*_: object, **__: object) -> None:
|
||||
msg = "no session bus"
|
||||
raise OSError(msg)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"python_pkg.syncyomi_guard.__main__.shutil.which",
|
||||
lambda _: "/usr/bin/notify-send",
|
||||
)
|
||||
monkeypatch.setattr("python_pkg.syncyomi_guard.__main__.subprocess.run", explode)
|
||||
assert main(_args(tmp_path, tmp_path / "absent.db")) == 1
|
||||
@ -1,119 +0,0 @@
|
||||
"""Tests for the ``Backup`` protobuf reader.
|
||||
|
||||
The guarantee under test is that the counts are *true*. A decoder that silently
|
||||
undercounts is worse than no guard at all, because it would report OK while the
|
||||
library drained away.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from python_pkg.syncyomi_guard.payload import PayloadError, decode_payload
|
||||
from python_pkg.syncyomi_guard.tests.conftest import (
|
||||
backup,
|
||||
encode_varint,
|
||||
length_delimited,
|
||||
manga,
|
||||
tag,
|
||||
varint_field,
|
||||
)
|
||||
|
||||
|
||||
def test_counts_every_top_level_field(healthy_payload: bytes) -> None:
|
||||
stats = decode_payload(healthy_payload)
|
||||
assert stats.manga == 10
|
||||
assert stats.chapters == 30
|
||||
assert stats.categories == 4
|
||||
assert stats.sources == 2
|
||||
assert stats.size_bytes == len(healthy_payload)
|
||||
|
||||
|
||||
def test_chapters_come_from_field_16_not_the_title(stub_payload: bytes) -> None:
|
||||
"""A title-only manga has zero chapters, not one.
|
||||
|
||||
Field 3 is the title and field 16 is a chapter. Counting field 3 yields
|
||||
exactly one chapter per manga, which looks plausible in aggregate.
|
||||
"""
|
||||
stats = decode_payload(manga(url="/manga/a", chapters=0, title="A Long Title"))
|
||||
assert stats.manga == 1
|
||||
assert stats.chapters == 0
|
||||
assert decode_payload(stub_payload).manga == 0
|
||||
|
||||
|
||||
def test_the_2026_08_09_stub_reports_an_empty_library(stub_payload: bytes) -> None:
|
||||
"""Sources present, library absent — the payload that overwrote the good one."""
|
||||
stats = decode_payload(stub_payload)
|
||||
assert stats.manga == 0
|
||||
assert stats.chapters == 0
|
||||
assert stats.categories == 0
|
||||
assert stats.sources == 2
|
||||
|
||||
|
||||
def test_unknown_fields_of_every_wire_type_are_skipped() -> None:
|
||||
"""Upstream may add fields; the guard must not break when it does."""
|
||||
payload = (
|
||||
manga(url="/manga/a", chapters=1)
|
||||
+ varint_field(900, 12345)
|
||||
+ tag(901, 5)
|
||||
+ b"\x00\x01\x02\x03"
|
||||
+ tag(902, 1)
|
||||
+ b"\x00\x01\x02\x03\x04\x05\x06\x07"
|
||||
+ length_delimited(903, b"opaque")
|
||||
)
|
||||
stats = decode_payload(payload)
|
||||
assert stats.manga == 1
|
||||
assert stats.chapters == 1
|
||||
|
||||
|
||||
def test_unknown_wire_type_inside_a_manga_entry_is_skipped() -> None:
|
||||
"""Exercises the non-length branch of the per-entry chapter scan."""
|
||||
body = length_delimited(2, b"/manga/a") + varint_field(8, 1) + tag(9, 5)
|
||||
payload = length_delimited(1, body + b"\x00\x01\x02\x03")
|
||||
assert decode_payload(payload).manga == 1
|
||||
|
||||
|
||||
def test_empty_payload_is_rejected() -> None:
|
||||
with pytest.raises(PayloadError, match="empty"):
|
||||
decode_payload(b"")
|
||||
|
||||
|
||||
def test_truncated_length_prefix_is_rejected() -> None:
|
||||
"""A field claiming more bytes than exist is corruption, not a short read."""
|
||||
payload = tag(1, 2) + encode_varint(500) + b"too short"
|
||||
with pytest.raises(PayloadError, match="overruns"):
|
||||
decode_payload(payload)
|
||||
|
||||
|
||||
def test_truncated_chapter_inside_a_manga_is_rejected() -> None:
|
||||
body = length_delimited(2, b"/manga/a") + tag(16, 2) + encode_varint(400) + b"xx"
|
||||
with pytest.raises(PayloadError, match="overruns manga entry"):
|
||||
decode_payload(length_delimited(1, body))
|
||||
|
||||
|
||||
def test_unterminated_varint_is_rejected() -> None:
|
||||
with pytest.raises(PayloadError, match="unterminated"):
|
||||
decode_payload(b"\xff" * 12)
|
||||
|
||||
|
||||
def test_varint_running_off_the_end_is_rejected() -> None:
|
||||
with pytest.raises(PayloadError, match="past end"):
|
||||
decode_payload(b"\xff\xff")
|
||||
|
||||
|
||||
def test_unsupported_wire_type_is_rejected() -> None:
|
||||
with pytest.raises(PayloadError, match="unsupported wire type"):
|
||||
decode_payload(tag(1, 3))
|
||||
|
||||
|
||||
def test_trailing_bytes_are_rejected() -> None:
|
||||
"""Splice damage from page-level recovery must not decode as success."""
|
||||
payload = manga(url="/manga/a") + tag(2, 1) + b"\x00" * 4
|
||||
with pytest.raises(PayloadError, match="not fully consumed"):
|
||||
decode_payload(payload)
|
||||
|
||||
|
||||
def test_describe_is_human_readable() -> None:
|
||||
text = decode_payload(backup(manga_count=2, chapters_each=1)).describe()
|
||||
assert "2 manga" in text
|
||||
assert "2 chapters" in text
|
||||
@ -1,239 +0,0 @@
|
||||
"""Tests for reading the payload and keeping snapshots.
|
||||
|
||||
Two behaviours here are load-bearing and easy to regress: the database is
|
||||
opened without taking a lock, and pruning never deletes more than it should.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import sqlite3
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import pytest
|
||||
|
||||
from python_pkg.syncyomi_guard.payload import PayloadError, PayloadStats
|
||||
from python_pkg.syncyomi_guard.store import (
|
||||
StoreError,
|
||||
latest_snapshot,
|
||||
load_baseline,
|
||||
read_payload,
|
||||
read_snapshot,
|
||||
save_baseline,
|
||||
write_snapshot,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
_STATS = PayloadStats(
|
||||
size_bytes=100,
|
||||
manga=10,
|
||||
categories=2,
|
||||
chapters=30,
|
||||
sources=3,
|
||||
)
|
||||
|
||||
|
||||
def _make_db(path: Path, blob: object) -> None:
|
||||
"""Create a minimal SyncYomi-shaped database holding ``blob``."""
|
||||
conn = sqlite3.connect(path)
|
||||
try:
|
||||
with conn:
|
||||
conn.execute("CREATE TABLE sync_data (id INTEGER PRIMARY KEY, data BLOB)")
|
||||
conn.execute("INSERT INTO sync_data (id, data) VALUES (1, ?)", (blob,))
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_reads_the_payload_blob(tmp_path: Path) -> None:
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_make_db(db, b"payload-bytes")
|
||||
assert read_payload(db) == b"payload-bytes"
|
||||
|
||||
|
||||
def test_missing_database_is_an_error_not_an_empty_payload(tmp_path: Path) -> None:
|
||||
"""Fail closed: a missing database must never read as "nothing to worry about"."""
|
||||
with pytest.raises(StoreError, match="no SyncYomi database"):
|
||||
read_payload(tmp_path / "absent.db")
|
||||
|
||||
|
||||
def test_database_without_the_table_is_an_error(tmp_path: Path) -> None:
|
||||
db = tmp_path / "empty.db"
|
||||
conn = sqlite3.connect(db)
|
||||
try:
|
||||
with conn:
|
||||
conn.execute("CREATE TABLE unrelated (id INTEGER)")
|
||||
finally:
|
||||
conn.close()
|
||||
with pytest.raises(StoreError, match="cannot read"):
|
||||
read_payload(db)
|
||||
|
||||
|
||||
def test_database_with_no_rows_is_an_error(tmp_path: Path) -> None:
|
||||
db = tmp_path / "norows.db"
|
||||
conn = sqlite3.connect(db)
|
||||
try:
|
||||
with conn:
|
||||
conn.execute("CREATE TABLE sync_data (id INTEGER PRIMARY KEY, data BLOB)")
|
||||
finally:
|
||||
conn.close()
|
||||
with pytest.raises(StoreError, match="no sync_data row"):
|
||||
read_payload(db)
|
||||
|
||||
|
||||
def test_unopenable_database_is_an_error(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A connect-time failure must surface, not propagate as a raw sqlite error."""
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_make_db(db, b"payload")
|
||||
|
||||
def refuse(*_: object, **__: object) -> object:
|
||||
msg = "unable to open database file"
|
||||
raise sqlite3.OperationalError(msg)
|
||||
|
||||
monkeypatch.setattr(
|
||||
"python_pkg.syncyomi_guard.store.sqlite3.connect",
|
||||
refuse,
|
||||
)
|
||||
with pytest.raises(StoreError, match="cannot read"):
|
||||
read_payload(db)
|
||||
|
||||
|
||||
def test_non_blob_payload_is_an_error(tmp_path: Path) -> None:
|
||||
db = tmp_path / "text.db"
|
||||
_make_db(db, "not bytes")
|
||||
with pytest.raises(StoreError, match="expected bytes"):
|
||||
read_payload(db)
|
||||
|
||||
|
||||
def test_reading_does_not_lock_the_database(tmp_path: Path) -> None:
|
||||
"""The guard runs against a database the container is writing.
|
||||
|
||||
An exclusive read would make the guard a cause of the very ``SQLITE_BUSY``
|
||||
class of failure it exists to detect.
|
||||
"""
|
||||
db = tmp_path / "syncyomi.db"
|
||||
_make_db(db, b"payload")
|
||||
writer = sqlite3.connect(db)
|
||||
try:
|
||||
writer.execute("BEGIN IMMEDIATE")
|
||||
writer.execute("UPDATE sync_data SET data = ? WHERE id = 1", (b"changed",))
|
||||
assert read_payload(db) == b"payload"
|
||||
writer.rollback()
|
||||
finally:
|
||||
writer.close()
|
||||
|
||||
|
||||
def test_baseline_round_trips(tmp_path: Path) -> None:
|
||||
save_baseline(tmp_path, _STATS)
|
||||
assert load_baseline(tmp_path) == _STATS
|
||||
|
||||
|
||||
def test_absent_baseline_is_none(tmp_path: Path) -> None:
|
||||
assert load_baseline(tmp_path) is None
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"content",
|
||||
["not json at all", '{"manga": 1}', '{"size_bytes": "x"}', "[]"],
|
||||
)
|
||||
def test_corrupt_baseline_reads_as_absent(tmp_path: Path, content: str) -> None:
|
||||
"""A damaged state file must re-baseline loudly, not crash the timer."""
|
||||
(tmp_path / "last_known_good.json").write_text(content, encoding="utf-8")
|
||||
assert load_baseline(tmp_path) is None
|
||||
|
||||
|
||||
def test_snapshot_is_a_restorable_tachibk(tmp_path: Path) -> None:
|
||||
"""The snapshot must be exactly what the app restores from."""
|
||||
path = write_snapshot(tmp_path, b"library-bytes", keep=5)
|
||||
assert path.suffix == ".tachibk"
|
||||
assert gzip.decompress(path.read_bytes()) == b"library-bytes"
|
||||
assert read_snapshot(path) == b"library-bytes"
|
||||
|
||||
|
||||
def test_snapshot_pruning_keeps_the_newest(tmp_path: Path) -> None:
|
||||
from datetime import UTC, datetime
|
||||
|
||||
def at(day: int) -> datetime:
|
||||
return datetime(2026, 8, day, 12, 0, tzinfo=UTC)
|
||||
|
||||
for day in range(1, 6):
|
||||
write_snapshot(tmp_path, f"day{day}".encode(), keep=3, now=at(day))
|
||||
|
||||
remaining = sorted(p.name for p in tmp_path.glob("*.tachibk"))
|
||||
assert len(remaining) == 3
|
||||
assert "2026-08-05" in remaining[-1]
|
||||
assert read_snapshot(tmp_path / remaining[-1]) == b"day5"
|
||||
|
||||
|
||||
def test_two_snapshots_in_the_same_second_do_not_overwrite(tmp_path: Path) -> None:
|
||||
"""Second-granular timestamps must not silently collapse two snapshots.
|
||||
|
||||
Overwriting would quietly keep fewer than ``keep`` snapshots, and in the
|
||||
worst case replace a good library with a stub written a fraction of a
|
||||
second later — the exact loss this package exists to prevent.
|
||||
"""
|
||||
from datetime import UTC, datetime
|
||||
|
||||
same_second = datetime(2026, 8, 9, 23, 45, 30, tzinfo=UTC)
|
||||
first = write_snapshot(tmp_path, b"good-library", keep=14, now=same_second)
|
||||
second = write_snapshot(tmp_path, b"degraded-stub", keep=14, now=same_second)
|
||||
|
||||
assert first != second
|
||||
assert read_snapshot(first) == b"good-library"
|
||||
assert read_snapshot(second) == b"degraded-stub"
|
||||
assert len(list(tmp_path.glob("*.tachibk"))) == 2
|
||||
|
||||
|
||||
def test_pruning_never_deletes_a_hand_named_recovery_file(tmp_path: Path) -> None:
|
||||
"""Regression guard for a real near-miss.
|
||||
|
||||
``syncyomi_recovered_2026-08-09.tachibk`` was the only surviving copy of a
|
||||
2182-manga library, rebuilt from the SQLite WAL. A ``syncyomi_*.tachibk``
|
||||
prune glob would have matched and deleted it.
|
||||
"""
|
||||
from datetime import UTC, datetime
|
||||
|
||||
precious = tmp_path / "syncyomi_recovered_2026-08-09.tachibk"
|
||||
precious.write_bytes(b"irreplaceable")
|
||||
|
||||
for day in range(1, 6):
|
||||
write_snapshot(
|
||||
tmp_path,
|
||||
b"routine",
|
||||
keep=1,
|
||||
now=datetime(2026, 8, day, tzinfo=UTC),
|
||||
)
|
||||
|
||||
assert precious.exists()
|
||||
assert precious.read_bytes() == b"irreplaceable"
|
||||
|
||||
|
||||
def test_keep_must_be_positive(tmp_path: Path) -> None:
|
||||
with pytest.raises(ValueError, match="keep must be"):
|
||||
write_snapshot(tmp_path, b"x", keep=0)
|
||||
|
||||
|
||||
def test_latest_snapshot_reports_none_when_empty(tmp_path: Path) -> None:
|
||||
assert latest_snapshot(tmp_path) is None
|
||||
|
||||
|
||||
def test_latest_snapshot_finds_the_newest(tmp_path: Path) -> None:
|
||||
from datetime import UTC, datetime
|
||||
|
||||
write_snapshot(tmp_path, b"old", keep=9, now=datetime(2026, 8, 1, tzinfo=UTC))
|
||||
write_snapshot(tmp_path, b"new", keep=9, now=datetime(2026, 8, 9, tzinfo=UTC))
|
||||
newest = latest_snapshot(tmp_path)
|
||||
assert newest is not None
|
||||
assert read_snapshot(newest) == b"new"
|
||||
|
||||
|
||||
def test_unreadable_snapshot_is_an_error(tmp_path: Path) -> None:
|
||||
bad = tmp_path / "syncyomi_broken.tachibk"
|
||||
bad.write_bytes(b"not gzip")
|
||||
with pytest.raises(PayloadError, match="cannot read snapshot"):
|
||||
read_snapshot(bad)
|
||||
@ -1,152 +0,0 @@
|
||||
"""Tests for the collapse-detection rules.
|
||||
|
||||
Anchored on the real 2026-08-09 numbers so the thresholds are judged against
|
||||
what actually happened, not against invented ones.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from python_pkg.syncyomi_guard.payload import PayloadStats
|
||||
from python_pkg.syncyomi_guard.verdict import Status, Thresholds, compare
|
||||
|
||||
# The library as it stood before the failed restore.
|
||||
_GOOD = PayloadStats(
|
||||
size_bytes=14_083_674,
|
||||
manga=2182,
|
||||
categories=18,
|
||||
chapters=75_376,
|
||||
sources=24,
|
||||
)
|
||||
|
||||
# What the "successful" restore pushed to the server at 22:53.
|
||||
_STUB = PayloadStats(
|
||||
size_bytes=267_737,
|
||||
manga=0,
|
||||
categories=0,
|
||||
chapters=0,
|
||||
sources=24,
|
||||
)
|
||||
|
||||
# After the good restore: manga intact, duplicate chapters collapsed,
|
||||
# categories silently gone.
|
||||
_DEDUPED = PayloadStats(
|
||||
size_bytes=12_277_887,
|
||||
manga=2182,
|
||||
categories=0,
|
||||
chapters=68_418,
|
||||
sources=24,
|
||||
)
|
||||
|
||||
|
||||
def test_the_real_incident_is_reported_as_collapsed() -> None:
|
||||
verdict = compare(_STUB, _GOOD)
|
||||
assert verdict.status is Status.COLLAPSED
|
||||
assert verdict.is_failure
|
||||
assert "manga went from 2182 to 0" in verdict.reason
|
||||
|
||||
|
||||
def test_category_only_loss_is_still_a_collapse() -> None:
|
||||
"""The subtle half of the incident.
|
||||
|
||||
Every manga and every distinct chapter survived; only the categories went.
|
||||
A manga-only check would have called this a clean sync.
|
||||
"""
|
||||
verdict = compare(_DEDUPED, _GOOD)
|
||||
assert verdict.is_failure
|
||||
assert "categories went from 18 to 0" in verdict.reason
|
||||
|
||||
|
||||
def test_chapter_dedup_alone_does_not_trip_the_guard() -> None:
|
||||
"""68418 vs 75376 is a 9.2 % drop caused by removing duplicates.
|
||||
|
||||
Real event: the restore collapsed 6958 duplicate chapter entries while the
|
||||
distinct chapter count was unchanged. Failing here would make the guard cry
|
||||
wolf on a legitimate cleanup.
|
||||
"""
|
||||
previous = PayloadStats(
|
||||
size_bytes=14_083_674,
|
||||
manga=2182,
|
||||
categories=18,
|
||||
chapters=75_376,
|
||||
sources=24,
|
||||
)
|
||||
current = PayloadStats(
|
||||
size_bytes=12_277_887,
|
||||
manga=2182,
|
||||
categories=18,
|
||||
chapters=68_418,
|
||||
sources=24,
|
||||
)
|
||||
assert compare(current, previous).status is Status.OK
|
||||
|
||||
|
||||
def test_growth_is_healthy() -> None:
|
||||
bigger = PayloadStats(
|
||||
size_bytes=15_000_000,
|
||||
manga=2200,
|
||||
categories=19,
|
||||
chapters=76_000,
|
||||
sources=24,
|
||||
)
|
||||
assert compare(bigger, _GOOD).status is Status.OK
|
||||
|
||||
|
||||
def test_small_drop_is_tolerated() -> None:
|
||||
"""Finishing and removing a few series is normal."""
|
||||
smaller = PayloadStats(
|
||||
size_bytes=14_000_000,
|
||||
manga=2100,
|
||||
categories=18,
|
||||
chapters=75_000,
|
||||
sources=24,
|
||||
)
|
||||
assert compare(smaller, _GOOD).status is Status.OK
|
||||
|
||||
|
||||
def test_drop_past_the_threshold_fails() -> None:
|
||||
halved = PayloadStats(
|
||||
size_bytes=7_000_000,
|
||||
manga=1000,
|
||||
categories=18,
|
||||
chapters=40_000,
|
||||
sources=24,
|
||||
)
|
||||
verdict = compare(halved, _GOOD)
|
||||
assert verdict.is_failure
|
||||
assert "manga dropped" in verdict.reason
|
||||
|
||||
|
||||
def test_missing_baseline_is_first_run_not_success() -> None:
|
||||
"""A lost state file must not silently turn the guard into a no-op."""
|
||||
verdict = compare(_GOOD, None)
|
||||
assert verdict.status is Status.FIRST_RUN
|
||||
assert not verdict.is_failure
|
||||
assert verdict.previous is None
|
||||
|
||||
|
||||
def test_growth_from_a_zero_baseline_is_not_a_drop() -> None:
|
||||
assert compare(_GOOD, _STUB).status is Status.OK
|
||||
|
||||
|
||||
def test_threshold_is_configurable() -> None:
|
||||
slightly_smaller = PayloadStats(
|
||||
size_bytes=13_000_000,
|
||||
manga=2100,
|
||||
categories=18,
|
||||
chapters=75_000,
|
||||
sources=24,
|
||||
)
|
||||
strict = Thresholds(max_drop_ratio=0.01)
|
||||
assert compare(slightly_smaller, _GOOD, strict).is_failure
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ratio", [0.0, 1.0, -0.5, 2.0])
|
||||
def test_nonsense_thresholds_are_rejected(ratio: float) -> None:
|
||||
with pytest.raises(ValueError, match="max_drop_ratio"):
|
||||
Thresholds(max_drop_ratio=ratio)
|
||||
|
||||
|
||||
def test_reason_describes_a_healthy_payload() -> None:
|
||||
assert "2182 manga" in compare(_GOOD, _GOOD).reason
|
||||
@ -1,132 +0,0 @@
|
||||
"""Decide whether a payload change is normal drift or a silent collapse.
|
||||
|
||||
The thresholds here encode one real incident. A pure "is it zero?" check would
|
||||
have caught 2026-08-09, but Kamiruku's report on upstream #1634 describes the
|
||||
harder case — a restore that keeps *some* entries and drops the rest. A library
|
||||
does not lose a tenth of itself between two syncs, so that is the line.
|
||||
|
||||
Categories get their own rule because on 2026-08-09 all 2182 manga survived the
|
||||
round trip and all 18 categories did not. A manga-only check would have called
|
||||
that a clean sync.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
import enum
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from python_pkg.syncyomi_guard.payload import PayloadStats
|
||||
|
||||
|
||||
class Status(enum.Enum):
|
||||
"""Outcome of comparing a payload against the last known good one."""
|
||||
|
||||
OK = "ok"
|
||||
FIRST_RUN = "first-run"
|
||||
COLLAPSED = "collapsed"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Thresholds:
|
||||
"""Limits distinguishing acceptable shrinkage from data loss.
|
||||
|
||||
``max_drop_ratio`` is a fraction: 0.10 fails a drop of more than 10 %.
|
||||
Removing a few finished series is normal; losing a tenth of a 2000-entry
|
||||
library in one sync is not something that happens on purpose.
|
||||
"""
|
||||
|
||||
max_drop_ratio: float = 0.10
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
"""Reject a ratio outside the open interval (0, 1).
|
||||
|
||||
Raises:
|
||||
ValueError: If ``max_drop_ratio`` is not a usable fraction.
|
||||
"""
|
||||
if not 0.0 < self.max_drop_ratio < 1.0:
|
||||
msg = f"max_drop_ratio must be in (0, 1), got {self.max_drop_ratio}"
|
||||
raise ValueError(msg)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Verdict:
|
||||
"""The guard's decision, plus the reason, for logs and notifications."""
|
||||
|
||||
status: Status
|
||||
reason: str
|
||||
current: PayloadStats
|
||||
previous: PayloadStats | None
|
||||
|
||||
@property
|
||||
def is_failure(self) -> bool:
|
||||
"""Whether this verdict should fail the run and alert."""
|
||||
return self.status is Status.COLLAPSED
|
||||
|
||||
|
||||
def _drop_reason(
|
||||
label: str,
|
||||
previous: int,
|
||||
current: int,
|
||||
thresholds: Thresholds,
|
||||
) -> str | None:
|
||||
"""Return a failure reason if ``label`` dropped too far, else ``None``."""
|
||||
if previous == 0 or current >= previous:
|
||||
return None
|
||||
if current == 0:
|
||||
return f"{label} went from {previous} to 0"
|
||||
dropped = (previous - current) / previous
|
||||
if dropped > thresholds.max_drop_ratio:
|
||||
return (
|
||||
f"{label} dropped {dropped:.0%} "
|
||||
f"({previous} to {current}, limit {thresholds.max_drop_ratio:.0%})"
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def compare(
|
||||
current: PayloadStats,
|
||||
previous: PayloadStats | None,
|
||||
thresholds: Thresholds | None = None,
|
||||
) -> Verdict:
|
||||
"""Compare a payload against the last known good snapshot.
|
||||
|
||||
A missing ``previous`` is reported as ``FIRST_RUN`` rather than success, so
|
||||
that a lost state file cannot quietly turn the guard into a no-op that
|
||||
reports OK forever.
|
||||
"""
|
||||
limits = thresholds or Thresholds()
|
||||
|
||||
if previous is None:
|
||||
return Verdict(
|
||||
status=Status.FIRST_RUN,
|
||||
reason=f"no baseline yet; recording {current.describe()}",
|
||||
current=current,
|
||||
previous=None,
|
||||
)
|
||||
|
||||
failures = [
|
||||
reason
|
||||
for label, before, after in (
|
||||
("manga", previous.manga, current.manga),
|
||||
("chapters", previous.chapters, current.chapters),
|
||||
("categories", previous.categories, current.categories),
|
||||
)
|
||||
if (reason := _drop_reason(label, before, after, limits)) is not None
|
||||
]
|
||||
|
||||
if failures:
|
||||
return Verdict(
|
||||
status=Status.COLLAPSED,
|
||||
reason="; ".join(failures),
|
||||
current=current,
|
||||
previous=previous,
|
||||
)
|
||||
|
||||
return Verdict(
|
||||
status=Status.OK,
|
||||
reason=current.describe(),
|
||||
current=current,
|
||||
previous=previous,
|
||||
)
|
||||
Loading…
Reference in New Issue
Block a user