kuhyx/testsAndMisc
1
0
Fork 0
mirror of https://github.com/kuhyx/testsAndMisc.git synced 2026-11-10 20:07:04 +01:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

wsg_grabber: scrape /wsg/, download every new video, triage with keep/pass
Some checks failed
Pre-commit checks / pre-commit (push) Has been cancelled
Details
Python tests / test (push) Has been cancelled
Details
Shell tests / shell-tests (push) Has been cancelled
Details
Shell tests / arch-tests (push) Has been cancelled
Details

Browse Source 
New package. Walks every thread on 4chan's /wsg/, downloads each video it has
not seen, and reviews them one at a time: mpv embedded in a Tk window via --wid
with Keep/Pass/Undo directly beneath the picture.

Dedupe is structural rather than a convention. Every post carrying a file also
carries md5 -- base64 MD5 of the whole file -- so it is the sqlite primary key.
An already-seen video costs zero bytes to skip and a byte-identical repost in
another thread collapses onto the existing row. Verified against a live file:
the API's Nz9OEKdMuMZEdYE6eLTKmA== matched base64(md5(bytes)) exactly.

Undo is unbounded and survives quitting: the verdict trail lives in the index,
not in memory, so yesterday's decisions can still be taken back.

No downloaded video is ever deleted. A pass moves the file to trash/ and leaves
it there; files.apply_move refuses to overwrite an existing destination, so the
guarantee holds by construction rather than per caller.

Three bugs only running it could find, each caught from evidence rather than
guesswork:
  - the video stayed black: a <Configure> handler set mpv's geometry, fighting
    the automatic sizing --wid already does
  - playback never advanced: refresh() ran every tick and re-sent loadfile ~7x
    a second, restarting the video continuously
  - it froze on one video around the 26th while filenames kept advancing: mpv
    pushes an event stream down the same socket and nothing drained it, so the
    buffer filled and mpv stopped reading commands. Isolated to command #35
    without a drain thread; all 60 honoured with one.

A five-lens review then found nine more, all reproduced before fixing: a failed
mpv start stranded the non-daemon worker forever; undo could clobber a file in
incoming/; a NUL-bearing md5 poisoned the index permanently; an oversized int
killed the worker with no event; gzip bombs on both hosts were accepted; the
board endpoints were fetched unconditionally forever; and Session.missing could
corrupt the undo trail.

mpv runs as a subprocess over its JSON IPC socket rather than via python-mpv,
which resolves libmpv at import time and would fail on CI while passing here.
Third-party surface is requests alone, already declared.

277 package tests, 100% branch coverage on all 22 modules, no suppressions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Krzysztof Rudnicki 2026-07-27 14:52:30 +02:00
parent 0fc3788d7a
commit 89ed9bc95d
45 changed files with 9124 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
docs/superpowers/contracts/wsg-grabber-2026-07-27.json Normal file
View File

@ -0,0 +1,21 @@
{
"title": "New package python_pkg/wsg_grabber: scrape 4chan /wsg/ for videos and triage them with an embedded-mpv keep/pass reviewer",
"objective": "There is no tool for pulling /wsg/ video content and deciding what to keep. Build one that walks every thread on the board, downloads every video it has not already seen, and presents them one at a time in a tinder-style reviewer -- mpv embedded in a Tk window via --wid with Keep/Pass controls directly beneath the video. The defining requirement is that a video reviewed once is never shown or downloaded again, across runs and across reposts. Success: running the reviewer plays a downloaded video within seconds while the rest of the board continues downloading behind it; keep and pass relocate the file; and a subsequent run offers only genuinely new videos.",
"acceptance_criteria": [
"Deduplication uses the API's own `md5` field (base64 MD5 of the whole file, verified byte-exact against a live download) as the sqlite primary key, so an already-seen file costs zero bytes to skip and a cross-thread repost of identical bytes collapses onto the existing row",
"A pass moves the file to trash/ and a keep moves it to keep/; there is no unlink() anywhere in the package except to discard a corrupt partial download, so nothing the user might want back is ever destroyed",
"The board is polled politely: at most one request per second to a.4cdn.org, If-Modified-Since on every API request so an unchanged thread costs a 304, one persistent requests.Session (required, because i.4cdn.org 429s under load and sets __cf_bm/_cfuvid cookies), and Retry-After-aware exponential backoff",
"A file that 404s or is deleted upstream reaches a terminal `gone` state and is never retried on any future run; a file already downloaded keeps its state even if the post is deleted upstream",
"Downloads are prioritised over further indexing, so the first playable video reaches the screen seconds after launch rather than after the whole board is catalogued",
"100% branch coverage across every module including ui.py and player.py, with zero `# noqa` and zero `# type: ignore` (both blocked by pre-commit), ruff `select = [\"ALL\"]` clean, mypy strict clean, pylint 10.00/10, and every file under the repo's 500-line cap",
"No test constructs a real tkinter.Tk(), starts a real mpv, opens a socket, or writes outside tmp_path; an autouse fixture redirects Path.home, HOME and XDG_DATA_HOME so the suite cannot touch the real ~/.local/share/wsg_grabber"
],
"out_of_scope": [
"Any board other than /wsg/ -- the board is a hardcoded constant, not a flag, per the explicit scope decision",
"A disk budget, cache eviction or download cap -- the requirement is to download everything, so a first run is deliberately ~9,000 files and 20-25GB",
"Automatic emptying of trash/ -- clearing it is the user's decision and the package must never do it",
"Perceptual/near-duplicate detection -- md5 catches byte-identical reposts only, which is the dominant case on /wsg/; a re-encoded repost is a different file and will be shown again",
"python-mpv as the player binding -- it resolves libmpv at import time and would break CI, so mpv is driven as a subprocess over its JSON IPC socket instead"
],
"verifier": "PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber scrape --seconds 100 (downloads real files); PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber (reviewer window renders video inside the Tk frame with the control bar beneath); mark files reviewed and rescan, confirming kept/passed counts are unchanged; python -m pytest python_pkg/wsg_grabber --cov --cov-branch --cov-fail-under=100; pre-commit run --files <changed>"
}

100
docs/superpowers/evidence/wsg-grabber-2026-07-27.json Normal file
View File

@ -0,0 +1,100 @@
{
"intent": "Add python_pkg/wsg_grabber: scrapes 4chan /wsg/ for videos, downloads every one not seen before in the background, and reviews them one at a time with mpv embedded in a Tk window and Keep/Pass controls directly beneath the video. A reviewed video is never shown or downloaded again.",
"scope": [
"New package python_pkg/wsg_grabber/ (19 modules + tests, README.md, install.sh)",
"New docs/superpowers/contracts/wsg-grabber-2026-07-27.json",
"No change to meta/requirements.txt: the only third-party dependency is requests, already declared. mpv and tk are system packages installed by install.sh",
"Non-goals: any board but /wsg/, a download cap, automatic deletion of anything, perceptual dedup"
],
"changes": [
"Dedup is structural rather than conventional: the API publishes `md5` (base64 MD5 of the entire file) on every post carrying an attachment, so it is used as the sqlite primary key. An already-seen file is skipped before a byte is transferred, and a repost of identical bytes in another thread collapses onto the existing row. Verified byte-exact against a live download: the API's Nz9OEKdMuMZEdYE6eLTKmA== matched base64(md5(bytes)) computed locally.",
"sqlite (stdlib) instead of this repo's usual JSON state files, because two threads write concurrently -- the downloader appending rows, the UI recording verdicts -- and the seen-set grows without bound. Noted as a deliberate deviation.",
"mpv is driven as a subprocess over its JSON IPC socket, NOT via python-mpv. python-mpv calls ctypes.util.find_library('mpv') at module scope (mpv.py:70-73) and raises OSError wherever libmpv is absent, which includes .github/workflows/python-tests.yml's ubuntu-latest runner: it would pass locally and fail on push. The lazy-import workaround trips PLC0415, which cannot be suppressed here.",
"Strict pure-logic/IO-shell split: ui.py (the only module importing tkinter) and player.py contain no decisions. review.py produces a ReviewCommand dataclass that the window applies verbatim, which is what makes 100% branch coverage of the GUI achievable without a display.",
"A terminal `gone` state stops a 404 or upstream-deleted file being retried on every future run -- necessary because threads fall off page 10 during a multi-hour first pass. An already-downloaded file keeps its state when the post is deleted, since the bytes are already ours.",
"Politeness: <=1 request/second to a.4cdn.org, If-Modified-Since on every API GET (verified 304), one persistent requests.Session (required because i.4cdn.org 429s under load and sets __cf_bm/_cfuvid cookies), Retry-After-aware exponential backoff, and Range-based resume of partial downloads verified against the API md5 before the file is offered.",
"Downloads outrank further indexing so the first playable video appears seconds after launch while the rest of the board loads behind it.",
"tests/conftest.py adds the repo's first autouse path-isolation fixture, redirecting Path.home, HOME and XDG_DATA_HOME, plus an AST guard in test_paths.py asserting no module resolves a home directory at import time -- the failure mode that previously leaked test writes into real user data."
],
"verification": [
{
"command": "PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber scrape --seconds 100",
"result": "pass",
"evidence": "Indexed 123 files and downloaded 97 of them (472 MB) in 100s at the intended ~1/s. Progress line advanced known 123 / ready 11 -> 96 / pending 111 -> 26."
},
{
"command": "PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber (reviewer window, screenshotted on HDMI-0)",
"result": "pass",
"evidence": "Video renders INSIDE the Tk frame with the Keep/Pass bar beneath it in the same window; status line read 'passed 0 - queued 165 - downloaded 10/2..' and the filename line 'Hinamatsuri 2.webm - 1920x1080 - 5.9 MiB', i.e. playback and downloading proceeding together. mpv reported vo-configured=True, geometry='' and osd-dimensions 636x1337, matching the frame."
},
{
"command": "First reviewer run exposed a real bug not caught by 213 passing tests",
"result": "fixed",
"evidence": "The video area was black. Querying mpv over IPC showed playback-time=10.1 and vo-configured=True but geometry='329x180': the <Configure> handler was sending set_property geometry, which fights the automatic sizing that --wid already performs ('the window will always be resized to cover the parent window fully', man mpv). Removed the resize plumbing from player.py, ui.py and review.py; re-ran and the video fills the frame."
},
{
"command": "Second reviewer run exposed a second real bug, also invisible to the tests",
"result": "fixed",
"evidence": "mpv reported osd-width=0 and playback that never advanced. Cause: tick() calls refresh() -> apply() -> _play() unconditionally, so `loadfile ... replace` was being sent roughly seven times a second and the video restarted continuously instead of playing. ui.py now tracks the currently-playing path and only loads when it changes; stop is likewise only sent when something is playing. Three regression tests added (test_the_same_video_is_only_loaded_once drives ten ticks and asserts play was called once). Re-ran: osd-width=849, and three screenshots taken three seconds apart differ by 2.6M and 5.2M pixels, i.e. playback demonstrably advancing."
},
{
"command": "Third bug, found from a user-reported session log rather than by reproduction",
"result": "fixed",
"evidence": "User reported the reviewer freezing on one video around the 26th while filenames kept advancing. The session log showed it exactly: four consecutive review.show records with mpv_before.path pinned to one file and playback-time tracking wall-clock (13.0s gap -> t=12.98, +1.4 -> 14.41, +1.8 -> 16.18, +1.3 -> 17.48), core-idle=false, every later file present on disk and every loadfile sent without error. Root cause: mpv pushes an async event stream plus a per-command reply down the same socket, and the client never read from it; the kernel buffer filled, mpv's writer for that connection blocked, and it stopped reading commands. Isolated with a GUI-free harness: without draining, commands were silently ignored from #35 while the identical command on a FRESH socket was honoured instantly; with a drain thread, all 60 were honoured. Fixed by a reader thread in player.py that drains the socket and logs mpv's error replies. Verified through the real app: 50 consecutive passes, zero mismatches between what the UI loaded and what mpv held. Two earlier hypotheses (a run of visually-identical reposts, and an mpv repaint failure) were tested and disproved -- the 31 files the user passed had zero near-duplicate first frames, and a minimal mpv --wid harness switched files correctly."
},
{
"command": "Undo, added on request after a misclick lost a video to the wrong pile",
"result": "pass",
"evidence": "`u` / Backspace and an UNDO button take back the last verdict: the file moves back from keep/ or trash/ into incoming/, the index row returns to ready, the counter is corrected, and the video is shown again -- with whatever was on screen pushed to the front of the queue so nothing is skipped. Multi-level within a session. Verified against real files: KEEP -> undo -> PASS -> PASS -> undo -> undo returned kept/passed to 0/0 with all four files back in incoming/ and keep/ and trash/ empty. Edge cases covered by tests: a destination that was renamed to avoid a collision (undo follows the real name), an empty history (no-op), and a file the user deleted from trash/ by hand (logged, entry dropped rather than failing repeatedly)."
},
{
"command": "Clean-shutdown check after closing the reviewer window",
"result": "pass",
"evidence": "ps shows no reviewer process and no orphan mpv; the IPC socket is removed; one .part file remains, which is correct and resumable."
},
{
"command": "Dedup acceptance: mark 189 downloaded files reviewed, then rescan",
"result": "pass",
"evidence": "Marked 94 kept / 95 passed. After a 45s rescan, kept=94 and passed=95 were unchanged while known files rose 221 -> 236, i.e. 15 genuinely new files were picked up and not one reviewed file was re-offered or re-downloaded."
},
{
"command": "systemd-run --user --scope -p MemoryMax=2G -p MemorySwapMax=0 python -m pytest python_pkg/wsg_grabber --cov --cov-branch -W error",
"result": "pass",
"evidence": "272 passed. 100% branch coverage on all 22 modules, including the two that touch the untestable world: ui.py (160 stmts / 30 branches) and player.py (127 / 22). Run under `systemd-run --user --scope -p MemoryMax=2G -p MemorySwapMax=0` and with -W error, so a leaked thread, unclosed sqlite connection or undestroyed Tk object fails the suite."
},
{
"command": "ruff check / mypy / pylint",
"result": "pass",
"evidence": "ruff (select=ALL): All checks passed. mypy strict: no issues in 36 source files. pylint: 10.00/10. grep confirms zero '# noqa' and zero '# type: ignore' in the package."
},
{
"command": "shellcheck --severity=warning python_pkg/wsg_grabber/install.sh",
"result": "pass",
"evidence": "Clean after splitting declare/assign for SC2155; --help runs."
},
{
"command": "Five-lens code review (undo/migration, threads, data safety, untrusted input, repo rules)",
"result": "fixed",
"evidence": "Reviewers found seven real defects, each reproduced before fixing and re-verified after. (1) threads.json and archive.json were fetched UNCONDITIONALLY every idle cycle -- two un-cached requests a minute forever against a host the README promises to be polite to; now conditional via a new http_cache table (schema v3), proven by probe: first sweep sends no stamp, second sends it back. (2) Session.undo built its destination directly, bypassing unique_destination, and Path.replace clobbers silently -- a video in incoming/ could be destroyed with no trash copy; undo now picks a collision-free name AND files.apply_move refuses any occupied destination, making the guarantee structural rather than per-caller. (3) A 24-char md5 containing NUL passed validation, became a PRIMARY KEY and a filename, and crashed the worker before the row could be marked gone -- poisoning the index permanently across restarts; _as_md5 now matches the true base64 shape [A-Za-z0-9+/]{22}==. (4) An API integer >= 2**63 raised OverflowError, which was NOT in the worker's recoverable set, killing the thread with no WorkerFailed event and a permanently idle UI; _as_int now bounds to sqlite's signed 64-bit range and ArithmeticError was added to _RECOVERABLE. (5) A hostile CDN could serve a gzip bomb: 200 KB on the wire decoded to 200 MB on disk and was ACCEPTED, because the same party publishes the md5; a non-identity Content-Encoding is now refused outright. (6) Nothing compared bytes written against the advertised fsize -- a lying server filled the disk (64 MiB written against fsize=4); the write loop now aborts past the advertised size plus slack. (7) The API session advertised gzip/deflate/br/zstd and json.loads had no size limit (6 KB zstd -> 200 MB decoded); Accept-Encoding is now identity and responses over 32 MB are refused. Session.missing could also overwrite an already-recorded verdict, silently corrupting the undo trail -- it now refuses rows in TERMINAL."
},
{
"command": "Real-data-dir safety, verified rather than assumed",
"result": "pass",
"evidence": "A reviewer snapshotted the live ~/.local/share/wsg_grabber (3676 entries, 4.6 GB, index.db md5 b4857900c1a1fcb20e1e05de97a349b2), ran the full suite, and diffed: empty, at every checkpoint, including after five adversarial probes. Every storage path in the package derives from paths.data_dir(), so there is no other write target."
}
],
"risks": [
"Keyboard delivery to the Tk bindings is UNVERIFIED. --input-vo-keyboard=no is set precisely so mpv declines the keyboard and X delivers keys to the focused Tk toplevel, but confirming it needs real key input, which requires stealing focus from the fullscreen game that was running throughout. Synthetic XSendEvent keys are ignored by both mpv and Tk and prove nothing. If it turns out mpv still swallows keys, the fallback is one IPC `keybind` command per shortcut plus a small socket reader, contained entirely in player.py. The mouse buttons are wired to the same handlers and are unaffected.",
"The package now holds real user data in ~/.local/share/wsg_grabber. The autouse isolation fixture plus the AST guard are what keep the test suite out of it.",
"A first full run is roughly 9,000 files and 20-25 GB over a couple of hours -- inherent to the explicit 'download everything, no cap' decision. It is resumable and restarting continues mid-file.",
"md5 catches byte-identical reposts only. A re-encoded repost is a different file and will be shown again; perceptual dedup is out of scope.",
"Keyboard delivery to the Tk bindings (k/j/u) is still UNVERIFIED. --input-vo-keyboard=no exists so mpv declines the keyboard and X delivers to the focused toplevel, but confirming it needs real key input, which means stealing focus from the fullscreen game that ran throughout. The mouse buttons are wired to the same handlers and are unaffected.",
"Two reviewer instances against one index can still interfere: there is no lock file. The worst case is now bounded -- Session.missing refuses to clobber a recorded verdict -- but the queues are not coordinated."
],
"rollback": [
"git rm -r python_pkg/wsg_grabber docs/superpowers/contracts/wsg-grabber-2026-07-27.json docs/superpowers/evidence/wsg-grabber-2026-07-27.json; nothing else in the repo imports it and meta/requirements.txt was not modified",
"sudo python_pkg/wsg_grabber/install.sh --uninstall removes /usr/local/bin/wsg-grabber and leaves downloaded videos alone",
"~/.local/share/wsg_grabber is untouched by removal; delete it by hand if the downloaded videos are no longer wanted",
"After rollback confirm the rest of the suite still reaches 100% coverage: python -m pytest python_pkg --cov=python_pkg --cov-branch"
]
}

187
python_pkg/wsg_grabber/README.md Normal file
View File

@ -0,0 +1,187 @@
# wsg_grabber
Scrapes 4chan's /wsg/ board for videos, downloads every one it has not seen
before, and shows them one at a time for a keep/pass decision — mpv embedded in
a Tk window with the buttons directly underneath.
```bash
PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber # review
PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber scrape # no UI
PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber stats
```
`k` / `→` keeps, `j` / `space` / `←` passes, `u` / `Backspace` undoes, `q` /
`Esc` quits. The buttons do the same thing for the mouse.
## Undo
Misclicked? `u` takes the last verdict back: the file returns from `keep/` or
`trash/` to `incoming/`, the index row goes back to _ready_, the counter is
corrected, and that video is on screen again. Whatever was showing when you hit
undo goes to the front of the queue, so nothing is skipped.
It steps back through **any number of verdicts, and survives quitting** — the
trail lives in the index, not in memory, so reopening the reviewer can still
undo what you decided yesterday. The status bar shows how many are left
(`u undoes (17)`), and the button greys out when there is nothing to take back.
If you cleared `trash/` by hand in the meantime, undo says so in the log and
drops that entry rather than failing on it repeatedly.
## Storage
```
~/.local/share/wsg_grabber/
├── index.db sqlite, remembers every file ever seen
├── incoming/ downloaded, awaiting your verdict
├── keep/ you kept these
└── trash/ you passed on these — NOTHING here is ever removed automatically
```
**No downloaded video is ever deleted.** A pass moves the file into `trash/`;
emptying it is your call. `unlink` appears only on things that are not videos —
a corrupt or abandoned `.part`, and mpv's control socket — and `files.apply_move`
refuses to overwrite an existing destination, so a move can never destroy a
video you already decided on.
## Why it never shows you the same video twice
Every 4chan post that carries a file also carries `md5` — the base64 MD5 of the
whole file, published by the API before you fetch anything. That is the primary
key of the `files` table, so:
- a video you have already reviewed is never downloaded again, at a cost of
zero bytes;
- the same clip reposted in another thread collapses onto the existing row,
which on /wsg/ is the common case;
- a truncated or mangled download is caught, because the bytes are checked
against that same value before the file is offered for review.
Verified against a live file during development: the API's
`Nz9OEKdMuMZEdYE6eLTKmA==` matched `base64(md5(bytes))` exactly.
## Being a good citizen
- One request per second to `a.4cdn.org`, which is what the API rules ask for.
- `If-Modified-Since` on every API request — thread bodies, `threads.json` and
`archive.json` alike — plus a `last_modified` comparison against the stored
value, so an unchanged thread costs a 304 and nothing else. A steady-state
rescan is a couple of conditional requests that return 304.
- `Accept-Encoding: identity` on both hosts, and hard caps on response and
download size. A compressed response is refused outright: a few hundred KB on
the wire can decode to gigabytes, and the md5 check does not help when the
same party publishes both the bytes and the digest.
- One long-lived `requests.Session`. This is required rather than merely tidy:
`i.4cdn.org` answers with 429 under load and sets `__cf_bm` / `_cfuvid`
cookies that have to persist across requests.
- 429 and 5xx back off exponentially, honouring `Retry-After`.
## Downloading is prioritised over indexing
While anything is queued the worker downloads rather than walking more threads.
That is deliberate: it is what puts a playable video on screen seconds after
launch instead of after the whole board has been catalogued. The consequence is
that `stats` shows the known-file count climbing in bursts rather than smoothly
— the board is fully indexed once downloads drain. A first full run is roughly
9,000 files and 20–25 GB, so expect a couple of hours; it is resumable and
restarting picks up mid-file via HTTP `Range`.
## Logs
Every run writes one JSON object per line to
`~/.local/share/wsg_grabber/logs/session-<time>-<pid>.jsonl`.
```bash
wsg-grabber logs # where they are, plus a tally of the newest
wsg-grabber --log-level debug # record every IPC command as well
wsg-grabber --echo-log # mirror records to stderr while running
```
The point is after-the-fact diagnosis. If the reviewer ever looks stuck on one
video while the filenames keep advancing, one query settles whether mpv was
handed the new file and was actually decoding it:
```bash
LOG=$(ls -t ~/.local/share/wsg_grabber/logs/*.jsonl | head -1)
jq -r 'select(.event=="review.show")
| "show \(.file) mpv_was: \(.mpv_before.path) t=\(.mpv_before["playback-time"]) \(.mpv_before.width)x\(.mpv_before.height)"' "$LOG"
```
A healthy run shows `playback-time` advancing and the dimensions changing per
video. A frozen one shows them static, which points at the player rather than
at the queue. Other useful queries:
```bash
jq -c 'select(.level=="error")' "$LOG" # what went wrong
jq -r 'select(.event=="review.verdict")|"\(.choice) \(.on_file)"' "$LOG"
jq -c 'select(.event|startswith("player."))' "$LOG"
```
Field names are stable; every record carries `ts`, `seq`, `level`, `event` and
`thread`.
## Debugging without touching your queue
`paths` honours `XDG_DATA_HOME`, so a throwaway instance costs nothing and can
neither consume real videos nor record real verdicts:
```bash
XDG_DATA_HOME=/tmp/wsg-sandbox python3 -m python_pkg.wsg_grabber stats
```
## Layout
| Module | Responsibility |
| ------------------- | ----------------------------------------------------------------- |
| `catalog.py` | Parses the board's JSON. Pure. |
| `scanner.py` | Scheduling, backoff, resume decisions. Pure. |
| `review.py` | Everything the reviewer does: queue, verdicts, status text. Pure. |
| `verdict.py` | What keep/pass means, as a planned file move. Pure. |
| `net.py` | The only module that speaks HTTP. |
| `db.py` | sqlite schema, connection setup, migrations. |
| `store.py` | The download lifecycle of a file. |
| `store_threads.py` | Which threads were visited, and their HTTP stamps. |
| `store_verdicts.py` | The review trail that undo walks back. |
| `downloader.py` | The background worker thread. |
| `files.py` | The only module that moves files. |
| `player.py` | mpv as a subprocess over its JSON IPC socket. |
| `ui.py` | The only module that imports `tkinter`. |
| `logs.py` | Structured JSONL event log. |
| `app.py` / `cli.py` | Composition and argument parsing. |
The split is what makes 100% branch coverage achievable: `ui.py` and
`player.py` are close to pure wiring, and every decision they would otherwise
make lives in a pure function tested from plain values with no display and no
network. Two decisions do remain in `ui.py`, both because a bug proved they had
to: the already-playing guard (without it `loadfile` fires every tick and the
video restarts continuously) and dropping an item whose file has vanished.
## mpv is a subprocess, not `python-mpv`
`python-mpv` resolves `libmpv` at _import_ time and raises `OSError` wherever it
is missing — including this repo's CI runner, so it would pass locally and fail
on push. Driving `mpv --wid=<frame> --input-ipc-server=<socket>` over a unix
socket needs only `subprocess`, `socket` and `json`.
**The socket must be drained.** mpv pushes an asynchronous event stream plus a
reply per command down the same connection. A client that only ever writes lets
that backlog fill the kernel buffer, at which point mpv's writer for that
connection blocks and it stops reading commands. Nothing errors: `sendall` still
succeeds, mpv stays alive and keeps playing whatever it had, so the reviewer
advances through filenames while the picture never changes. Measured: commands
were silently ignored from about the thirty-fifth video onwards, and the very
same command sent on a fresh connection worked instantly. `player.py` therefore
runs a reader thread whose only job is to drain — deleting it reintroduces the
freeze roughly half an hour into a session.
Two more mpv details worth knowing if you touch `ui.py`:
- `--wid` does not draw into your widget. mpv creates its own X window and
reparents it onto the frame, covering it completely — which is why the
control bar is packed _below_ the video rather than over it.
- `--input-vo-keyboard=no` is what lets the Tk key bindings fire at all. Without
it mpv's child window takes keyboard focus and Tk never sees a keypress.
## Requirements
`mpv` and `tk` (system packages, installed by `install.sh`) and `requests`,
which the repo already declares. Nothing else.

13
python_pkg/wsg_grabber/__init__.py Normal file
View File

@ -0,0 +1,13 @@
"""Scrape 4chan's /wsg/ board and triage its videos with a keep/pass reviewer.
The board exposes a read-only JSON API whose post records carry ``md5`` --
``base64(md5(file_bytes))`` over the whole file. That single field is why this
package can honour "never show me the same video twice" without spending a byte:
dedupe happens against the catalogue, before any download starts, and reposts of
an identical file across threads collapse onto one row for free.
Nothing here ever deletes a file. A "pass" verdict moves the video to
``trash/`` and leaves it for the user to clear by hand.
"""
from __future__ import annotations

14
python_pkg/wsg_grabber/__main__.py Normal file
View File

@ -0,0 +1,14 @@
"""Module entry point.
Usage:
PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber --help
"""
from __future__ import annotations
import sys
from python_pkg.wsg_grabber.cli import main
if __name__ == "__main__":
sys.exit(main())

248
python_pkg/wsg_grabber/app.py Normal file
View File

@ -0,0 +1,248 @@
"""Wires the pieces together and owns an orderly shutdown.
Kept separate from :mod:`python_pkg.wsg_grabber.cli` so the composition can be
exercised without argument parsing, and separate from
:mod:`python_pkg.wsg_grabber.ui` so nothing here needs a display.
"""
from __future__ import annotations
from dataclasses import dataclass
import queue
import threading
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import (
db,
downloader,
files,
logs,
net,
paths,
review,
store,
store_verdicts,
verdict,
)
from python_pkg.wsg_grabber.constants import JOIN_TIMEOUT_S
from python_pkg.wsg_grabber.models import FileMove, Verdict, WorkerFailed
from python_pkg.wsg_grabber.states import TERMINAL, FileState
if TYPE_CHECKING:
import sqlite3
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.review import SessionState
@dataclass
class Session:
"""A running review session and everything it must clean up."""
conn: sqlite3.Connection
events: queue.SimpleQueue[DownloadEvent]
stop: threading.Event
worker: threading.Thread | None = None
def commit(self, state: SessionState, choice: Verdict) -> SessionState:
"""Apply a verdict: move the file, record it, advance the queue.
Args:
state: Current session state.
choice: What the user decided.
Returns:
SessionState: Updated state.
"""
item = state.current
if item is None:
return state
move = verdict.plan_move(
item,
choice,
paths.keep_dir(),
paths.trash_dir(),
files.existing_names(
verdict.target_dir(choice, paths.keep_dir(), paths.trash_dir()),
),
)
if not files.apply_move(move):
logs.warning(
"verdict.source_gone",
file=move.src.name,
choice=choice.value,
)
return self.missing(state)
logs.event(
"verdict.moved",
choice=choice.value,
src=str(move.src),
dst=str(move.dst),
md5=item.md5,
)
store_verdicts.record_verdict(
self.conn,
item.md5,
verdict.target_state(choice),
move.dst.name,
)
return review.on_verdict(state, choice)
def undo(self, state: SessionState) -> SessionState:
"""Take back the last verdict, putting the file back in the queue.
Args:
state: Current session state.
Returns:
SessionState: Updated state showing the restored video.
"""
action = store_verdicts.newest_verdict(self.conn, paths.incoming_dir())
if action is None:
return state
source = (
paths.keep_dir() if action.choice is Verdict.KEEP else paths.trash_dir()
) / action.reviewed_name
# Mirror what commit does: never assume the old name is free again.
destination = verdict.unique_destination(
paths.incoming_dir(),
action.item.path.name,
files.existing_names(paths.incoming_dir()),
)
back = FileMove(md5=action.item.md5, src=source, dst=destination)
if not files.apply_move(back):
# The file is no longer where the verdict left it -- most likely the
# user cleared trash/ by hand. Drop it from the undo trail so a
# second attempt does not keep failing on the same entry.
logs.warning(
"undo.file_gone",
expected=str(source),
md5=action.item.md5,
)
store_verdicts.forget_verdict(self.conn, action.item.md5)
return review.forget_last(state)
store_verdicts.restore_for_review(
self.conn,
action.item.md5,
destination.name,
)
logs.event(
"undo.applied",
choice=action.choice.value,
file=destination.name,
restored_from=str(source),
remaining=store_verdicts.reviewed_count(self.conn),
)
return review.on_undo(state, action)
def missing(self, state: SessionState) -> SessionState:
"""Write off a video whose file has disappeared.
Args:
state: Current session state.
Returns:
SessionState: Updated state showing the next video.
"""
item = state.current
if item is None:
return state
# A row that already carries a verdict must not be forced to GONE: that
# would drop it out of the undo trail while the in-memory count still
# believes it is there, and the verdict itself would be unrecoverable.
# Reachable when the same item is queued twice, e.g. two reviewers
# running against one index.
if store.state_of(self.conn, item.md5) in TERMINAL:
logs.warning("missing.already_reviewed", md5=item.md5)
return review.on_missing_locally(state)
store.set_state(
self.conn,
item.md5,
FileState.GONE,
error="file missing locally",
)
return review.on_missing_locally(state)
def shutdown(self) -> None:
"""Stop the worker and close the index.
Joining matters: the worker owns a database connection and an HTTP
session, and abandoning it leaks both. Note the suite does NOT catch
that for us -- ``meta/pyproject.toml`` downgrades
``PytestUnraisableExceptionWarning`` to a warning -- so this has to be
right by construction rather than by test pressure.
"""
self.stop.set()
if self.worker is not None:
self.worker.join(timeout=JOIN_TIMEOUT_S)
if self.worker.is_alive():
# A worker blocked in a socket read can outlive the join, since
# READ_TIMEOUT_S exceeds JOIN_TIMEOUT_S. Keep the handle so the
# failure is visible rather than silently discarded.
logs.warning("shutdown.worker_still_running")
else:
self.worker = None
self.conn.close()
def open_session(*, include_archive: bool = True) -> Session:
"""Prepare storage and start the background worker.
Args:
include_archive: Whether to walk the board's archive as well.
Returns:
Session: A running session; call :meth:`Session.shutdown` when done.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
store.reset_in_flight(conn)
events: queue.SimpleQueue[DownloadEvent] = queue.SimpleQueue()
stop = threading.Event()
session = Session(conn=conn, events=events, stop=stop)
incoming = paths.incoming_dir()
db_file = paths.db_path()
def build() -> downloader.WorkerDeps:
"""Open the worker's own resources on the worker's own thread.
Returns:
downloader.WorkerDeps: Dependencies owned by the worker.
"""
return downloader.WorkerDeps(
conn=db.open_index(db_file),
session=net.build_session(),
incoming=incoming,
events=events,
stop=stop,
include_archive=include_archive,
)
try:
session.worker = downloader.start(
build,
lambda message: events.put(WorkerFailed(message=message)),
)
except BaseException:
conn.close()
raise
return session
def initial_state(session: Session) -> SessionState:
"""Build the state a window should open with.
Args:
session: The running session.
Returns:
SessionState: Starting state, including anything left unreviewed by a
previous run.
"""
counts = store.counts(session.conn)
return review.initial_state(
counts.get("total", 0),
store.ready_items(session.conn, paths.incoming_dir()),
store_verdicts.reviewed_count(session.conn),
)

251
python_pkg/wsg_grabber/catalog.py Normal file
View File

@ -0,0 +1,251 @@
"""Parsing of the /wsg/ JSON payloads. No network, no filesystem.
The API hands back loosely-typed JSON where file fields are simply absent on
posts without an attachment, so every accessor here treats a missing or
wrong-typed field as "not a video" rather than trusting the shape.
"""
from __future__ import annotations
import re
from typing import TYPE_CHECKING, Any
from python_pkg.wsg_grabber.constants import MEDIA_HOST, VIDEO_EXTENSIONS
from python_pkg.wsg_grabber.models import RemoteFile, ThreadRef
if TYPE_CHECKING:
from collections.abc import Iterable, Sequence
# base64 of a 16-byte digest is always 22 characters plus "==". Matching the
# exact shape -- not merely the length -- is what keeps a NUL byte out of a
# filename: one such value poisons the index permanently, because the crash
# happens before the row can ever be marked gone.
_MD5_SHAPE = re.compile(r"[A-Za-z0-9+/]{22}==")
# sqlite stores signed 64-bit integers; anything wider raises OverflowError deep
# inside executemany, which used to take the worker thread down with it.
_INT64_MIN = -(2**63)
_INT64_MAX = 2**63 - 1
def media_url(tim: int, ext: str) -> str:
"""Return the CDN address of an attachment.
Args:
tim: The post's ``tim`` field, which is the CDN filename stem.
ext: Extension including the leading dot.
Returns:
str: Absolute URL on the media host.
"""
return f"{MEDIA_HOST}/wsg/{tim}{ext}"
def parse_thread_list(payload: object) -> list[ThreadRef]:
"""Flatten ``threads.json`` into thread references.
Args:
payload: Decoded JSON: a list of pages, each with a ``threads`` list.
Returns:
list[ThreadRef]: Every advertised thread, in page order.
"""
refs: list[ThreadRef] = []
for page in _as_list(payload):
for entry in _as_list(_get(page, "threads")):
thread_no = _as_int(_get(entry, "no"))
if thread_no is None:
continue
refs.append(
ThreadRef(
thread_no=thread_no,
api_last_modified=_as_int(_get(entry, "last_modified")) or 0,
),
)
return refs
def parse_archive(payload: object) -> list[ThreadRef]:
"""Turn ``archive.json`` into thread references.
The archive is a bare list of thread numbers with no timestamps, so each
gets a zero stamp; they are immutable anyway and only need fetching once.
Args:
payload: Decoded JSON list of integers.
Returns:
list[ThreadRef]: Archived threads.
"""
refs: list[ThreadRef] = []
for entry in _as_list(payload):
thread_no = _as_int(entry)
if thread_no is not None:
refs.append(ThreadRef(thread_no=thread_no, api_last_modified=0))
return refs
def parse_thread(thread_no: int, payload: object) -> list[RemoteFile]:
"""Extract every video attachment from one thread payload.
Posts whose file was deleted upstream are skipped: there is nothing left to
fetch, and they must not be recorded as pending work.
Args:
thread_no: The thread being parsed, used to stamp each file.
payload: Decoded ``thread/<no>.json``.
Returns:
list[RemoteFile]: Attachments with a video extension.
"""
files: list[RemoteFile] = []
for post in _as_list(_get(payload, "posts")):
parsed = _parse_post(thread_no, post)
if parsed is not None:
files.append(parsed)
return files
def deleted_md5s(payload: object) -> set[str]:
"""Return md5s of posts whose attachment was removed upstream.
Args:
payload: Decoded ``thread/<no>.json``.
Returns:
set[str]: Identities that should be written off rather than retried.
"""
gone: set[str] = set()
for post in _as_list(_get(payload, "posts")):
if not _as_int(_get(post, "filedeleted")):
continue
digest = _as_md5(_get(post, "md5"))
if digest is not None:
gone.add(digest)
return gone
def new_files(
parsed: Iterable[RemoteFile],
known: Sequence[str] | set[str] | frozenset[str],
) -> list[RemoteFile]:
"""Drop attachments already present in the index.
This is where "never download the same video twice" is enforced, and it
happens before a single byte is transferred because the API supplies the
md5 up front.
Args:
parsed: Candidates from a thread.
known: md5s already recorded, in any state including terminal ones.
Returns:
list[RemoteFile]: Genuinely new attachments, deduped within the batch.
"""
seen = set(known)
fresh: list[RemoteFile] = []
for item in parsed:
if item.md5 in seen:
continue
seen.add(item.md5)
fresh.append(item)
return fresh
def _parse_post(thread_no: int, post: object) -> RemoteFile | None:
"""Turn one post into a RemoteFile when it carries a usable video.
Args:
thread_no: Owning thread.
post: Decoded post object.
Returns:
RemoteFile | None: None when the post has no video attachment.
"""
if _as_int(_get(post, "filedeleted")):
return None
tim = _as_int(_get(post, "tim"))
ext = _get(post, "ext")
digest = _as_md5(_get(post, "md5"))
post_no = _as_int(_get(post, "no"))
if tim is None or post_no is None or digest is None:
return None
if not isinstance(ext, str) or ext not in VIDEO_EXTENSIONS:
return None
name = _get(post, "filename")
return RemoteFile(
md5=digest,
tim=tim,
ext=ext,
orig_name=name if isinstance(name, str) else str(tim),
fsize=_as_int(_get(post, "fsize")) or 0,
width=_as_int(_get(post, "w")) or 0,
height=_as_int(_get(post, "h")) or 0,
thread_no=thread_no,
post_no=post_no,
)
def _get(container: object, key: str) -> object:
"""Read *key* from *container* when it is a mapping.
Args:
container: Anything; only dicts yield a value.
key: Key to read.
Returns:
object: The value, or None.
"""
if isinstance(container, dict):
return container.get(key)
return None
def _as_list(value: object) -> list[Any]:
"""Return *value* when it is a list, else an empty list.
Args:
value: Candidate.
Returns:
list[Any]: Safe list to iterate.
"""
return value if isinstance(value, list) else []
def _as_int(value: object) -> int | None:
"""Coerce *value* to int when it is a real integer.
Booleans are rejected: ``True`` is an int in Python but never a valid post
number or timestamp. Values outside sqlite's signed 64-bit range are
rejected too -- they raise OverflowError on insert, which is not in the
worker's recoverable set and would kill it silently.
Args:
value: Candidate.
Returns:
int | None: The integer, or None.
"""
if isinstance(value, bool) or not isinstance(value, int):
return None
return value if _INT64_MIN <= value <= _INT64_MAX else None
def _as_md5(value: object) -> str | None:
"""Validate the API's base64 md5 field.
The charset check is not decoration: the tail of this value ends up in a
filename via ``store.local_name``, and it arrives from an anonymous
imageboard. Length alone would let through a NUL byte, which raises deep
inside the filesystem call instead of being rejected here.
Args:
value: Candidate.
Returns:
str | None: The 24-character digest, or None when malformed.
"""
if not isinstance(value, str):
return None
return value if _MD5_SHAPE.fullmatch(value) else None

282
python_pkg/wsg_grabber/cli.py Normal file
View File

@ -0,0 +1,282 @@
"""Command-line entry point for the /wsg/ grabber.
Usage:
PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber
PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber scrape --seconds 60
PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber stats
"""
from __future__ import annotations
import argparse
from collections import Counter
import json
import sys
import time
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import (
app,
db,
logs,
paths,
player,
store,
store_threads,
ui,
)
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
_REPORT_EVERY_S = 15.0
_DESCRIPTION = (
"Scrape 4chan /wsg/ for videos and triage them with a keep/pass reviewer. "
"Nothing is ever deleted: a pass moves the file into the trash directory "
"for you to clear by hand."
)
def _emit(message: str) -> None:
"""Write a line to stdout.
Args:
message: Line to write, without a trailing newline.
"""
sys.stdout.write(f"{message}\n")
def build_parser() -> argparse.ArgumentParser:
"""Construct the argument parser.
Returns:
argparse.ArgumentParser: Parser with the ``review``, ``scrape`` and
``stats`` subcommands. ``review`` is the default.
"""
parser = argparse.ArgumentParser(
prog="python -m python_pkg.wsg_grabber",
description=_DESCRIPTION,
)
parser.add_argument(
"--log-level",
choices=logs.LEVELS,
default="info",
help="detail written to the session log (default: info)",
)
parser.add_argument(
"--echo-log",
action="store_true",
help="also print log records to stderr as they happen",
)
subparsers = parser.add_subparsers(dest="command")
subparsers.add_parser(
"review",
help="download in the background and review on screen (default)",
)
subparsers.add_parser("stats", help="print index counts and exit")
subparsers.add_parser(
"logs",
help="show where session logs live and summarise the newest one",
)
scrape_parser = subparsers.add_parser(
"scrape",
help="index and download without opening the reviewer",
)
scrape_parser.add_argument(
"--seconds",
type=float,
default=0.0,
help="stop after this long (0 means run until interrupted)",
)
return parser
def main(argv: list[str] | None = None) -> int:
"""Run the requested subcommand.
Args:
argv: Argument list, defaulting to ``sys.argv[1:]``.
Returns:
int: Process exit status.
"""
args = build_parser().parse_args(argv)
logs.start(args.log_level, echo=args.echo_log)
try:
return _dispatch(args)
finally:
logs.stop()
def _dispatch(args: argparse.Namespace) -> int:
"""Run the subcommand named in *args*.
Args:
args: Parsed arguments.
Returns:
int: Process exit status.
"""
if args.command == "logs":
return show_logs()
if args.command == "stats":
return stats()
if args.command == "scrape":
return scrape(args.seconds)
return open_reviewer()
def stats() -> int:
"""Print what the index knows.
Returns:
int: Exit status.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
try:
counts = store.counts(conn)
_emit(f"index {paths.db_path()}")
_emit(f"known files {counts.get('total', 0)}")
for state in FileState:
_emit(f" {state.value:<12} {counts.get(state.value, 0)}")
_emit(f"threads {len(store_threads.known_threads(conn))}")
_emit(f"keep dir {paths.keep_dir()}")
_emit(f"trash dir {paths.trash_dir()} (never emptied automatically)")
finally:
conn.close()
return 0
def scrape(seconds: float, sleeper: Callable[[float], None] | None = None) -> int:
"""Index and download without opening a window.
Args:
seconds: Stop after this long; 0 runs until interrupted.
sleeper: Optional stand-in for :func:`time.sleep`, used by tests to
avoid actually waiting.
Returns:
int: Exit status.
"""
rest = time.sleep if sleeper is None else sleeper
session = app.open_session()
started = time.monotonic()
reported = started
try:
_emit("scraping /wsg/ — Ctrl-C to stop")
while True:
rest(1.0)
now = time.monotonic()
if now - reported >= _REPORT_EVERY_S:
reported = now
_emit(progress(session))
if seconds and now - started >= seconds:
break
except KeyboardInterrupt:
_emit("stopping")
finally:
_emit(progress(session))
session.shutdown()
return 0
def progress(session: app.Session) -> str:
"""Summarise index progress.
Args:
session: The running session.
Returns:
str: One-line summary.
"""
counts = store.counts(session.conn)
return (
f"known {counts.get('total', 0)} · "
f"ready {counts.get(FileState.READY.value, 0)} · "
f"pending {store.pending_downloads(session.conn)}"
)
def open_reviewer() -> int:
"""Open the reviewer window and hand control to it.
Returns:
int: Exit status.
"""
session = app.open_session()
try:
window = ui.ReviewWindow(
app.initial_state(session),
session.events,
ui.Callbacks(
commit=session.commit,
missing=session.missing,
undo=session.undo,
shutdown=session.shutdown,
),
)
window.attach_player(
player.MpvPlayer(window.video_wid(), paths.ipc_socket_path()),
)
window.run()
except BaseException:
# The worker is a non-daemon thread, so anything raised between opening
# the session and entering the Tk loop -- mpv missing, a stale socket --
# would otherwise leave the process alive forever with no window.
logs.error("review.startup_failed")
session.shutdown()
raise
return 0
def show_logs() -> int:
"""Print where logs live and summarise the most recent session.
Returns:
int: Exit status.
"""
directory = logs.logs_dir()
_emit(f"log directory {directory}")
sessions = sorted(directory.glob("session-*.jsonl"))
if not sessions:
_emit("no session logs yet")
return 0
newest = sessions[-1]
_emit(f"newest session {newest}")
_emit(f"sessions kept {len(sessions)}")
_emit("")
_emit(summarise(newest))
_emit("")
_emit("Useful queries:")
_emit(f" jq -c 'select(.level==\"error\")' {newest}")
_emit(f" jq -r 'select(.event==\"review.show\")|.file' {newest}")
_emit(f" jq -c 'select(.event|startswith(\"player.\"))' {newest}")
return 0
def summarise(path: Path) -> str:
"""Count events by name and level in one session log.
Args:
path: A session JSONL file.
Returns:
str: A short human-readable tally.
"""
by_event: Counter[str] = Counter()
by_level: Counter[str] = Counter()
for line in path.read_text(encoding="utf-8").splitlines():
try:
record = json.loads(line)
except ValueError:
continue
by_event[str(record.get("event"))] += 1
by_level[str(record.get("level"))] += 1
levels = " ".join(f"{name}={count}" for name, count in sorted(by_level.items()))
lines = [f"records by level: {levels or 'none'}", "events:"]
lines.extend(f" {name:<24} {count}" for name, count in by_event.most_common(15))
return "\n".join(lines)

104
python_pkg/wsg_grabber/constants.py Normal file
View File

@ -0,0 +1,104 @@
"""Fixed configuration for the /wsg/ grabber.
Values here are deliberately plain data so the pure planning helpers in
:mod:`python_pkg.wsg_grabber.scanner` can be exercised without touching the
network, the filesystem or a display.
"""
from __future__ import annotations
from types import MappingProxyType
from typing import TYPE_CHECKING, Final
if TYPE_CHECKING:
from collections.abc import Mapping
BOARD: Final[str] = "wsg"
# The only board this tool targets.
API_HOST: Final[str] = "https://a.4cdn.org"
MEDIA_HOST: Final[str] = "https://i.4cdn.org"
BOARDS_URL: Final[str] = "https://boards.4chan.org/"
THREADS_URL: Final[str] = f"{API_HOST}/{BOARD}/threads.json"
ARCHIVE_URL: Final[str] = f"{API_HOST}/{BOARD}/archive.json"
VIDEO_EXTENSIONS: Final[frozenset[str]] = frozenset({".webm", ".mp4", ".gif"})
# /wsg/ serves all three. A live catalogue sample was 55% webm, 37% mp4, 8%
# gif, so filtering to {".webm", ".gif"} would silently drop a third of it.
# i.4cdn.org answers a bare request with 429 once it has seen a burst, so the
# session presents itself as a browser following a link from the board. The
# cookies 4chan sets in response (``__cf_bm``, ``_cfuvid``) are why a single
# long-lived Session is mandatory rather than merely efficient.
USER_AGENT: Final[str] = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"
)
API_HEADERS: Final[Mapping[str, str]] = MappingProxyType(
{
"User-Agent": USER_AGENT,
"Accept": "application/json,text/plain,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
# Ask for no compression so a hostile or compromised host cannot turn a
# few hundred KB on the wire into gigabytes of decoded JSON.
"Accept-Encoding": "identity",
"Referer": BOARDS_URL,
},
)
MEDIA_HEADERS: Final[Mapping[str, str]] = MappingProxyType(
{
"User-Agent": USER_AGENT,
"Accept": "video/webm,video/mp4,image/gif,video/*;q=0.9,*/*;q=0.5",
"Accept-Language": "en-US,en;q=0.9",
# Never gzip a webm: it cannot compress and it breaks Range resumption.
"Accept-Encoding": "identity",
"Referer": BOARDS_URL,
"Sec-Fetch-Dest": "video",
"Sec-Fetch-Mode": "no-cors",
"Sec-Fetch-Site": "cross-site",
},
)
MIN_REQUEST_INTERVAL_S: Final[float] = 1.0
# The 4chan API rules state at most one request per second.
BACKOFF_BASE_S: Final[float] = 2.0
BACKOFF_CAP_S: Final[float] = 120.0
MAX_ATTEMPTS: Final[int] = 3
# Download attempts before a file is written off as ``gone``.
CONNECT_TIMEOUT_S: Final[float] = 10.0
READ_TIMEOUT_S: Final[float] = 30.0
CHUNK_BYTES: Final[int] = 65536
# Refuse an API response larger than this; the real ones are ~300 KB.
MAX_JSON_BYTES: Final[int] = 32 * 1024 * 1024
# How far a download may exceed its advertised size before it is abandoned.
SIZE_SLACK_BYTES: Final[int] = 1024 * 1024
SCHEMA_VERSION: Final[int] = 3
DB_TIMEOUT_S: Final[float] = 30.0
DB_BUSY_TIMEOUT_MS: Final[int] = 5000
POLL_INTERVAL_MS: Final[int] = 150
# How often the Tk loop drains the downloader's event queue.
IDLE_POLL_S: Final[float] = 1.0
# How long the worker sleeps per tick once the board is exhausted.
IDLE_RESCAN_S: Final[float] = 60.0
# How long a finished sweep stays quiet before the board is walked again.
JOIN_TIMEOUT_S: Final[float] = 10.0
IPC_READY_TIMEOUT_S: Final[float] = 10.0
CONTROL_BAR_HEIGHT_PX: Final[int] = 96
WINDOW_WIDTH_PX: Final[int] = 1100
WINDOW_HEIGHT_PX: Final[int] = 800
KEEP_KEYS: Final[tuple[str, ...]] = ("<k>", "<Right>")
PASS_KEYS: Final[tuple[str, ...]] = ("<j>", "<space>", "<Left>")
UNDO_KEYS: Final[tuple[str, ...]] = ("<u>", "<BackSpace>")
QUIT_KEYS: Final[tuple[str, ...]] = ("<q>", "<Escape>")

164
python_pkg/wsg_grabber/db.py Normal file
View File

@ -0,0 +1,164 @@
"""Schema definition, connection setup and migration for the file index.
sqlite rather than the JSON state files used elsewhere in this repo: two threads
write concurrently (the downloader appending rows, the UI recording verdicts),
the seen-set grows without bound, and "have I seen this md5" must stay O(1).
Rewriting a whole JSON document per verdict would not survive either property.
``check_same_thread`` is deliberately left at its default so a connection that
escapes its owning thread raises loudly instead of corrupting the file.
"""
from __future__ import annotations
import sqlite3
from typing import TYPE_CHECKING, Final
from python_pkg.wsg_grabber.constants import (
DB_BUSY_TIMEOUT_MS,
DB_TIMEOUT_S,
SCHEMA_VERSION,
)
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
_SCHEMA: Final[tuple[str, ...]] = (
"""
CREATE TABLE IF NOT EXISTS files (
md5 TEXT PRIMARY KEY,
tim INTEGER NOT NULL,
ext TEXT NOT NULL,
orig_name TEXT NOT NULL,
fsize INTEGER NOT NULL,
width INTEGER NOT NULL,
height INTEGER NOT NULL,
thread_no INTEGER NOT NULL,
post_no INTEGER NOT NULL,
state TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
bytes_done INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
local_name TEXT,
first_seen TEXT NOT NULL,
downloaded_at TEXT,
reviewed_at TEXT,
reviewed_name TEXT
)
""",
"CREATE INDEX IF NOT EXISTS idx_files_state ON files(state)",
"CREATE INDEX IF NOT EXISTS idx_files_claim ON files(state, first_seen)",
"""
CREATE TABLE IF NOT EXISTS threads (
thread_no INTEGER PRIMARY KEY,
api_last_mod INTEGER NOT NULL,
http_last_mod TEXT,
last_checked TEXT NOT NULL,
status TEXT NOT NULL
)
""",
"CREATE INDEX IF NOT EXISTS idx_threads_status ON threads(status)",
"CREATE INDEX IF NOT EXISTS idx_files_reviewed ON files(reviewed_at)",
"""
CREATE TABLE IF NOT EXISTS http_cache (
url TEXT PRIMARY KEY,
last_modified TEXT NOT NULL
)
""",
)
class SchemaTooNewError(RuntimeError):
"""Raised when the index was written by a newer version of this tool."""
def connect(path: Path) -> sqlite3.Connection:
"""Open *path* with the pragmas this package relies on.
WAL is what lets the downloader thread write while the UI thread reads and
writes verdicts, without either blocking the other.
Args:
path: Location of the sqlite file; its parent must already exist.
Returns:
sqlite3.Connection: Autocommit connection with row access by name.
"""
conn = sqlite3.connect(path, timeout=DB_TIMEOUT_S, isolation_level=None)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
conn.execute(f"PRAGMA busy_timeout={DB_BUSY_TIMEOUT_MS}")
conn.execute("PRAGMA foreign_keys=ON")
return conn
def ensure_schema(conn: sqlite3.Connection) -> int:
"""Create or migrate the schema, returning the version now in force.
Args:
conn: Open connection.
Returns:
int: The schema version after this call.
Raises:
SchemaTooNewError: If the file was written by a newer release, in which
case silently downgrading it would lose data.
"""
version = int(conn.execute("PRAGMA user_version").fetchone()[0])
if version > SCHEMA_VERSION:
msg = (
f"index schema is version {version} but this build understands "
f"at most {SCHEMA_VERSION}; upgrade the tool instead of "
f"downgrading the index"
)
raise SchemaTooNewError(msg)
if version == SCHEMA_VERSION:
return version
for statement in _SCHEMA:
conn.execute(statement)
# A fresh database already has the current shape from _SCHEMA, so most
# steps have no migration. Each one that does is idempotent, which keeps
# this safe either way.
for step in range(version, SCHEMA_VERSION):
migrate = _MIGRATIONS.get(step)
if migrate is not None:
migrate(conn)
conn.execute(f"PRAGMA user_version={SCHEMA_VERSION}")
return SCHEMA_VERSION
def _to_v2(conn: sqlite3.Connection) -> None:
"""Add the column that makes a verdict reversible after a restart.
Undo needs to know the name the file was given in keep/ or trash/, which
may differ from its name in incoming/ when a collision forced a rename.
Args:
conn: Open connection.
"""
columns = {row["name"] for row in conn.execute("PRAGMA table_info(files)")}
if "reviewed_name" not in columns:
conn.execute("ALTER TABLE files ADD COLUMN reviewed_name TEXT")
# Keyed by the version being upgraded FROM.
_MIGRATIONS: Final[dict[int, Callable[[sqlite3.Connection], None]]] = {
1: _to_v2,
}
def open_index(path: Path) -> sqlite3.Connection:
"""Open *path* and bring its schema up to date.
Args:
path: Location of the sqlite file.
Returns:
sqlite3.Connection: Ready-to-use connection.
"""
conn = connect(path)
ensure_schema(conn)
return conn

420
python_pkg/wsg_grabber/downloader.py Normal file
View File

@ -0,0 +1,420 @@
"""The background worker: walk the board, fetch files, announce each one.
Structured so that :meth:`Worker.run_once` performs exactly one unit of work
and returns. The threaded loop is a thin wrapper around it, which is what lets
the tests drive the whole state machine synchronously without starting a thread
-- important because pytest treats warnings as errors and a leaked thread would
fail the suite.
The worker never touches Tk. It publishes events onto a queue that the UI
drains from its own event loop.
"""
from __future__ import annotations
from dataclasses import dataclass, field
import logging
import sqlite3
import threading
import time
from typing import TYPE_CHECKING
import requests
from python_pkg.wsg_grabber import catalog, net, scanner, store, store_threads
from python_pkg.wsg_grabber.constants import (
API_HOST,
ARCHIVE_URL,
BOARD,
IDLE_POLL_S,
IDLE_RESCAN_S,
THREADS_URL,
)
from python_pkg.wsg_grabber.models import (
Downloaded,
FileReady,
Indexed,
JsonResponse,
Outcome,
ScanFinished,
TaskKind,
ThreadRef,
WorkerFailed,
)
from python_pkg.wsg_grabber.states import (
CLAIMABLE,
FileEvent,
FileState,
next_state,
)
_LOGGER = logging.getLogger(__name__)
# Every failure family this worker can plausibly hit: the network, the disk,
# the index, and bad data coming back from the board. Named explicitly rather
# than catching Exception, which the linters reject and which would also
# swallow genuine programming errors that ought to be loud.
_RECOVERABLE = (
ArithmeticError,
OSError,
RuntimeError,
ValueError,
LookupError,
TypeError,
AttributeError,
sqlite3.Error,
requests.RequestException,
)
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
import queue
from python_pkg.wsg_grabber.models import DownloadEvent, RemoteFile
@dataclass
class WorkerDeps:
"""Collaborators supplied to the worker, so tests can swap them out."""
conn: sqlite3.Connection
session: requests.Session
incoming: Path
events: queue.SimpleQueue[DownloadEvent]
stop: threading.Event = field(default_factory=threading.Event)
include_archive: bool = True
class Worker:
"""Walks /wsg/ and downloads every file it has not seen before."""
def __init__(self, deps: WorkerDeps) -> None:
"""Store collaborators and initialise scan bookkeeping.
Args:
deps: Connection, session, destination and event queue.
"""
self._deps = deps
self._pending_threads: list[ThreadRef] = []
self._catalog_fresh = False
self._downloaded = 0
self._last_request = 0.0
self._finished_announced = False
self._sweep_finished_at = 0.0
@property
def downloaded(self) -> int:
"""Return how many files this worker has fetched.
Returns:
int: Completed downloads since construction.
"""
return self._downloaded
def run(self) -> None:
"""Loop until the stop flag is set, reporting any crash to the UI.
Failures on a background thread would otherwise vanish, leaving the
reviewer waiting for downloads that never arrive, so anything in
``_RECOVERABLE`` is logged and surfaced in the status bar. The session
and connection are closed on every exit path. The suite does not police
that for us: ``meta/pyproject.toml`` downgrades
``PytestUnraisableExceptionWarning``, so a leak here would pass CI
unnoticed.
"""
try:
while not self._deps.stop.is_set():
self.run_once()
except _RECOVERABLE as exc:
_LOGGER.exception("wsg download worker stopped")
self._publish(WorkerFailed(message=str(exc)))
finally:
self._deps.session.close()
self._deps.conn.close()
def run_once(self) -> TaskKind:
"""Perform one unit of work and return what it was.
Returns:
TaskKind: The action taken, so a caller can drive the worker step
by step.
"""
task = scanner.choose_task(
store.pending_downloads(self._deps.conn),
len(self._pending_threads),
catalog_fresh=self._catalog_fresh,
)
if task is TaskKind.DOWNLOAD:
self._download_one()
elif task is TaskKind.SCAN:
self._scan_step()
else:
self._idle()
return task
def _idle(self) -> None:
"""Report the board exhausted, then poll again after a quiet period.
The rescan is deliberately not immediate. Resetting the catalogue every
idle tick would walk the board once a second and re-announce completion
each time; waiting keeps a finished sweep quiet until there is a real
chance of new content.
"""
self._announce_finished()
self._deps.stop.wait(IDLE_POLL_S)
if time.monotonic() - self._sweep_finished_at >= IDLE_RESCAN_S:
self._catalog_fresh = False
self._finished_announced = False
def _scan_step(self) -> None:
"""Read the thread list, or fetch the next thread that changed."""
if not self._catalog_fresh:
self._refresh_thread_list()
return
ref = self._pending_threads.pop(0)
self._fetch_thread(ref)
def _refresh_thread_list(self) -> None:
"""Fetch ``threads.json`` and work out which threads to visit."""
response = self._conditional_get(THREADS_URL)
live = catalog.parse_thread_list(response.payload)
if self._deps.include_archive:
live = live + self._archive_refs()
known = store_threads.known_threads(self._deps.conn)
self._pending_threads = scanner.stale_threads(known, live)
self._catalog_fresh = True
self._finished_announced = False
def _archive_refs(self) -> list[ThreadRef]:
"""Fetch the archive listing, tolerating boards without one.
Returns:
list[ThreadRef]: Archived threads, empty when unavailable.
"""
response = self._conditional_get(ARCHIVE_URL)
if response.not_found:
return []
return catalog.parse_archive(response.payload)
def _conditional_get(self, url: str) -> JsonResponse:
"""Fetch a board-level endpoint, sending back the stamp we hold.
Without this an idle reviewer re-fetches threads.json and archive.json
unconditionally every cycle. With it an unchanged board costs a 304.
Args:
url: Absolute API URL.
Returns:
JsonResponse: The response; on 304 the payload is None and the
caller keeps whatever it already knew.
"""
stamp = store_threads.cached_last_modified(self._deps.conn, url)
response = self._get_json(url, stamp)
if not response.not_modified:
store_threads.record_last_modified(
self._deps.conn,
url,
response.last_modified,
)
return response
def _fetch_thread(self, ref: ThreadRef) -> None:
"""Read one thread and record any files it introduces.
Args:
ref: Thread to fetch.
"""
url = f"{API_HOST}/{BOARD}/thread/{ref.thread_no}.json"
stamp = store_threads.http_last_modified(self._deps.conn, ref.thread_no)
response = self._get_json(url, stamp)
if response.not_found:
store_threads.mark_thread_gone(self._deps.conn, ref.thread_no)
return
if response.not_modified:
store_threads.record_thread(self._deps.conn, ref, stamp)
return
parsed = catalog.parse_thread(ref.thread_no, response.payload)
known = store.known_md5s(self._deps.conn)
added = store.record_files(
self._deps.conn,
catalog.new_files(parsed, known),
)
self._write_off_deleted(catalog.deleted_md5s(response.payload), known)
store_threads.record_thread(self._deps.conn, ref, response.last_modified)
if added:
self._publish(Indexed(known=len(known) + added))
def _write_off_deleted(self, gone: set[str], known: set[str]) -> None:
"""Mark not-yet-fetched files terminal when the board drops them.
Only files still waiting to download are affected. Once the bytes are
on disk the post being deleted upstream is irrelevant, so an already
downloaded or reviewed file keeps its state -- the same rule the
transition table encodes by rejecting ``READY + DELETED_UPSTREAM``.
Args:
gone: md5s the board reports as deleted.
known: md5s already in the index.
"""
for digest in gone & known:
if store.state_of(self._deps.conn, digest) not in CLAIMABLE:
continue
store.set_state(
self._deps.conn,
digest,
FileState.GONE,
error="deleted upstream",
)
def _download_one(self) -> None:
"""Claim the next pending file and fetch it."""
claimed = store.claim_next(self._deps.conn)
if claimed is None:
return
part = (
self._deps.incoming
/ f"{store.local_name(claimed.md5, claimed.tim, claimed.ext)}.part"
)
self._wait_for_slot()
result = net.download(
self._deps.session,
net.Transfer(
url=catalog.media_url(claimed.tim, claimed.ext),
part_path=part,
expected_md5=claimed.md5,
expected_size=claimed.fsize,
should_stop=self._deps.stop.is_set,
on_progress=lambda done: store.record_progress(
self._deps.conn,
claimed.md5,
done,
),
),
)
self._apply_result(claimed, part, result.outcome, result.retry_after)
def _apply_result(
self,
claimed: RemoteFile,
part: Path,
outcome: Outcome,
retry_after: float | None,
) -> None:
"""Translate a download outcome into index state and an event.
Args:
claimed: The file that was attempted.
part: Its partial-file path.
outcome: How the download ended.
retry_after: Server-requested pause, when supplied.
"""
if outcome is Outcome.ABORTED:
store.set_state(self._deps.conn, claimed.md5, FileState.FAILED)
return
if outcome is Outcome.COMPLETED:
final = part.with_suffix("")
part.replace(final)
store.mark_downloaded(self._deps.conn, claimed.md5, final.name)
self._downloaded += 1
self._publish(
FileReady(
item=store.ready_item(claimed, final),
),
)
self._publish(Downloaded(count=self._downloaded))
return
event = {
Outcome.NOT_FOUND: FileEvent.NOT_FOUND,
Outcome.CHECKSUM_MISMATCH: FileEvent.CHECKSUM_MISMATCH,
Outcome.TRANSIENT: FileEvent.TRANSIENT_ERROR,
}[outcome]
attempts = store.attempts_for(self._deps.conn, claimed.md5)
store.set_state(
self._deps.conn,
claimed.md5,
next_state(FileState.DOWNLOADING, event, attempts),
error=outcome.value,
)
if retry_after is not None:
self._deps.stop.wait(scanner.backoff_seconds(attempts, retry_after))
def _announce_finished(self) -> None:
"""Emit ScanFinished once per completed sweep of the board."""
if self._finished_announced:
return
self._finished_announced = True
self._sweep_finished_at = time.monotonic()
self._publish(ScanFinished())
def _get_json(self, url: str, stamp: str | None = None) -> JsonResponse:
"""Fetch JSON while respecting the one-request-per-second rule.
Args:
url: Absolute API URL.
stamp: Stored ``Last-Modified`` to send conditionally.
Returns:
JsonResponse: The decoded response.
"""
self._wait_for_slot()
return net.get_json(self._deps.session, url, stamp)
def _wait_for_slot(self) -> None:
"""Sleep just long enough to stay within the API's rate limit."""
now = time.monotonic()
delay = scanner.next_delay(self._last_request, now)
if delay > 0:
self._deps.stop.wait(delay)
self._last_request = time.monotonic()
def _publish(self, event: DownloadEvent) -> None:
"""Hand an event to the UI thread.
Args:
event: What happened.
"""
self._deps.events.put(event)
def start(
build_deps: Callable[[], WorkerDeps],
on_failure: Callable[[str], None] = lambda _message: None,
) -> threading.Thread:
"""Run a worker on its own non-daemon thread.
A *factory* rather than ready-made dependencies, because sqlite forbids
using a connection from any thread but the one that opened it. Building
them inside the thread keeps ownership and disposal on the same side of the
boundary; passing a pre-opened connection in raises ProgrammingError the
moment the worker closes it.
Args:
build_deps: Called on the new thread to open the connection, session
and everything else the worker owns.
on_failure: Called with a message when *build_deps* itself fails, so
the UI learns there will be no downloads.
Returns:
threading.Thread: The started thread; set the stop flag, then join it.
"""
def _run() -> None:
try:
deps = build_deps()
except _RECOVERABLE as exc:
# Worker.run reports its own failures, but a crash while building
# the connection and session happens before there is a Worker at
# all. Without this the reviewer waits on downloads forever and the
# only trace is threading.excepthook.
_LOGGER.exception("wsg worker could not start")
on_failure(str(exc))
return
Worker(deps).run()
thread = threading.Thread(target=_run, name="wsg-downloader", daemon=False)
thread.start()
return thread

64
python_pkg/wsg_grabber/files.py Normal file
View File

@ -0,0 +1,64 @@
"""The only code in this package that moves bytes around on disk.
Deliberately tiny, because everything interesting about a verdict was already
decided in :mod:`python_pkg.wsg_grabber.verdict`. There is no ``unlink`` here:
a passed video is relocated to ``trash/`` and stays there until the user
removes it.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import logs
if TYPE_CHECKING:
from pathlib import Path
from python_pkg.wsg_grabber.models import FileMove
def existing_names(directory: Path) -> set[str]:
"""Return the filenames already present in *directory*.
Args:
directory: Directory to inspect; need not exist.
Returns:
set[str]: Names currently in the directory.
"""
if not directory.is_dir():
return set()
return {entry.name for entry in directory.iterdir()}
def apply_move(move: FileMove) -> bool:
"""Carry out a planned relocation.
Refuses to overwrite an existing destination. ``Path.replace`` clobbers
silently, so without this check a caller that forgot to pick a
collision-free name would destroy a video the user had already decided to
keep -- exactly the thing this package promises never to do. Callers pick
the name with :func:`verdict.unique_destination`; this is the backstop that
makes the guarantee structural rather than a convention.
Args:
move: Where the file is and where it should go.
Returns:
bool: True when the file was moved. False when the source had already
vanished, or when the destination is occupied -- the caller treats
either as "this file is not where I thought", never as success.
"""
if not move.src.exists():
return False
if move.dst.exists():
logs.error(
"files.destination_occupied",
src=str(move.src),
dst=str(move.dst),
)
return False
move.dst.parent.mkdir(parents=True, exist_ok=True)
move.src.replace(move.dst)
return True

83
python_pkg/wsg_grabber/install.sh Executable file
View File

@ -0,0 +1,83 @@
#!/bin/bash
# ============================================================================
# Install the /wsg/ grabber: system dependencies plus a launcher on PATH.
# ============================================================================
set -euo pipefail
SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_NAME
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
readonly REPO_ROOT
readonly LAUNCHER="/usr/local/bin/wsg-grabber"
# mpv provides the player; tk provides the window. Both are runtime-only, and
# the package deliberately depends on neither via pip: python-mpv resolves
# libmpv at import time and would break any machine without it.
readonly PACKAGES=(mpv tk)
usage() {
echo "Usage: $SCRIPT_NAME [--uninstall]"
echo " Installs mpv + tk and a 'wsg-grabber' launcher."
exit 0
}
require_root() {
if [[ $EUID -ne 0 ]]; then
echo "Error: needs root to write $LAUNCHER and install packages." >&2
echo "Re-run with: sudo $0" >&2
exit 1
fi
}
install_packages() {
local missing=()
local package
for package in "${PACKAGES[@]}"; do
if ! pacman -Qq "$package" &>/dev/null; then
missing+=("$package")
fi
done
if [[ ${#missing[@]} -eq 0 ]]; then
echo "System packages already present: ${PACKAGES[*]}"
return
fi
echo "Installing: ${missing[*]}"
pacman -S --needed --noconfirm "${missing[@]}"
}
write_launcher() {
cat > "$LAUNCHER" <<EOF
#!/bin/bash
# Launcher for the /wsg/ grabber. Generated by $SCRIPT_NAME.
set -euo pipefail
export PYTHONPATH="$REPO_ROOT\${PYTHONPATH:+:\$PYTHONPATH}"
exec python3 -m python_pkg.wsg_grabber "\$@"
EOF
chmod 755 "$LAUNCHER"
echo "Installed $LAUNCHER"
}
uninstall() {
rm -f "$LAUNCHER"
echo "Removed $LAUNCHER"
echo "Downloaded videos in ~/.local/share/wsg_grabber were left alone."
}
main() {
require_root
install_packages
write_launcher
echo
echo "Done. Try:"
echo " wsg-grabber stats"
echo " wsg-grabber # opens the reviewer"
}
case "${1:-}" in
-h|--help) usage ;;
--uninstall) require_root; uninstall ;;
"") main ;;
*) echo "Unknown option: $1" >&2; exit 1 ;;
esac

192
python_pkg/wsg_grabber/logs.py Normal file
View File

@ -0,0 +1,192 @@
r"""Structured, machine-readable event logging.
Every interesting thing this package does emits one JSON object per line, so a
session can be reconstructed afterwards without a debugger and without having
reproduced the problem live. That matters here more than usual: the parts most
likely to misbehave -- what mpv actually did with a command, whether a file was
still on disk when it was played -- sit exactly on the boundaries the test
suite has to mock, so tests cannot see them.
Reading a log back:
jq -c 'select(.event|startswith("player."))' <logfile>
jq -r 'select(.level=="error") | "\\(.ts) \\(.event) \\(.detail)"' <file>
jq -r 'select(.event=="review.show") | .file' <file> # what was shown, in order
Field names are stable. Every record carries ``ts``, ``seq``, ``level``,
``event`` and ``thread``; anything else is event-specific and always appears
under the same key.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import datetime, timezone
import itertools
import json
import os
import sys
import threading
from typing import TYPE_CHECKING, Final, TextIO
from python_pkg.wsg_grabber import paths
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
LEVELS: Final[tuple[str, ...]] = ("debug", "info", "warning", "error")
_RANK: Final[dict[str, int]] = {name: index for index, name in enumerate(LEVELS)}
@dataclass
class _Session:
"""Where records go and which ones are kept."""
sink: TextIO | None = None
path: Path | None = None
threshold: int = _RANK["info"]
echo: bool = False
lock: threading.Lock = field(default_factory=threading.Lock)
counter: Iterator[int] = field(default_factory=lambda: itertools.count(1))
_SESSION = _Session()
def _free_path(directory: Path, stem: str) -> Path:
"""Return a log path that does not already exist.
The stem carries a whole-second timestamp and the pid, so restarting twice
inside one second would otherwise append a second session to the first
one's file and make the record numbering look corrupt.
Args:
directory: Where logs live.
stem: Preferred filename without its extension.
Returns:
Path: An unused path.
"""
candidate = directory / f"{stem}.jsonl"
suffix = 2
while candidate.exists():
candidate = directory / f"{stem}-{suffix}.jsonl"
suffix += 1
return candidate
def logs_dir() -> Path:
"""Return the directory holding session logs.
Returns:
Path: ``<data dir>/logs``.
"""
return paths.data_dir() / "logs"
def current_path() -> Path | None:
"""Return the log file this process is writing to.
Returns:
Path | None: The path, or None when logging has not been started.
"""
return _SESSION.path
def start(level: str = "info", *, echo: bool = False) -> Path:
"""Begin writing a session log and return its path.
Args:
level: Lowest level to record; one of :data:`LEVELS`.
echo: Also mirror records to stderr, for watching a run live.
Returns:
Path: The file now being written.
"""
stop()
directory = logs_dir()
directory.mkdir(parents=True, exist_ok=True)
stamp = datetime.now(tz=timezone.utc).strftime("%Y%m%dT%H%M%S")
path = _free_path(directory, f"session-{stamp}-{os.getpid()}")
with _SESSION.lock:
_SESSION.sink = path.open("a", encoding="utf-8")
_SESSION.path = path
_SESSION.threshold = _RANK.get(level, _RANK["info"])
_SESSION.echo = echo
# Restart numbering: seq orders records within one file, so carrying it
# over from a previous session would make the first line read as if
# records had been lost.
_SESSION.counter = itertools.count(1)
event("session.start", level_wanted=level, log=str(path), pid=os.getpid())
return path
def stop() -> None:
"""Close the session log, if one is open."""
with _SESSION.lock:
if _SESSION.sink is not None:
_SESSION.sink.flush()
_SESSION.sink.close()
_SESSION.sink = None
_SESSION.path = None
def event(name: str, level: str = "info", **fields: object) -> None:
"""Record one event.
Args:
name: Dotted event name, e.g. ``player.command``.
level: One of :data:`LEVELS`.
**fields: Event-specific values. Anything not JSON-encodable is
stringified rather than dropped, so a log line is never lost to a
serialisation error.
"""
if _RANK.get(level, _RANK["info"]) < _SESSION.threshold:
return
record = {
"ts": datetime.now(tz=timezone.utc).isoformat(),
"seq": next(_SESSION.counter),
"level": level,
"event": name,
"thread": threading.current_thread().name,
**fields,
}
line = json.dumps(record, default=str)
with _SESSION.lock:
if _SESSION.sink is not None:
_SESSION.sink.write(f"{line}\n")
_SESSION.sink.flush()
if _SESSION.echo:
sys.stderr.write(f"{line}\n")
def debug(name: str, **fields: object) -> None:
"""Record a debug-level event.
Args:
name: Dotted event name.
**fields: Event-specific values.
"""
event(name, level="debug", **fields)
def warning(name: str, **fields: object) -> None:
"""Record a warning-level event.
Args:
name: Dotted event name.
**fields: Event-specific values.
"""
event(name, level="warning", **fields)
def error(name: str, **fields: object) -> None:
"""Record an error-level event.
Args:
name: Dotted event name.
**fields: Event-specific values.
"""
event(name, level="error", **fields)

209
python_pkg/wsg_grabber/models.py Normal file
View File

@ -0,0 +1,209 @@
"""Plain-data records shared across the package.
Everything crossing a module boundary is a frozen dataclass or an enum, so the
pure planning code can be driven from hand-built literals and the GUI never has
to be constructed to test a decision.
"""
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from pathlib import Path
# ``str, Enum`` rather than ``StrEnum`` throughout: mypy runs with
# ``python_version = "3.10"`` and ``StrEnum`` only exists from 3.11, so it would
# degrade to ``Any`` and silently disable type checking on every enum here.
class Verdict(str, Enum):
"""What the user decided about a video.
The member is ``SKIP`` rather than ``PASS`` only because ruff's S105 reads a
name of ``PASS`` as a hardcoded password and this repo forbids ``noqa``. The
stored value stays ``"pass"``, which is the word the UI shows.
"""
KEEP = "keep"
SKIP = "pass"
class TaskKind(str, Enum):
"""What the background worker should do next."""
DOWNLOAD = "download"
SCAN = "scan"
IDLE = "idle"
class Emptiness(str, Enum):
"""Why the reviewer has nothing to show."""
NOT_EMPTY = "not_empty"
WAITING = "waiting"
EXHAUSTED = "exhausted"
@dataclass(frozen=True, slots=True)
class Size:
"""A pixel size."""
width: int
height: int
@dataclass(frozen=True, slots=True)
class ThreadRef:
"""A thread as advertised by ``threads.json``."""
thread_no: int
api_last_modified: int
@dataclass(frozen=True, slots=True)
class RemoteFile:
"""An attachment described by a post, before anything is downloaded.
``md5`` is the API's own ``base64(md5(file_bytes))`` and is the identity of
the file everywhere in this package.
"""
md5: str
tim: int
ext: str
orig_name: str
fsize: int
width: int
height: int
thread_no: int
post_no: int
@dataclass(frozen=True, slots=True)
class ReviewItem:
"""A downloaded video waiting for a verdict."""
md5: str
path: Path
orig_name: str
fsize: int
width: int
height: int
@dataclass(frozen=True, slots=True)
class FileMove:
"""A planned relocation. Applying it is the only filesystem write."""
md5: str
src: Path
dst: Path
@dataclass(frozen=True, slots=True)
class ReviewedItem:
"""A verdict recorded in the index, and therefore reversible.
``reviewed_name`` is the name the file was given in keep/ or trash/, which
differs from ``item.path.name`` whenever a collision forced a rename. Undo
needs it to find the file again, which is why it is persisted rather than
held in memory.
"""
item: ReviewItem
choice: Verdict
reviewed_name: str
@dataclass(frozen=True, slots=True)
class ResumePlan:
"""How to continue a partial download."""
offset: int
discard: bool
class Outcome(str, Enum):
"""How a download ended."""
COMPLETED = "completed"
NOT_FOUND = "not_found"
TRANSIENT = "transient"
CHECKSUM_MISMATCH = "checksum_mismatch"
ABORTED = "aborted"
@dataclass(frozen=True, slots=True)
class JsonResponse:
"""The result of a conditional GET against the API."""
payload: object | None
last_modified: str | None
not_modified: bool
not_found: bool
@dataclass(frozen=True, slots=True)
class DownloadResult:
"""The result of streaming one file to disk."""
outcome: Outcome
bytes_done: int
retry_after: float | None = None
message: str | None = None
@dataclass(frozen=True, slots=True)
class ReviewCommand:
"""Everything the GUI must render or do, already decided.
The window applies this verbatim; it never formats a string or picks a file.
"""
play: Path | None
stop: bool
status: str
title: str
filename: str
verdicts_enabled: bool
undo_enabled: bool
quit_app: bool
@dataclass(frozen=True, slots=True)
class FileReady:
"""A newly downloaded video is available for review."""
item: ReviewItem
@dataclass(frozen=True, slots=True)
class Indexed:
"""The catalogue scan learned about more files."""
known: int
@dataclass(frozen=True, slots=True)
class Downloaded:
"""Running count of files fetched this session."""
count: int
@dataclass(frozen=True, slots=True)
class ScanFinished:
"""The board has been walked and every known file downloaded."""
@dataclass(frozen=True, slots=True)
class WorkerFailed:
"""The background worker died; the message is shown in the status bar."""
message: str
DownloadEvent = FileReady | Indexed | Downloaded | ScanFinished | WorkerFailed

335
python_pkg/wsg_grabber/net.py Normal file
View File

@ -0,0 +1,335 @@
"""The only module that speaks HTTP.
Two hosts with different manners. ``a.4cdn.org`` is polite and supports
``If-Modified-Since``, so a steady-state rescan costs almost nothing.
``i.4cdn.org`` answered a bare request with 429 during development, and the
cookies it sets in reply (``__cf_bm``, ``_cfuvid``) live in the session jar --
which is why one long-lived :class:`requests.Session` is required rather than
merely convenient.
"""
from __future__ import annotations
import base64
from dataclasses import dataclass
import hashlib
import json
from typing import TYPE_CHECKING
import requests
from python_pkg.wsg_grabber.constants import (
API_HEADERS,
CHUNK_BYTES,
CONNECT_TIMEOUT_S,
MAX_JSON_BYTES,
MEDIA_HEADERS,
READ_TIMEOUT_S,
SIZE_SLACK_BYTES,
)
from python_pkg.wsg_grabber.models import DownloadResult, JsonResponse, Outcome
from python_pkg.wsg_grabber.scanner import parse_retry_after, resume_plan
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
_TIMEOUT = (CONNECT_TIMEOUT_S, READ_TIMEOUT_S)
_OK = 200
_PARTIAL = 206
_NOT_MODIFIED = 304
_NOT_FOUND = 404
_GONE = 410
_RANGE_NOT_SATISFIABLE = 416
_TOO_MANY = 429
_SERVER_ERROR = 500
@dataclass(frozen=True, slots=True)
class Transfer:
"""Everything one download needs, bundled to stay within the arg limit."""
url: str
part_path: Path
expected_md5: str
expected_size: int
should_stop: Callable[[], bool]
on_progress: Callable[[int], None] | None = None
def build_session() -> requests.Session:
"""Return a session configured for the API host.
Returns:
requests.Session: Caller owns it and must close it.
"""
session = requests.Session()
session.headers.update(API_HEADERS)
return session
def digest_of(payload: bytes) -> str:
"""Return the base64 MD5 the API uses to identify a file.
MD5 here is 4chan's content identifier, not a security control, hence
``usedforsecurity=False``.
Args:
payload: Full file contents.
Returns:
str: 24-character base64 digest.
"""
digest = hashlib.md5(payload, usedforsecurity=False).digest()
return base64.b64encode(digest).decode("ascii")
def get_json(
session: requests.Session,
url: str,
last_modified: str | None = None,
) -> JsonResponse:
"""Fetch and decode a JSON endpoint, conditionally when possible.
Args:
session: Shared session.
url: Absolute API URL.
last_modified: Previously seen ``Last-Modified`` value, echoed back as
``If-Modified-Since`` so an unchanged thread costs a 304.
Returns:
JsonResponse: Decoded payload, or a flag explaining why there is none.
"""
headers = {"If-Modified-Since": last_modified} if last_modified else {}
response = session.get(url, headers=headers, timeout=_TIMEOUT)
status = response.status_code
if status == _NOT_MODIFIED:
return JsonResponse(
payload=None,
last_modified=last_modified,
not_modified=True,
not_found=False,
)
if status in (_NOT_FOUND, _GONE):
return JsonResponse(
payload=None,
last_modified=None,
not_modified=False,
not_found=True,
)
response.raise_for_status()
body = response.content[: MAX_JSON_BYTES + 1]
if len(body) > MAX_JSON_BYTES:
msg = f"API response exceeded {MAX_JSON_BYTES} bytes"
raise ValueError(msg)
return JsonResponse(
payload=json.loads(body),
last_modified=response.headers.get("Last-Modified"),
not_modified=False,
not_found=False,
)
def download(session: requests.Session, transfer: Transfer) -> DownloadResult:
"""Stream a file to disk, resuming where possible and verifying the result.
The bytes are checked against the API's md5 before the download counts as
finished, so a truncated or mangled transfer can never reach the reviewer.
Args:
session: Shared session.
transfer: What to fetch and where to put it.
Returns:
DownloadResult: Outcome plus the byte count now on disk.
"""
plan = resume_plan(size_of(transfer.part_path), transfer.expected_size)
if plan.discard:
transfer.part_path.unlink(missing_ok=True)
headers = dict(MEDIA_HEADERS)
if plan.offset:
headers["Range"] = f"bytes={plan.offset}-"
try:
return _stream(session, transfer, headers, plan.offset)
except requests.RequestException as exc:
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
message=str(exc),
)
def _stream(
session: requests.Session,
transfer: Transfer,
headers: dict[str, str],
offset: int,
) -> DownloadResult:
"""Perform the request and write the body out.
Args:
session: Shared session.
transfer: What to fetch and where to put it.
headers: Media headers, possibly carrying a ``Range``.
offset: Byte offset requested; zero for a fresh download.
Returns:
DownloadResult: Outcome plus the byte count now on disk.
"""
response = session.get(
transfer.url,
headers=headers,
timeout=_TIMEOUT,
stream=True,
)
rejection = _rejection(response, transfer)
if rejection is not None:
return rejection
# A server that ignores Range replies 200 with the whole file, so anything
# already on disk has to be thrown away rather than appended to.
append = offset > 0 and response.status_code == _PARTIAL
written = _write_body(response, transfer, append=append)
if written is None:
return DownloadResult(
outcome=Outcome.ABORTED,
bytes_done=size_of(transfer.part_path),
)
if _over_budget(written, transfer.expected_size):
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=0,
message=f"body exceeded the advertised {transfer.expected_size} bytes",
)
payload = transfer.part_path.read_bytes()
if digest_of(payload) != transfer.expected_md5:
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.CHECKSUM_MISMATCH,
bytes_done=0,
message="md5 did not match the value published by the API",
)
return DownloadResult(outcome=Outcome.COMPLETED, bytes_done=len(payload))
def _rejection(
response: requests.Response,
transfer: Transfer,
) -> DownloadResult | None:
"""Classify a non-success status, if any.
Args:
response: The (streamed) response.
transfer: The transfer being attempted.
Returns:
DownloadResult | None: None when the response is usable.
"""
status = response.status_code
if status in (_NOT_FOUND, _GONE):
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.NOT_FOUND,
bytes_done=0,
message=f"HTTP {status}",
)
encoding = response.headers.get("Content-Encoding", "identity")
if encoding != "identity":
# We asked for identity. A compressed body means the far side is not
# playing by the rules, and requests would silently expand it -- a
# 200 KB response can decode to 200 MB, and the md5 would still match
# because the same party published it.
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
message=f"refused Content-Encoding: {encoding}",
)
if status == _RANGE_NOT_SATISFIABLE:
# The leftover part is longer than the file actually is; start over.
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=0,
message="range rejected; restarting",
)
if status == _TOO_MANY or status >= _SERVER_ERROR:
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
retry_after=parse_retry_after(response.headers.get("Retry-After")),
message=f"HTTP {status}",
)
if status not in (_OK, _PARTIAL):
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
message=f"unexpected HTTP {status}",
)
return None
def _over_budget(written: int, expected: int) -> bool:
"""Report whether a transfer outgrew the size the API advertised.
Without this a server that lies about Content-Length can write until the
disk is full; the md5 check alone does not help, because whoever serves the
bytes also publishes the digest.
Args:
written: Bytes now on disk.
expected: Size the API declared; zero means it did not say.
Returns:
bool: True when the transfer should be abandoned.
"""
if expected <= 0:
return False
return written > expected + SIZE_SLACK_BYTES
def _write_body(
response: requests.Response,
transfer: Transfer,
*,
append: bool,
) -> int | None:
"""Write the response body to the part file.
Args:
response: Streamed response.
transfer: What is being fetched.
append: Whether to continue an existing part file.
Returns:
int | None: Bytes written, or None when shutdown interrupted it.
"""
mode = "ab" if append else "wb"
done = size_of(transfer.part_path) if append else 0
with transfer.part_path.open(mode) as handle:
for chunk in response.iter_content(chunk_size=CHUNK_BYTES):
if transfer.should_stop():
return None
if not chunk:
continue
handle.write(chunk)
done += len(chunk)
if transfer.on_progress is not None:
transfer.on_progress(done)
if _over_budget(done, transfer.expected_size):
return done
return done
def size_of(path: Path) -> int:
"""Return the size of *path*, or zero when it does not exist.
Args:
path: File to measure.
Returns:
int: Size in bytes.
"""
return path.stat().st_size if path.exists() else 0

85
python_pkg/wsg_grabber/paths.py Normal file
View File

@ -0,0 +1,85 @@
"""Storage locations, resolved lazily.
Every path is a *function*. A module-level ``Path.home() / ...`` constant is
evaluated at import time, which makes it impossible for a test to redirect --
``brother_printer.consumables`` learned that the hard way. Resolving on each
call keeps the whole package redirectable from one fixture.
Nothing in this module deletes anything; ``trash_dir`` is a destination, not a
bin that gets emptied.
"""
from __future__ import annotations
import os
from pathlib import Path
from python_pkg.wsg_grabber.constants import BOARD
_APP_DIRNAME = "wsg_grabber"
def data_dir() -> Path:
"""Return the root directory holding the index and every downloaded file.
Honours ``XDG_DATA_HOME`` when set, else ``~/.local/share``.
Returns:
Path: ``<data home>/wsg_grabber`` (not guaranteed to exist).
"""
override = os.environ.get("XDG_DATA_HOME")
base = Path(override) if override else Path.home() / ".local" / "share"
return base / _APP_DIRNAME
def db_path() -> Path:
"""Return the sqlite index file.
Returns:
Path: ``<data dir>/index.db``.
"""
return data_dir() / "index.db"
def incoming_dir() -> Path:
"""Return the directory holding downloaded, not-yet-reviewed videos.
Returns:
Path: ``<data dir>/incoming``.
"""
return data_dir() / "incoming"
def keep_dir() -> Path:
"""Return the directory videos are moved to when kept.
Returns:
Path: ``<data dir>/keep``.
"""
return data_dir() / "keep"
def trash_dir() -> Path:
"""Return the directory videos are moved to when passed.
Nothing ever removes files from here; clearing it is the user's call.
Returns:
Path: ``<data dir>/trash``.
"""
return data_dir() / "trash"
def ipc_socket_path() -> Path:
"""Return the unix socket used to drive the embedded mpv process.
Returns:
Path: ``<data dir>/mpv-<board>.sock``.
"""
return data_dir() / f"mpv-{BOARD}.sock"
def ensure_dirs() -> None:
"""Create the data, incoming, keep and trash directories if absent."""
for directory in (data_dir(), incoming_dir(), keep_dir(), trash_dir()):
directory.mkdir(parents=True, exist_ok=True)

373
python_pkg/wsg_grabber/player.py Normal file
View File

@ -0,0 +1,373 @@
"""Drives mpv as a subprocess embedded in a Tk frame, over its JSON IPC socket.
Not ``python-mpv``: that binding calls ``ctypes.util.find_library("mpv")`` at
import time and raises ``OSError`` wherever libmpv is absent, which includes
this repo's CI runner. It would pass locally and fail on push, and the usual
lazy-import escape trips ``PLC0415`` with no suppression available. A
subprocess needs only ``subprocess``, ``socket`` and ``json``.
``--wid`` is an mpv feature, so embedding behaves identically either way: mpv
reparents its own window onto the Tk frame and, per ``man mpv``, "will always be
resized to cover the parent window fully". That last part is why there is no
resize method here: setting ``geometry`` on an embedded mpv fights the
automatic sizing and leaves the video drawn at the wrong size.
``build_argv`` and ``encode`` are pure and tested directly. The rest is process
and socket handling: connecting, draining, and shutting down without leaving an
orphan. No reviewing decisions are made here.
"""
from __future__ import annotations
import json
import socket
import subprocess
import threading
import time
from typing import TYPE_CHECKING, Final, Protocol
from python_pkg.wsg_grabber import logs
from python_pkg.wsg_grabber.constants import IPC_READY_TIMEOUT_S
if TYPE_CHECKING:
from collections.abc import Sequence
from pathlib import Path
_SOCKET_POLL_S = 0.05
_READ_CHUNK = 65536
_READ_TIMEOUT_S = 0.2
_READER_JOIN_S = 5.0
_PROBE_ID = 9001
_PROBE_TIMEOUT_S = 2.0
# What to record when diagnosing an apparently frozen reviewer: which file mpv
# holds, whether it is actually advancing, and the size it configured its video
# output to (which changes per video, so a static value is itself a symptom).
_PROBE_PROPERTIES: Final[tuple[str, ...]] = (
"path",
"pause",
"core-idle",
"eof-reached",
"playback-time",
"width",
"height",
)
def _read_reply(conn: socket.socket) -> object:
"""Read until mpv answers the probe request.
Args:
conn: A freshly connected control socket.
Returns:
object: The ``data`` field of the reply, or an ``error:`` string.
"""
buffer = b""
while True:
chunk = conn.recv(65536)
if not chunk:
return "error: socket closed"
buffer += chunk
for line in buffer.split(b"\n"):
if not line:
continue
try:
message = json.loads(line)
except ValueError:
continue
if message.get("request_id") == _PROBE_ID:
return message.get("data")
class Player(Protocol):
"""What the reviewer needs from a video player."""
def play(self, path: Path) -> None:
"""Show *path*, replacing whatever is on screen.
Args:
path: Video file to play.
"""
... # pragma: no cover
def stop(self) -> None:
"""Blank the video area."""
... # pragma: no cover
def is_alive(self) -> bool:
"""Report whether the player is still running.
Returns:
bool: True while the process lives.
"""
... # pragma: no cover
def probe(self) -> dict[str, object]:
"""Report what the player is currently doing, for diagnostics.
Returns:
dict[str, object]: Property name to value.
"""
... # pragma: no cover
def close(self) -> None:
"""Shut the player down and release its resources."""
... # pragma: no cover
def build_argv(wid: int, ipc_path: Path) -> list[str]:
"""Build the mpv command line.
``--input-vo-keyboard=no`` matters: with ``--wid`` mpv reparents a child
window over the Tk frame, and if mpv claims the keyboard the Tk bindings
never see a keypress. Declining it leaves X delivering keys to the focused
toplevel, which is the Tk window.
Args:
wid: X11 window id of the frame to draw into.
ipc_path: Where to create the control socket.
Returns:
list[str]: Argument vector for :func:`subprocess.Popen`.
"""
return [
"mpv",
f"--wid={wid}",
f"--input-ipc-server={ipc_path}",
"--idle=yes",
"--loop-file=inf",
"--keep-open=yes",
"--no-terminal",
"--osc=no",
"--input-default-bindings=no",
"--input-vo-keyboard=no",
"--no-config",
]
def encode(command: Sequence[object]) -> bytes:
"""Encode one mpv IPC command.
Args:
command: Command name followed by its arguments.
Returns:
bytes: Newline-terminated JSON, ready for the socket.
"""
return json.dumps({"command": list(command)}).encode("utf-8") + b"\n"
class MpvPlayer:
"""An mpv process drawing into a Tk frame."""
def __init__(self, wid: int, ipc_path: Path) -> None:
"""Start mpv and connect to its control socket.
Args:
wid: X11 window id of the frame to draw into.
ipc_path: Where mpv should create its control socket.
Raises:
TimeoutError: If mpv does not create the socket in time, which
means it failed to start and there is nothing to talk to.
OSError: If the socket exists but refuses the connection, e.g. a
stale file left by a previous run.
"""
ipc_path.unlink(missing_ok=True)
ipc_path.parent.mkdir(parents=True, exist_ok=True)
self._ipc_path = ipc_path
argv = build_argv(wid, ipc_path)
logs.event("player.start", wid=wid, socket=str(ipc_path), argv=argv)
self._process = subprocess.Popen(argv)
try:
self._socket = self._connect()
except BaseException:
# Reaping matters as much as killing: an unwaited child stays a
# zombie, and the caller is about to abandon this object entirely.
self._process.kill()
self._process.wait(timeout=IPC_READY_TIMEOUT_S)
raise
self._stopping = threading.Event()
self._reader = threading.Thread(
target=self._drain,
name="mpv-ipc-reader",
daemon=False,
)
self._reader.start()
logs.event("player.ready", pid=self._process.pid)
def _drain(self) -> None:
"""Continuously read and discard what mpv sends us.
This is not optional bookkeeping, it is what keeps the player working.
mpv pushes an asynchronous event stream plus a reply per command down
the same socket. A client that only ever writes lets that backlog fill
the kernel buffer; mpv's writer for this connection then blocks, and it
stops reading our commands. The symptom is nasty because nothing
errors: ``sendall`` still succeeds, mpv stays alive and keeps playing
whatever it had, so the reviewer advances through filenames while the
picture never changes. Measured before this existed: commands were
silently ignored from roughly the thirty-fifth video onwards, while the
very same command sent on a fresh connection worked instantly.
Replies carrying an error are logged, which is also how a file mpv
cannot decode becomes visible instead of just looking stuck.
"""
while not self._stopping.is_set():
try:
chunk = self._socket.recv(_READ_CHUNK)
except TimeoutError:
continue
except OSError:
return
if not chunk:
return
self._note_errors(chunk)
def _note_errors(self, chunk: bytes) -> None:
"""Log any failure mpv reported back.
Args:
chunk: Raw bytes just read from the socket.
"""
for line in chunk.split(b"\n"):
if not line.strip():
continue
try:
message = json.loads(line)
except ValueError:
continue
failure = message.get("error")
if isinstance(failure, str) and failure != "success":
logs.warning("player.reply_error", detail=failure, reply=message)
def _connect(self) -> socket.socket:
"""Wait for mpv's socket to appear and connect to it.
Returns:
socket.socket: Connected unix socket.
Raises:
TimeoutError: If the socket never appears.
OSError: If connecting to it fails.
"""
deadline = time.monotonic() + IPC_READY_TIMEOUT_S
while not self._ipc_path.exists():
if time.monotonic() >= deadline:
# __init__ owns killing and reaping the process on every
# failure path; doing it here too would just double up.
msg = f"mpv did not create {self._ipc_path} within the timeout"
raise TimeoutError(msg)
time.sleep(_SOCKET_POLL_S)
control = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
try:
control.connect(str(self._ipc_path))
except BaseException:
control.close()
raise
# so the reader thread can notice the stop flag rather than
# blocking in recv() forever at shutdown
control.settimeout(_READ_TIMEOUT_S)
return control
def _send(self, command: Sequence[object]) -> None:
"""Write a command to mpv.
A failure here is logged rather than raised -- mpv may simply have
exited, and is_alive() reports that to the caller -- but it is never
silently discarded, because a dead socket makes a working reviewer and
a frozen one look identical from the outside.
Args:
command: Command name followed by its arguments.
"""
try:
self._socket.sendall(encode(command))
except OSError as exc:
logs.error(
"player.send_failed",
command=list(command),
detail=str(exc),
alive=self.is_alive(),
)
return
logs.debug("player.command", command=list(command))
def play(self, path: Path) -> None:
"""Show *path*, replacing whatever is on screen.
Args:
path: Video file to play.
"""
logs.event("player.load", file=path.name, path=str(path), exists=path.exists())
self._send(["loadfile", str(path), "replace"])
def stop(self) -> None:
"""Blank the video area."""
self._send(["stop"])
def probe(self) -> dict[str, object]:
"""Ask mpv what it is actually doing.
This exists for after-the-fact diagnosis. If the reviewer ever appears
to freeze on one video while the filenames advance, the recorded answer
settles in one line whether mpv was handed the new file and was
decoding it, or whether the problem is upstream of mpv.
Returns:
dict[str, object]: Property name to value; a property that could
not be read maps to a string starting with ``error:``.
"""
return {name: self._query(name) for name in _PROBE_PROPERTIES}
def _query(self, name: str) -> object:
"""Read one mpv property over a short-lived connection.
A separate connection avoids interleaving with the asynchronous events
mpv pushes down the main socket, which would otherwise make replies
easy to mis-attribute.
Args:
name: Property name.
Returns:
object: The value, or an ``error:`` string.
"""
request = json.dumps(
{"command": ["get_property", name], "request_id": _PROBE_ID}
)
try:
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as probe:
probe.settimeout(_PROBE_TIMEOUT_S)
probe.connect(str(self._ipc_path))
probe.sendall(request.encode("utf-8") + b"\n")
return _read_reply(probe)
except OSError as exc:
return f"error: {exc}"
def is_alive(self) -> bool:
"""Report whether mpv is still running.
Returns:
bool: True while the process lives.
"""
return self._process.poll() is None
def close(self) -> None:
"""Ask mpv to quit, then make sure it has.
Leaving an orphaned mpv behind would keep a window on screen after the
reviewer exits, so a process that ignores ``quit`` is killed.
"""
logs.event("player.close", alive=self.is_alive())
self._stopping.set()
self._send(["quit"])
try:
self._process.wait(timeout=IPC_READY_TIMEOUT_S)
except subprocess.TimeoutExpired:
self._process.kill()
self._process.wait(timeout=IPC_READY_TIMEOUT_S)
self._reader.join(timeout=_READER_JOIN_S)
self._socket.close()
self._ipc_path.unlink(missing_ok=True)

330
python_pkg/wsg_grabber/review.py Normal file
View File

@ -0,0 +1,330 @@
"""The reviewer's behaviour, with no toolkit anywhere in sight.
This module is the whole application as far as behaviour is concerned. It
decides what plays next, what a verdict does, when the queue has run dry and
exactly what the status bar reads. :mod:`python_pkg.wsg_grabber.ui` only
applies the :class:`ReviewCommand` this produces, so every decision is testable
from plain values and none of them need a display.
"""
from __future__ import annotations
from dataclasses import dataclass, replace
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import (
Downloaded,
Emptiness,
FileReady,
Indexed,
ReviewCommand,
ReviewedItem,
ScanFinished,
Verdict,
WorkerFailed,
)
if TYPE_CHECKING:
from collections.abc import Sequence
from python_pkg.wsg_grabber.models import DownloadEvent, ReviewItem
_BYTES_PER_MIB = 1024 * 1024
@dataclass(frozen=True, slots=True)
class SessionState:
"""Everything the reviewer knows, as one immutable value."""
pending: tuple[ReviewItem, ...] = ()
current: ReviewItem | None = None
kept: int = 0
passed: int = 0
indexed: int = 0
downloaded: int = 0
scan_complete: bool = False
quitting: bool = False
error: str | None = None
undoable: int = 0
"""Verdicts recorded in the index that could still be taken back."""
def initial_state(
indexed: int,
ready: Sequence[ReviewItem],
undoable: int = 0,
) -> SessionState:
"""Build the state a session starts from.
Args:
indexed: How many files the index already knows about.
ready: Videos downloaded on a previous run and still unreviewed.
undoable: Verdicts already in the index that can still be reversed, so
undo carries across restarts rather than starting empty.
Returns:
SessionState: Starting state, already showing the first video if any.
"""
state = SessionState(indexed=indexed, undoable=undoable)
return on_new_files(state, ready)
def on_new_files(state: SessionState, items: Sequence[ReviewItem]) -> SessionState:
"""Add freshly downloaded videos to the queue.
The first arrival is promoted straight to the screen, which is what lets
watching start while the rest of the board is still downloading.
Args:
state: Current state.
items: Newly available videos.
Returns:
SessionState: Updated state.
"""
if not items:
return state
queue = [*state.pending, *items]
current = state.current
if current is None:
current = queue.pop(0)
return replace(state, pending=tuple(queue), current=current)
def on_verdict(state: SessionState, choice: Verdict) -> SessionState:
"""Record a decision about the current video and advance.
Args:
state: Current state.
choice: What the user pressed.
Returns:
SessionState: Updated state showing the next video.
"""
if state.current is None:
return state
counted = (
replace(state, kept=state.kept + 1)
if choice is Verdict.KEEP
else replace(state, passed=state.passed + 1)
)
return _advance(replace(counted, undoable=state.undoable + 1))
def can_undo(state: SessionState) -> bool:
"""Report whether there is a verdict to take back.
Args:
state: Current state.
Returns:
bool: True when the index holds at least one reversible verdict.
"""
return state.undoable > 0
def on_undo(state: SessionState, action: ReviewedItem) -> SessionState:
"""Show a restored video again after its verdict was reversed.
The video currently on screen goes to the front of the queue, so changing
your mind never skips anything.
Args:
state: Current state.
action: The verdict that was just reversed on disk and in the index.
Returns:
SessionState: Updated state showing the restored video.
"""
queue = list(state.pending)
if state.current is not None:
queue.insert(0, state.current)
corrected = (
replace(state, kept=max(0, state.kept - 1))
if action.choice is Verdict.KEEP
else replace(state, passed=max(0, state.passed - 1))
)
return replace(
corrected,
undoable=max(0, state.undoable - 1),
pending=tuple(queue),
current=action.item,
)
def forget_last(state: SessionState) -> SessionState:
"""Drop one verdict from the undo count without reversing it.
Used when the file is no longer where the verdict left it, so a second
attempt does not keep failing on the same entry.
Args:
state: Current state.
Returns:
SessionState: Updated state.
"""
return replace(state, undoable=max(0, state.undoable - 1))
def on_missing_locally(state: SessionState) -> SessionState:
"""Drop the current video because its file has disappeared.
Args:
state: Current state.
Returns:
SessionState: Updated state showing the next video.
"""
if state.current is None:
return state
return _advance(state)
def on_event(state: SessionState, event: DownloadEvent) -> SessionState:
"""Fold one background-worker event into the state.
Args:
state: Current state.
event: What the worker reported.
Returns:
SessionState: Updated state.
"""
if isinstance(event, FileReady):
return on_new_files(state, [event.item])
if isinstance(event, Indexed):
return replace(state, indexed=event.known)
if isinstance(event, Downloaded):
return replace(state, downloaded=event.count)
if isinstance(event, ScanFinished):
return replace(state, scan_complete=True)
return replace(state, error=_worker_message(event))
def on_quit(state: SessionState) -> SessionState:
"""Mark the session as finishing.
Args:
state: Current state.
Returns:
SessionState: Updated state.
"""
return replace(state, quitting=True)
def emptiness(state: SessionState) -> Emptiness:
"""Explain why there is nothing to show, if that is the case.
Args:
state: Current state.
Returns:
Emptiness: Whether the queue is merely waiting or genuinely exhausted.
"""
if state.current is not None:
return Emptiness.NOT_EMPTY
return Emptiness.EXHAUSTED if state.scan_complete else Emptiness.WAITING
def status_line(state: SessionState) -> str:
"""Build the text under the video.
Args:
state: Current state.
Returns:
str: One line summarising progress.
"""
if state.error is not None:
return f"downloader stopped: {state.error}"
counts = (
f"kept {state.kept} · passed {state.passed} · "
f"queued {len(state.pending)} · downloaded {state.downloaded}"
)
if state.indexed:
counts = f"{counts}/{state.indexed}"
undo = f" · u undoes ({state.undoable})" if state.undoable else ""
mood = {
Emptiness.NOT_EMPTY: "",
Emptiness.WAITING: " · waiting for downloads…",
Emptiness.EXHAUSTED: " · all caught up",
}[emptiness(state)]
return f"{counts}{mood}{undo}"
def filename_line(state: SessionState) -> str:
"""Describe the video currently on screen.
Args:
state: Current state.
Returns:
str: Original filename with size and dimensions, or a placeholder.
"""
item = state.current
if item is None:
return "—"
size = item.fsize / _BYTES_PER_MIB
return f"{item.orig_name} · {item.width}x{item.height} · {size:.1f} MiB"
def title_line(state: SessionState) -> str:
"""Build the window title.
Args:
state: Current state.
Returns:
str: Title including the running verdict tally.
"""
return f"/wsg/ — kept {state.kept}, passed {state.passed}"
def render(state: SessionState) -> ReviewCommand:
"""Turn the state into instructions the window applies verbatim.
Args:
state: Current state.
Returns:
ReviewCommand: Everything the GUI should show or do.
"""
item = state.current
return ReviewCommand(
play=None if item is None else item.path,
stop=item is None,
status=status_line(state),
title=title_line(state),
filename=filename_line(state),
verdicts_enabled=item is not None and not state.quitting,
undo_enabled=can_undo(state) and not state.quitting,
quit_app=state.quitting,
)
def _advance(state: SessionState) -> SessionState:
"""Move to the next queued video.
Args:
state: State whose current item has been dealt with.
Returns:
SessionState: Updated state.
"""
queue = list(state.pending)
following = queue.pop(0) if queue else None
return replace(state, pending=tuple(queue), current=following)
def _worker_message(event: WorkerFailed) -> str:
"""Extract the message from a worker failure.
Args:
event: The failure report.
Returns:
str: Text for the status bar.
"""
return event.message

147
python_pkg/wsg_grabber/scanner.py Normal file
View File

@ -0,0 +1,147 @@
"""Pure planning helpers for the background worker.
Nothing here performs IO, so every scheduling and retry decision is testable
from plain values. The worker in :mod:`python_pkg.wsg_grabber.downloader` is a
thin shell that calls these and then acts.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.constants import (
BACKOFF_BASE_S,
BACKOFF_CAP_S,
MIN_REQUEST_INTERVAL_S,
)
from python_pkg.wsg_grabber.models import ResumePlan, TaskKind
if TYPE_CHECKING:
from collections.abc import Mapping, Sequence
from python_pkg.wsg_grabber.models import ThreadRef
def choose_task(
pending_downloads: int,
stale_thread_count: int,
*,
catalog_fresh: bool,
) -> TaskKind:
"""Decide what the worker should do next.
Downloads win over scanning so the reviewer gets its first playable video
as soon as possible, which is the whole point of loading in the background.
Args:
pending_downloads: Files known but not yet fetched.
stale_thread_count: Threads whose contents may have changed.
catalog_fresh: Whether the thread list has been read at least once this
cycle.
Returns:
TaskKind: The next action.
"""
if pending_downloads > 0:
return TaskKind.DOWNLOAD
if stale_thread_count > 0 or not catalog_fresh:
return TaskKind.SCAN
return TaskKind.IDLE
def stale_threads(
known: Mapping[int, int],
live: Sequence[ThreadRef],
) -> list[ThreadRef]:
"""Return threads worth fetching.
A thread is worth fetching when it has never been seen, or when the board
reports a newer ``last_modified`` than the one already stored. On a steady
board this reduces a cycle to a single request.
Args:
known: Thread number mapped to the stored timestamp.
live: Threads currently advertised by the board.
Returns:
list[ThreadRef]: Threads to fetch, in the order given.
"""
return [
ref
for ref in live
if ref.thread_no not in known or ref.api_last_modified > known[ref.thread_no]
]
def next_delay(last_request: float, now: float) -> float:
"""Return how long to wait before the next API request.
The 4chan API rules ask for at most one request per second.
Args:
last_request: Monotonic timestamp of the previous request, or 0.
now: Current monotonic timestamp.
Returns:
float: Seconds to sleep, never negative.
"""
elapsed = now - last_request
remaining = MIN_REQUEST_INTERVAL_S - elapsed
return max(0.0, remaining)
def backoff_seconds(attempt: int, retry_after: float | None = None) -> float:
"""Return how long to pause after a rejected request.
An explicit ``Retry-After`` always wins; otherwise the delay doubles per
attempt up to a cap so a sustained block does not spin.
Args:
attempt: 1 for the first failure.
retry_after: Server-supplied delay in seconds, when present.
Returns:
float: Seconds to wait, capped.
"""
if retry_after is not None and retry_after > 0:
return min(retry_after, BACKOFF_CAP_S)
exponent = max(0, attempt - 1)
return min(BACKOFF_BASE_S * (2.0**exponent), BACKOFF_CAP_S)
def parse_retry_after(header: str | None) -> float | None:
"""Read a ``Retry-After`` header expressed in seconds.
The HTTP-date form is deliberately not handled; falling back to exponential
backoff is safe and avoids a clock-skew dependency.
Args:
header: Raw header value, if any.
Returns:
float | None: Seconds to wait, or None when absent or not numeric.
"""
if header is None:
return None
try:
value = float(header.strip())
except ValueError:
return None
return value if value >= 0 else None
def resume_plan(part_size: int, expected_size: int) -> ResumePlan:
"""Decide how to continue a partially downloaded file.
Args:
part_size: Bytes already on disk in the ``.part`` file.
expected_size: Size the API says the file should be.
Returns:
ResumePlan: Byte offset to request, and whether to bin what is there.
"""
if part_size <= 0:
return ResumePlan(offset=0, discard=False)
if 0 < expected_size <= part_size:
return ResumePlan(offset=0, discard=True)
return ResumePlan(offset=part_size, discard=False)

145
python_pkg/wsg_grabber/states.py Normal file
View File

@ -0,0 +1,145 @@
"""The lifecycle of a single file, as a pure transition function.
``GONE`` is the load-bearing state. A full first pass over /wsg/ takes hours, so
threads fall off page 10 and files 404 between being catalogued and being
fetched. Without a terminal sink those files would be retried on every future
run forever.
"""
from __future__ import annotations
from enum import Enum
from typing import Final
from python_pkg.wsg_grabber.constants import MAX_ATTEMPTS
# ``str, Enum`` not ``StrEnum``: mypy targets 3.10, where ``StrEnum`` is absent
# and would collapse to ``Any``, disabling checking on every member below.
class FileState(str, Enum):
"""Where a file is in its lifecycle."""
NEW = "new"
DOWNLOADING = "downloading"
READY = "ready"
FAILED = "failed"
CORRUPT = "corrupt"
KEPT = "kept"
PASSED = "passed"
GONE = "gone"
class FileEvent(str, Enum):
"""Something that happened to a file."""
CLAIMED = "claimed"
COMPLETED = "completed"
NOT_FOUND = "not_found"
DELETED_UPSTREAM = "deleted_upstream"
TRANSIENT_ERROR = "transient_error"
CHECKSUM_MISMATCH = "checksum_mismatch"
KEPT = "kept"
PASSED = "passed"
MISSING_LOCALLY = "missing_locally"
TERMINAL: Final[frozenset[FileState]] = frozenset(
{FileState.KEPT, FileState.PASSED, FileState.GONE},
)
CLAIMABLE: Final[frozenset[FileState]] = frozenset(
{FileState.NEW, FileState.FAILED, FileState.CORRUPT},
)
def is_terminal(state: FileState) -> bool:
"""Report whether *state* can never change again.
Args:
state: State to classify.
Returns:
bool: True when no further transition is possible.
"""
return state in TERMINAL
def next_state(current: FileState, event: FileEvent, attempts: int) -> FileState:
"""Return the state *current* moves to on *event*.
Args:
current: The file's present state.
event: What happened.
attempts: Download attempts made so far, including the one that just
failed. Retryable failures give up once this reaches
``MAX_ATTEMPTS``.
Returns:
FileState: The new state.
Raises:
ValueError: If the transition is not legal, which means a caller has a
bug rather than the board behaving oddly.
"""
if current in CLAIMABLE and event is FileEvent.CLAIMED:
return FileState.DOWNLOADING
if current in CLAIMABLE and event is FileEvent.DELETED_UPSTREAM:
return FileState.GONE
if current is FileState.DOWNLOADING:
return _from_downloading(event, attempts)
if current is FileState.READY:
return _from_ready(event)
msg = f"illegal transition: {current.value} + {event.value}"
raise ValueError(msg)
def _from_downloading(event: FileEvent, attempts: int) -> FileState:
"""Resolve a transition out of ``DOWNLOADING``.
Args:
event: What happened.
attempts: Download attempts made so far.
Returns:
FileState: The new state.
Raises:
ValueError: If *event* cannot follow ``DOWNLOADING``.
"""
if event is FileEvent.COMPLETED:
return FileState.READY
if event is FileEvent.NOT_FOUND:
return FileState.GONE
exhausted = attempts >= MAX_ATTEMPTS
if event is FileEvent.TRANSIENT_ERROR:
return FileState.GONE if exhausted else FileState.FAILED
if event is FileEvent.CHECKSUM_MISMATCH:
return FileState.GONE if exhausted else FileState.CORRUPT
msg = f"illegal transition: downloading + {event.value}"
raise ValueError(msg)
def _from_ready(event: FileEvent) -> FileState:
"""Resolve a transition out of ``READY``.
Upstream deletion is deliberately not handled here: once the bytes are on
disk it no longer matters that the post was removed.
Args:
event: What happened.
Returns:
FileState: The new state.
Raises:
ValueError: If *event* cannot follow ``READY``.
"""
if event is FileEvent.KEPT:
return FileState.KEPT
if event is FileEvent.PASSED:
return FileState.PASSED
if event is FileEvent.MISSING_LOCALLY:
return FileState.GONE
msg = f"illegal transition: ready + {event.value}"
raise ValueError(msg)

382
python_pkg/wsg_grabber/store.py Normal file
View File

@ -0,0 +1,382 @@
"""The download lifecycle of a file, as typed reads and writes.
One of three query modules, split to stay under this repo's 500-line-per-file
cap: this one owns how a file got downloaded, :mod:`store_threads` owns which
threads were visited, and :mod:`store_verdicts` owns what the user decided.
Between them they hold all the SQL outside :mod:`db`, so the rest of the
package deals in dataclasses. Every function takes an already-open connection
owned by the calling thread.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import (
RemoteFile,
ReviewItem,
)
from python_pkg.wsg_grabber.states import CLAIMABLE, FileState
if TYPE_CHECKING:
from collections.abc import Iterable, Mapping, Sequence
from pathlib import Path
import sqlite3
def _now() -> str:
"""Return the current UTC time as an ISO-8601 string.
Returns:
str: Timestamp suitable for the index's text date columns.
"""
return datetime.now(tz=timezone.utc).isoformat()
def local_name(md5_b64: str, tim: int, ext: str) -> str:
"""Return the on-disk filename for a file.
The poster-supplied name is untrusted text and is never used to build a
path; it is carried alongside purely for display.
Args:
md5_b64: The API's base64 MD5, used only to disambiguate.
tim: The CDN filename stem.
ext: Extension including the leading dot.
Returns:
str: A filesystem-safe name unique per file.
"""
suffix = md5_b64.rstrip("=")[-6:].replace("/", "_").replace("+", "-")
return f"{tim}-{suffix}{ext}"
def record_files(conn: sqlite3.Connection, files: Iterable[RemoteFile]) -> int:
"""Insert any files not already known.
Dedupe is structural: ``md5`` is the primary key, so a repost of identical
bytes in another thread collapses onto the existing row and never costs a
download.
Args:
conn: Open connection.
files: Candidate attachments parsed from a thread.
Returns:
int: How many rows were genuinely new.
"""
rows = [
(
item.md5,
item.tim,
item.ext,
item.orig_name,
item.fsize,
item.width,
item.height,
item.thread_no,
item.post_no,
FileState.NEW.value,
_now(),
)
for item in files
]
if not rows:
return 0
before = _scalar(conn, "SELECT COUNT(*) FROM files")
conn.executemany(
"""
INSERT INTO files (md5, tim, ext, orig_name, fsize, width, height,
thread_no, post_no, state, first_seen)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(md5) DO NOTHING
""",
rows,
)
return _scalar(conn, "SELECT COUNT(*) FROM files") - before
def known_md5s(conn: sqlite3.Connection) -> set[str]:
"""Return every md5 the index has ever seen.
Args:
conn: Open connection.
Returns:
set[str]: Base64 md5 values, including terminal ones, so a reviewed or
written-off file is never offered again.
"""
return {str(row[0]) for row in conn.execute("SELECT md5 FROM files")}
def claimable_values() -> list[str]:
"""Return the state values a download worker may pick up, in a stable order.
The queries below spell out one placeholder per member rather than building
the ``IN`` list at runtime, so the SQL stays a literal. ``test_store``
pins the member count so adding a state cannot silently desync the two.
Returns:
list[str]: Sorted state values.
"""
return [state.value for state in sorted(CLAIMABLE)]
def claim_next(conn: sqlite3.Connection) -> RemoteFile | None:
"""Atomically take the oldest downloadable file and mark it in flight.
Args:
conn: Open connection.
Returns:
RemoteFile | None: The claimed file, or None when nothing is pending.
"""
conn.execute("BEGIN IMMEDIATE")
try:
row = conn.execute(
"""
SELECT md5, tim, ext, orig_name, fsize, width, height,
thread_no, post_no
FROM files
WHERE state IN (?, ?, ?)
ORDER BY first_seen, md5
LIMIT 1
""",
claimable_values(),
).fetchone()
if row is None:
conn.execute("COMMIT")
return None
conn.execute(
"UPDATE files SET state = ?, attempts = attempts + 1 WHERE md5 = ?",
(FileState.DOWNLOADING.value, row["md5"]),
)
conn.execute("COMMIT")
except Exception:
conn.execute("ROLLBACK")
raise
return RemoteFile(
md5=row["md5"],
tim=row["tim"],
ext=row["ext"],
orig_name=row["orig_name"],
fsize=row["fsize"],
width=row["width"],
height=row["height"],
thread_no=row["thread_no"],
post_no=row["post_no"],
)
def state_of(conn: sqlite3.Connection, md5_b64: str) -> FileState | None:
"""Return a file's current state.
Args:
conn: Open connection.
md5_b64: File identity.
Returns:
FileState | None: The state, or None when the file is unknown.
"""
row = conn.execute(
"SELECT state FROM files WHERE md5 = ?",
(md5_b64,),
).fetchone()
return None if row is None else FileState(row["state"])
def attempts_for(conn: sqlite3.Connection, md5_b64: str) -> int:
"""Return how many download attempts a file has had.
Args:
conn: Open connection.
md5_b64: File identity.
Returns:
int: Attempt count, zero when the file is unknown.
"""
row = conn.execute(
"SELECT attempts FROM files WHERE md5 = ?",
(md5_b64,),
).fetchone()
return 0 if row is None else int(row["attempts"])
def set_state(
conn: sqlite3.Connection,
md5_b64: str,
state: FileState,
*,
error: str | None = None,
) -> None:
"""Move a file to *state*.
Args:
conn: Open connection.
md5_b64: File identity.
state: New state.
error: Optional diagnostic kept for the next run's benefit.
"""
conn.execute(
"UPDATE files SET state = ?, last_error = ? WHERE md5 = ?",
(state.value, error, md5_b64),
)
def mark_downloaded(conn: sqlite3.Connection, md5_b64: str, name: str) -> None:
"""Record that a file finished downloading and is ready to review.
Args:
conn: Open connection.
md5_b64: File identity.
name: Filename inside the incoming directory.
"""
conn.execute(
"""
UPDATE files
SET state = ?, local_name = ?, downloaded_at = ?, bytes_done = 0,
last_error = NULL
WHERE md5 = ?
""",
(FileState.READY.value, name, _now(), md5_b64),
)
def record_progress(conn: sqlite3.Connection, md5_b64: str, done: int) -> None:
"""Persist partial download progress so a restart can resume.
Args:
conn: Open connection.
md5_b64: File identity.
done: Bytes written to the ``.part`` file so far.
"""
conn.execute(
"UPDATE files SET bytes_done = ? WHERE md5 = ?",
(done, md5_b64),
)
def ready_items(conn: sqlite3.Connection, directory: Path) -> list[ReviewItem]:
"""Return downloaded files still awaiting a verdict, oldest first.
Args:
conn: Open connection.
directory: Where downloaded files live.
Returns:
list[ReviewItem]: Reviewable videos.
"""
rows = conn.execute(
"""
SELECT md5, local_name, orig_name, ext, fsize, width, height
FROM files
WHERE state = ? AND local_name IS NOT NULL
ORDER BY downloaded_at, md5
""",
(FileState.READY.value,),
).fetchall()
return [
ReviewItem(
md5=row["md5"],
path=directory / row["local_name"],
orig_name=f"{row['orig_name']}{row['ext']}",
fsize=row["fsize"],
width=row["width"],
height=row["height"],
)
for row in rows
]
def ready_item(source: RemoteFile, path: Path) -> ReviewItem:
"""Build the reviewer's view of a file that just finished downloading.
Args:
source: The catalogue record it came from.
path: Where the verified bytes now live.
Returns:
ReviewItem: Value handed to the UI thread.
"""
return ReviewItem(
md5=source.md5,
path=path,
orig_name=f"{source.orig_name}{source.ext}",
fsize=source.fsize,
width=source.width,
height=source.height,
)
def reset_in_flight(conn: sqlite3.Connection) -> int:
"""Return files interrupted mid-download to the pending pool.
A row left in ``downloading`` means the process died; the partial file on
disk is still valid and will be resumed.
Args:
conn: Open connection.
Returns:
int: How many rows were reset.
"""
cursor = conn.execute(
"UPDATE files SET state = ? WHERE state = ?",
(FileState.FAILED.value, FileState.DOWNLOADING.value),
)
return int(cursor.rowcount)
def counts(conn: sqlite3.Connection) -> dict[str, int]:
"""Return the number of files in each state.
Args:
conn: Open connection.
Returns:
dict[str, int]: State value mapped to row count, plus ``total``.
"""
result = {
str(row["state"]): int(row["n"])
for row in conn.execute(
"SELECT state, COUNT(*) AS n FROM files GROUP BY state",
)
}
result["total"] = _scalar(conn, "SELECT COUNT(*) FROM files")
return result
def pending_downloads(conn: sqlite3.Connection) -> int:
"""Return how many files are still waiting to be fetched.
Args:
conn: Open connection.
Returns:
int: Count of claimable rows.
"""
return _scalar(
conn,
"SELECT COUNT(*) FROM files WHERE state IN (?, ?, ?)",
claimable_values(),
)
def _scalar(
conn: sqlite3.Connection,
sql: str,
params: Sequence[object] | Mapping[str, object] | None = None,
) -> int:
"""Run a single-value query and return it as an int.
Args:
conn: Open connection.
sql: Query yielding exactly one column.
params: Optional bound parameters.
Returns:
int: The first column of the first row.
"""
cursor = conn.execute(sql) if params is None else conn.execute(sql, params)
return int(cursor.fetchone()[0])

157
python_pkg/wsg_grabber/store_threads.py Normal file
View File

@ -0,0 +1,157 @@
"""Bookkeeping for the ``threads`` table.
Split out of :mod:`python_pkg.wsg_grabber.store` to stay under this repo's
500-line-per-file cap. The seam is the natural one: this module owns which
threads have been visited and what their last ``Last-Modified`` was, which is
what lets an unchanged thread cost a 304 rather than a re-parse.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import sqlite3
from python_pkg.wsg_grabber.models import ThreadRef
def _now() -> str:
"""Return the current UTC time as an ISO-8601 string.
Returns:
str: Timestamp suitable for the index's text date columns.
"""
return datetime.now(tz=timezone.utc).isoformat()
def known_threads(conn: sqlite3.Connection) -> dict[int, int]:
"""Return each known thread's last-modified stamp.
Args:
conn: Open connection.
Returns:
dict[int, int]: Thread number mapped to its stored API timestamp.
"""
return {
int(row["thread_no"]): int(row["api_last_mod"])
for row in conn.execute("SELECT thread_no, api_last_mod FROM threads")
}
def http_last_modified(conn: sqlite3.Connection, thread_no: int) -> str | None:
"""Return the stored ``Last-Modified`` header for a thread.
Args:
conn: Open connection.
thread_no: Thread identifier.
Returns:
str | None: Header value to send back as ``If-Modified-Since``.
"""
row = conn.execute(
"SELECT http_last_mod FROM threads WHERE thread_no = ?",
(thread_no,),
).fetchone()
if row is None or row["http_last_mod"] is None:
return None
return str(row["http_last_mod"])
def record_thread(
conn: sqlite3.Connection,
ref: ThreadRef,
header: str | None,
status: str = "live",
) -> None:
"""Insert or update a thread's bookkeeping row.
Args:
conn: Open connection.
ref: Thread number and its API timestamp.
header: ``Last-Modified`` header from the thread fetch, if any.
status: ``live`` or ``gone``.
"""
conn.execute(
"""
INSERT INTO threads (thread_no, api_last_mod, http_last_mod,
last_checked, status)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(thread_no) DO UPDATE SET
api_last_mod = excluded.api_last_mod,
http_last_mod = COALESCE(excluded.http_last_mod, threads.http_last_mod),
last_checked = excluded.last_checked,
status = excluded.status
""",
(ref.thread_no, ref.api_last_modified, header, _now(), status),
)
def mark_thread_gone(conn: sqlite3.Connection, thread_no: int) -> None:
"""Flag a thread as no longer fetchable.
This inserts rather than merely updating: a thread can 404 the very first
time it is seen, and a plain UPDATE would touch no rows, leaving nothing to
record that it is dead. It would then be re-fetched on every future run.
Args:
conn: Open connection.
thread_no: Thread identifier.
"""
conn.execute(
"""
INSERT INTO threads (thread_no, api_last_mod, http_last_mod,
last_checked, status)
VALUES (?, 0, NULL, ?, 'gone')
ON CONFLICT(thread_no) DO UPDATE SET
status = 'gone',
last_checked = excluded.last_checked
""",
(thread_no, _now()),
)
def cached_last_modified(conn: sqlite3.Connection, url: str) -> str | None:
"""Return the stored ``Last-Modified`` for a board-level endpoint.
``threads.json`` and ``archive.json`` are not tied to any one thread, so
their stamps live here rather than in the threads table. Without this the
reviewer re-fetches both unconditionally on every idle cycle, forever.
Args:
conn: Open connection.
url: Absolute API URL.
Returns:
str | None: Header value to send back as ``If-Modified-Since``.
"""
row = conn.execute(
"SELECT last_modified FROM http_cache WHERE url = ?",
(url,),
).fetchone()
return None if row is None else str(row["last_modified"])
def record_last_modified(
conn: sqlite3.Connection,
url: str,
stamp: str | None,
) -> None:
"""Store the ``Last-Modified`` a board-level endpoint returned.
Args:
conn: Open connection.
url: Absolute API URL.
stamp: Header value; ignored when the server did not send one.
"""
if stamp is None:
return
conn.execute(
"""
INSERT INTO http_cache (url, last_modified) VALUES (?, ?)
ON CONFLICT(url) DO UPDATE SET last_modified = excluded.last_modified
""",
(url, stamp),
)

167
python_pkg/wsg_grabber/store_verdicts.py Normal file
View File

@ -0,0 +1,167 @@
"""The review trail: verdicts, and everything undo needs to reverse them.
Split out of :mod:`python_pkg.wsg_grabber.store` to stay under this repo's
500-line-per-file cap, along the seam that matters: this module owns what the
user decided, the other owns how a file got downloaded.
The trail lives in the index rather than in memory, which is what makes undo
unbounded and lets it survive quitting the reviewer.
"""
from __future__ import annotations
from datetime import datetime, timezone
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import ReviewedItem, ReviewItem, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from pathlib import Path
import sqlite3
def _now() -> str:
"""Return the current UTC time as an ISO-8601 string.
Returns:
str: Timestamp suitable for the index's text date columns.
"""
return datetime.now(tz=timezone.utc).isoformat()
def _scalar(conn: sqlite3.Connection, sql: str, params: list[str]) -> int:
"""Run a single-value query and return it as an int.
Args:
conn: Open connection.
sql: Query yielding exactly one column.
params: Bound parameters.
Returns:
int: The first column of the first row.
"""
return int(conn.execute(sql, params).fetchone()[0])
def record_verdict(
conn: sqlite3.Connection,
md5_b64: str,
state: FileState,
reviewed_name: str | None = None,
) -> None:
"""Store the user's decision about a file.
Args:
conn: Open connection.
md5_b64: File identity.
state: Either ``KEPT`` or ``PASSED``.
reviewed_name: Name the file now has in keep/ or trash/. Persisting it
is what lets the verdict be undone in a later session.
"""
conn.execute(
"""
UPDATE files
SET state = ?, reviewed_at = ?, reviewed_name = ?
WHERE md5 = ?
""",
(state.value, _now(), reviewed_name, md5_b64),
)
def reviewed_count(conn: sqlite3.Connection) -> int:
"""Return how many verdicts could still be taken back.
Args:
conn: Open connection.
Returns:
int: Count of reviewed files whose destination name is known.
"""
return _scalar(
conn,
"""
SELECT COUNT(*) FROM files
WHERE state IN (?, ?) AND reviewed_name IS NOT NULL
""",
[FileState.KEPT.value, FileState.PASSED.value],
)
def newest_verdict(
conn: sqlite3.Connection,
directory: Path,
) -> ReviewedItem | None:
"""Return the most recent verdict that can be undone.
Reading this from the index rather than from memory is what makes undo
survive quitting, and unbounded: every verdict ever recorded is a candidate,
newest first.
Args:
conn: Open connection.
directory: Where an undone file should be put back.
Returns:
ReviewedItem | None: The verdict to reverse, or None when there is none.
"""
row = conn.execute(
"""
SELECT md5, local_name, reviewed_name, orig_name, ext, fsize,
width, height, state
FROM files
WHERE state IN (?, ?) AND reviewed_name IS NOT NULL
ORDER BY reviewed_at DESC, md5 DESC
LIMIT 1
""",
(FileState.KEPT.value, FileState.PASSED.value),
).fetchone()
if row is None:
return None
return ReviewedItem(
item=ReviewItem(
md5=row["md5"],
path=directory / row["local_name"],
orig_name=f"{row['orig_name']}{row['ext']}",
fsize=row["fsize"],
width=row["width"],
height=row["height"],
),
choice=Verdict.KEEP if row["state"] == FileState.KEPT.value else Verdict.SKIP,
reviewed_name=row["reviewed_name"],
)
def forget_verdict(conn: sqlite3.Connection, md5_b64: str) -> None:
"""Make a verdict un-undoable without changing it.
Used when the file is no longer where the verdict left it, so the same
entry does not keep surfacing as the next thing to undo.
Args:
conn: Open connection.
md5_b64: File identity.
"""
conn.execute(
"UPDATE files SET reviewed_name = NULL WHERE md5 = ?",
(md5_b64,),
)
def restore_for_review(conn: sqlite3.Connection, md5_b64: str, name: str) -> None:
"""Undo a verdict, returning the file to the review queue.
Args:
conn: Open connection.
md5_b64: File identity.
name: Filename it has again inside the incoming directory.
"""
conn.execute(
"""
UPDATE files
SET state = ?, local_name = ?, reviewed_at = NULL,
reviewed_name = NULL
WHERE md5 = ?
""",
(FileState.READY.value, name, md5_b64),
)

0
python_pkg/wsg_grabber/tests/__init__.py Normal file
View File

47
python_pkg/wsg_grabber/tests/conftest.py Normal file
View File

@ -0,0 +1,47 @@
"""Test isolation for the /wsg/ grabber.
This package writes an index and tens of gigabytes of video into
``~/.local/share/wsg_grabber``. A test that escapes into the real directory
would corrupt live user data, so redirection is autouse and belt-and-braces:
1. ``Path.home`` is patched on the *class*, so it holds no matter whether a
module wrote ``from pathlib import Path`` or ``import pathlib``.
2. ``HOME`` and ``XDG_DATA_HOME`` are redirected, catching anything that goes
through ``os.environ`` or ``expanduser`` instead.
3. Every module reaches storage only through :mod:`python_pkg.wsg_grabber.paths`
functions, which resolve on each call rather than at import.
``tests/test_paths.py`` adds a static check that no module captures a home
directory at import time, so layer 3 stays true as the package grows.
"""
from __future__ import annotations
from pathlib import Path
import pytest
@pytest.fixture(autouse=True)
def isolate_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Point every storage location at a throwaway directory.
No teardown is needed: ``monkeypatch`` unwinds both the environment and the
``Path.home`` patch when the test ends.
Args:
tmp_path: Per-test temporary directory supplied by pytest.
monkeypatch: Patching helper, unwound automatically after the test.
Returns:
Path: The fake home directory backing this test.
"""
home = tmp_path / "home"
data_home = home / ".local" / "share"
data_home.mkdir(parents=True)
monkeypatch.setenv("HOME", str(home))
monkeypatch.setenv("XDG_DATA_HOME", str(data_home))
monkeypatch.setattr(Path, "home", classmethod(lambda _cls: home))
return home

429
python_pkg/wsg_grabber/tests/test_app.py Normal file
View File

@ -0,0 +1,429 @@
"""Tests for the composition root and verdict application."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import (
app,
db,
downloader,
paths,
review,
store,
store_verdicts,
)
from python_pkg.wsg_grabber.models import RemoteFile, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Callable, Iterator
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
@pytest.fixture
def session() -> Iterator[app.Session]:
"""Build a session over a sandboxed index, with no worker running.
Yields:
app.Session: Session whose connection is closed afterwards.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
try:
yield built
finally:
conn.close()
def _downloaded(session: app.Session, name: str = "clip") -> None:
"""Put one downloaded, unreviewed file into the index and on disk.
Args:
session: The session to populate.
name: Local filename stem.
"""
store.record_files(
session.conn,
[
RemoteFile(
md5=_MD5,
tim=1,
ext=".webm",
orig_name=name,
fsize=5,
width=4,
height=3,
thread_no=7,
post_no=1,
),
],
)
local = f"{name}.webm"
(paths.incoming_dir() / local).write_bytes(b"video")
store.mark_downloaded(session.conn, _MD5, local)
def test_keep_moves_the_file_and_records_it(session: app.Session) -> None:
_downloaded(session)
state = app.initial_state(session)
result = session.commit(state, Verdict.KEEP)
assert (paths.keep_dir() / "clip.webm").exists()
assert not (paths.incoming_dir() / "clip.webm").exists()
assert store.state_of(session.conn, _MD5) is FileState.KEPT
assert result.kept == 1
def test_pass_moves_to_trash_and_never_deletes(session: app.Session) -> None:
"""The whole point of trash/: a pass is reversible until the user says so."""
_downloaded(session)
state = app.initial_state(session)
result = session.commit(state, Verdict.SKIP)
assert (paths.trash_dir() / "clip.webm").exists()
assert (paths.trash_dir() / "clip.webm").read_bytes() == b"video"
assert store.state_of(session.conn, _MD5) is FileState.PASSED
assert result.passed == 1
def test_a_name_collision_in_the_destination_is_resolved(
session: app.Session,
) -> None:
_downloaded(session)
(paths.keep_dir() / "clip.webm").write_bytes(b"older")
session.commit(app.initial_state(session), Verdict.KEEP)
assert (paths.keep_dir() / "clip.webm").read_bytes() == b"older"
assert (paths.keep_dir() / "clip-2.webm").read_bytes() == b"video"
def test_a_verdict_on_a_vanished_file_writes_it_off(session: app.Session) -> None:
_downloaded(session)
state = app.initial_state(session)
(paths.incoming_dir() / "clip.webm").unlink()
result = session.commit(state, Verdict.KEEP)
assert store.state_of(session.conn, _MD5) is FileState.GONE
assert result.kept == 0
assert result.current is None
def test_a_verdict_with_nothing_showing_is_ignored(session: app.Session) -> None:
state = review.initial_state(0, [])
assert session.commit(state, Verdict.KEEP) is state
assert session.missing(state) is state
def test_initial_state_resumes_a_previous_run(session: app.Session) -> None:
_downloaded(session)
state = app.initial_state(session)
assert state.current is not None
assert state.current.md5 == _MD5
assert state.indexed == 1
def test_shutdown_stops_the_worker_and_closes_the_index() -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
worker = MagicMock()
worker.is_alive.return_value = False
built = app.Session(
conn=conn,
events=queue.SimpleQueue(),
stop=threading.Event(),
worker=worker,
)
built.shutdown()
assert built.stop.is_set()
worker.join.assert_called_once()
assert built.worker is None
def test_shutdown_without_a_worker() -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
built.shutdown()
assert built.stop.is_set()
def test_open_session_starts_a_worker_owning_its_own_connection() -> None:
"""sqlite forbids sharing a connection across threads, so the factory must
build one on the worker's side of the boundary."""
captured: list[downloader.WorkerDeps] = []
def fake_start(
build: Callable[[], downloader.WorkerDeps],
_on_failure: Callable[[str], None] = lambda _m: None,
) -> MagicMock:
deps = build()
captured.append(deps)
deps.conn.close()
deps.session.close()
return MagicMock()
with patch.object(downloader, "start", side_effect=fake_start):
built = app.open_session(include_archive=False)
try:
assert captured
assert captured[0].incoming == paths.incoming_dir()
assert not captured[0].include_archive
assert paths.incoming_dir().is_dir()
assert paths.trash_dir().is_dir()
finally:
built.shutdown()
def test_open_session_recovers_interrupted_downloads() -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
store.record_files(
conn,
[
RemoteFile(
md5=_MD5,
tim=1,
ext=".webm",
orig_name="x",
fsize=1,
width=1,
height=1,
thread_no=1,
post_no=1,
),
],
)
store.claim_next(conn)
conn.close()
with patch.object(downloader, "start", return_value=MagicMock()):
built = app.open_session()
try:
assert store.pending_downloads(built.conn) == 1
finally:
built.shutdown()
def test_undo_puts_a_kept_file_back_in_the_queue(session: app.Session) -> None:
"""The misclick this exists for: kept something you meant to pass."""
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.KEEP)
assert (paths.keep_dir() / "clip.webm").exists()
state = session.undo(state)
assert not (paths.keep_dir() / "clip.webm").exists()
assert (paths.incoming_dir() / "clip.webm").exists()
assert store.state_of(session.conn, _MD5) is FileState.READY
assert state.kept == 0
assert state.current is not None
assert state.current.md5 == _MD5
def test_undo_puts_a_passed_file_back(session: app.Session) -> None:
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.SKIP)
assert (paths.trash_dir() / "clip.webm").exists()
state = session.undo(state)
assert (paths.incoming_dir() / "clip.webm").exists()
assert not (paths.trash_dir() / "clip.webm").exists()
assert state.passed == 0
def test_undo_survives_a_collision_rename(session: app.Session) -> None:
"""A verdict may have renamed the file; undo must follow the real name."""
_downloaded(session)
(paths.trash_dir() / "clip.webm").write_bytes(b"older")
state = session.commit(app.initial_state(session), Verdict.SKIP)
assert (paths.trash_dir() / "clip-2.webm").exists()
state = session.undo(state)
assert (paths.incoming_dir() / "clip.webm").read_bytes() == b"video"
assert (paths.trash_dir() / "clip.webm").read_bytes() == b"older"
assert not (paths.trash_dir() / "clip-2.webm").exists()
def test_undo_steps_back_through_several_verdicts(session: app.Session) -> None:
for index in range(3):
store.record_files(
session.conn,
[
RemoteFile(
md5=f"M{index}",
tim=index,
ext=".webm",
orig_name=f"v{index}",
fsize=5,
width=1,
height=1,
thread_no=1,
post_no=index,
),
],
)
(paths.incoming_dir() / f"v{index}.webm").write_bytes(b"video")
store.mark_downloaded(session.conn, f"M{index}", f"v{index}.webm")
state = app.initial_state(session)
for _ in range(3):
state = session.commit(state, Verdict.SKIP)
assert state.passed == 3
for expected in ("M2", "M1", "M0"):
state = session.undo(state)
assert state.current is not None
assert state.current.md5 == expected
assert state.passed == 0
assert not review.can_undo(state)
def test_undo_with_nothing_to_undo_is_a_no_op(session: app.Session) -> None:
state = app.initial_state(session)
assert session.undo(state) is state
def test_undo_gives_up_gracefully_when_the_file_was_removed(
session: app.Session,
) -> None:
"""You may have emptied trash/ by hand; undo must not wedge on it."""
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.SKIP)
(paths.trash_dir() / "clip.webm").unlink()
state = session.undo(state)
assert not review.can_undo(state)
assert store.state_of(session.conn, _MD5) is FileState.PASSED
def test_undo_survives_quitting_and_reopening(session: app.Session) -> None:
"""The whole point of persisting the trail: undo is not lost on exit."""
_downloaded(session)
session.commit(app.initial_state(session), Verdict.SKIP)
assert (paths.trash_dir() / "clip.webm").exists()
session.conn.close()
# a brand new session, as if the reviewer had been restarted
reopened = app.Session(
conn=db.open_index(paths.db_path()),
events=queue.SimpleQueue(),
stop=threading.Event(),
)
try:
fresh = app.initial_state(reopened)
assert fresh.undoable == 1
restored = reopened.undo(fresh)
assert (paths.incoming_dir() / "clip.webm").exists()
assert store.state_of(reopened.conn, _MD5) is FileState.READY
assert restored.current is not None
assert restored.current.md5 == _MD5
assert restored.undoable == 0
finally:
reopened.conn.close()
def test_the_undo_trail_is_unbounded(session: app.Session) -> None:
"""Any number of last verdicts, not a fixed window."""
total = 25
for index in range(total):
store.record_files(
session.conn,
[
RemoteFile(
md5=f"U{index:02d}",
tim=index,
ext=".webm",
orig_name=f"u{index:02d}",
fsize=5,
width=1,
height=1,
thread_no=1,
post_no=index,
),
],
)
(paths.incoming_dir() / f"u{index:02d}.webm").write_bytes(b"video")
store.mark_downloaded(session.conn, f"U{index:02d}", f"u{index:02d}.webm")
state = app.initial_state(session)
for _ in range(total):
state = session.commit(state, Verdict.SKIP)
assert state.passed == total
assert store_verdicts.reviewed_count(session.conn) == total
for index in reversed(range(total)):
state = session.undo(state)
assert state.current is not None
assert state.current.md5 == f"U{index:02d}"
assert state.passed == 0
assert store_verdicts.reviewed_count(session.conn) == 0
assert len(list(paths.trash_dir().iterdir())) == 0
def test_undo_never_overwrites_a_file_sitting_in_incoming(
session: app.Session,
) -> None:
"""A verdict must never be reversed on top of another video.
Path.replace clobbers silently, so undo picks a collision-free name the
same way commit does.
"""
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.KEEP)
# something else now occupies the name the file used to have
(paths.incoming_dir() / "clip.webm").write_bytes(b"DO NOT DESTROY ME")
session.undo(state)
assert (paths.incoming_dir() / "clip.webm").read_bytes() == b"DO NOT DESTROY ME"
assert (paths.incoming_dir() / "clip-2.webm").read_bytes() == b"video"
row = session.conn.execute(
"SELECT local_name FROM files WHERE md5 = ?",
(_MD5,),
).fetchone()
assert row["local_name"] == "clip-2.webm"
def test_a_missing_file_never_clobbers_an_existing_verdict(
session: app.Session,
) -> None:
"""Two reviewers on one index must not corrupt each other's undo trail."""
_downloaded(session)
state = app.initial_state(session)
session.commit(state, Verdict.KEEP)
assert store_verdicts.reviewed_count(session.conn) == 1
# the same item, still queued in a second reviewer, whose file is now gone
session.missing(state)
assert store.state_of(session.conn, _MD5) is FileState.KEPT
assert store_verdicts.reviewed_count(session.conn) == 1
def test_open_session_closes_the_index_if_the_worker_cannot_start() -> None:
with (
patch.object(downloader, "start", side_effect=RuntimeError("no thread")),
pytest.raises(RuntimeError),
):
app.open_session()
# a second open must succeed, which it cannot if the first leaked a handle
with patch.object(downloader, "start", return_value=MagicMock()):
again = app.open_session()
again.shutdown()
def test_a_worker_that_outlives_the_join_is_not_silently_forgotten(
session: app.Session,
) -> None:
worker = MagicMock()
worker.is_alive.return_value = True
session.worker = worker
session.shutdown()
assert session.worker is worker # handle kept so the failure is visible

171
python_pkg/wsg_grabber/tests/test_catalog.py Normal file
View File

@ -0,0 +1,171 @@
"""Tests for parsing the /wsg/ JSON payloads.
Fixtures mirror the shape of real responses captured from the live API,
including the fields that are simply absent on text-only posts.
"""
from __future__ import annotations
from python_pkg.wsg_grabber import catalog
_WEBM = "Nz9OEKdMuMZEdYE6eLTKmA=="
_MP4 = "b9EO3Kaq7PHBgc6kWmHgew=="
def _post(**overrides: object) -> dict[str, object]:
"""Build a post carrying a webm attachment.
Args:
**overrides: Fields to replace or, with a None value, remove.
Returns:
dict[str, object]: Post payload.
"""
post: dict[str, object] = {
"no": 6195878,
"resto": 0,
"tim": 1784059525831380,
"ext": ".webm",
"filename": "kino moment",
"fsize": 1008689,
"md5": _WEBM,
"w": 480,
"h": 360,
}
post.update(overrides)
return {key: value for key, value in post.items() if value is not None}
def test_media_url_uses_tim_not_the_poster_filename() -> None:
assert (
catalog.media_url(1784059525831380, ".webm")
== "https://i.4cdn.org/wsg/1784059525831380.webm"
)
def test_parse_thread_list_flattens_every_page() -> None:
payload = [
{"page": 1, "threads": [{"no": 1, "last_modified": 100, "replies": 2}]},
{"page": 2, "threads": [{"no": 2, "last_modified": 200, "replies": 5}]},
]
refs = catalog.parse_thread_list(payload)
assert [(ref.thread_no, ref.api_last_modified) for ref in refs] == [
(1, 100),
(2, 200),
]
def test_parse_thread_list_survives_malformed_payloads() -> None:
assert catalog.parse_thread_list("not a list") == []
assert catalog.parse_thread_list([{"threads": "nope"}]) == []
assert catalog.parse_thread_list([{"threads": [{"no": None}]}]) == []
assert catalog.parse_thread_list([{"threads": [{"no": True}]}]) == []
refs = catalog.parse_thread_list([{"threads": [{"no": 5}]}])
assert refs[0].api_last_modified == 0
def test_parse_archive_reads_bare_thread_numbers() -> None:
refs = catalog.parse_archive([11, 22, "junk", None])
assert [ref.thread_no for ref in refs] == [11, 22]
assert refs[0].api_last_modified == 0
def test_parse_thread_keeps_all_three_video_extensions() -> None:
payload = {
"posts": [
_post(ext=".webm", md5=_WEBM, no=1),
_post(ext=".mp4", md5=_MP4, no=2),
_post(ext=".gif", md5="c" * 22 + "==", no=3),
],
}
files = catalog.parse_thread(99, payload)
assert [item.ext for item in files] == [".webm", ".mp4", ".gif"]
assert all(item.thread_no == 99 for item in files)
def test_parse_thread_skips_images_and_text_posts() -> None:
payload = {
"posts": [
{"no": 1, "com": "text only"},
_post(ext=".jpg", no=2),
_post(ext=".png", no=3),
_post(no=4),
],
}
files = catalog.parse_thread(99, payload)
assert len(files) == 1
assert files[0].post_no == 4
def test_parse_thread_skips_deleted_attachments() -> None:
payload = {"posts": [_post(filedeleted=1)]}
assert catalog.parse_thread(99, payload) == []
def test_parse_thread_rejects_malformed_file_fields() -> None:
cases = [
_post(tim=None),
_post(no=None),
_post(md5=None),
_post(md5="too-short"),
_post(ext=12345),
]
for post in cases:
assert catalog.parse_thread(99, {"posts": [post]}) == []
def test_parse_thread_defaults_missing_optional_fields() -> None:
payload = {"posts": [_post(filename=None, fsize=None, w=None, h=None)]}
item = catalog.parse_thread(99, payload)[0]
assert item.orig_name == str(item.tim)
assert (item.fsize, item.width, item.height) == (0, 0, 0)
def test_parse_thread_survives_a_payload_that_is_not_a_dict() -> None:
assert catalog.parse_thread(1, "nonsense") == []
assert catalog.parse_thread(1, {"posts": "nonsense"}) == []
assert catalog.parse_thread(1, {"posts": ["nonsense"]}) == []
def test_deleted_md5s_collects_removed_attachments() -> None:
payload = {
"posts": [
_post(no=1, filedeleted=1),
_post(no=2, md5=_MP4),
_post(no=3, filedeleted=1, md5=None),
],
}
assert catalog.deleted_md5s(payload) == {_WEBM}
def test_new_files_drops_anything_already_indexed() -> None:
parsed = catalog.parse_thread(
1,
{"posts": [_post(no=1), _post(no=2, md5=_MP4)]},
)
fresh = catalog.new_files(parsed, {_WEBM})
assert [item.md5 for item in fresh] == [_MP4]
def test_new_files_dedupes_within_one_batch() -> None:
"""A file quoted twice in the same thread must only be queued once."""
parsed = catalog.parse_thread(
1,
{"posts": [_post(no=1), _post(no=2, tim=999)]},
)
assert len(parsed) == 2
assert len(catalog.new_files(parsed, set())) == 1
def test_a_hostile_md5_is_rejected_before_it_can_reach_a_filename() -> None:
"""The digest tail ends up in a filename; length alone is not enough."""
for hostile in ("../.." + "x" * 18, "a" * 23 + "\x00", "/" * 24, "a" * 23 + "\n"):
assert catalog.parse_thread(1, {"posts": [_post(md5=hostile)]}) == []
assert catalog.parse_thread(1, {"posts": [_post(md5=_WEBM)]}) != []
def test_an_out_of_range_integer_is_rejected() -> None:
"""Wider than sqlite's signed 64-bit range kills the worker on insert."""
assert catalog.parse_thread(1, {"posts": [_post(tim=2**63)]}) == []
assert catalog.parse_thread(1, {"posts": [_post(no=-(2**63) - 1)]}) == []
assert catalog.parse_thread(1, {"posts": [_post(tim=2**63 - 1)]}) != []

241
python_pkg/wsg_grabber/tests/test_cli.py Normal file
View File

@ -0,0 +1,241 @@
"""Tests for the command-line interface."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import app, cli, db, logs, paths, store
from python_pkg.wsg_grabber.models import RemoteFile
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
@pytest.fixture
def fake_session() -> Iterator[app.Session]:
"""Build a session over a sandboxed index with no worker.
Yields:
app.Session: Session closed afterwards.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
try:
yield built
finally:
conn.close()
def test_parser_defaults_to_the_reviewer() -> None:
assert cli.build_parser().parse_args([]).command is None
assert cli.build_parser().parse_args(["review"]).command == "review"
def test_parser_exposes_the_subcommands() -> None:
assert cli.build_parser().parse_args(["stats"]).command == "stats"
parsed = cli.build_parser().parse_args(["scrape", "--seconds", "5"])
assert parsed.command == "scrape"
assert parsed.seconds == pytest.approx(5.0)
def test_help_mentions_that_nothing_is_deleted() -> None:
text = cli.build_parser().format_help()
assert "Nothing is ever deleted" in text
assert "trash" in text
def test_stats_reports_every_state(capsys: pytest.CaptureFixture[str]) -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
store.record_files(
conn,
[
RemoteFile(
md5="A" * 24,
tim=1,
ext=".webm",
orig_name="x",
fsize=1,
width=1,
height=1,
thread_no=1,
post_no=1,
),
],
)
conn.close()
assert cli.main(["stats"]) == 0
out = capsys.readouterr().out
assert "known files 1" in out
assert "new 1" in out
assert "never emptied automatically" in out
def test_scrape_runs_and_shuts_down(
fake_session: app.Session,
capsys: pytest.CaptureFixture[str],
) -> None:
ticks = {"n": 0}
def fake_sleep(_seconds: float) -> None:
ticks["n"] += 1
with (
patch.object(app, "open_session", return_value=fake_session),
patch.object(fake_session, "shutdown") as stopped,
patch(
"python_pkg.wsg_grabber.cli.time.monotonic",
side_effect=[0.0, 1.0, 99.0, 99.0],
),
):
assert cli.scrape(5.0, sleeper=fake_sleep) == 0
assert ticks["n"] >= 1
stopped.assert_called_once()
assert "scraping /wsg/" in capsys.readouterr().out
def test_scrape_reports_progress_periodically(
fake_session: app.Session,
capsys: pytest.CaptureFixture[str],
) -> None:
with (
patch.object(app, "open_session", return_value=fake_session),
patch.object(fake_session, "shutdown"),
patch(
"python_pkg.wsg_grabber.cli.time.monotonic",
side_effect=[0.0, 999.0, 999.0, 999.0],
),
):
cli.scrape(1.0, sleeper=lambda _s: None)
assert "known 0" in capsys.readouterr().out
def test_scrape_handles_an_interrupt(
fake_session: app.Session,
capsys: pytest.CaptureFixture[str],
) -> None:
def interrupt(_seconds: float) -> None:
raise KeyboardInterrupt
with (
patch.object(app, "open_session", return_value=fake_session),
patch.object(fake_session, "shutdown") as stopped,
):
assert cli.scrape(0.0, sleeper=interrupt) == 0
stopped.assert_called_once()
assert "stopping" in capsys.readouterr().out
def test_main_dispatches_to_the_reviewer() -> None:
with patch.object(cli, "open_reviewer", return_value=0) as opened:
assert cli.main([]) == 0
opened.assert_called_once()
def test_main_dispatches_to_scrape() -> None:
with patch.object(cli, "scrape", return_value=0) as scraped:
assert cli.main(["scrape", "--seconds", "2"]) == 0
assert scraped.call_args.args[0] == pytest.approx(2.0)
def test_open_reviewer_wires_the_window_to_the_player(
fake_session: app.Session,
) -> None:
window = MagicMock()
window.video_wid.return_value = 4321
with (
patch.object(app, "open_session", return_value=fake_session),
patch(
"python_pkg.wsg_grabber.cli.ui.ReviewWindow",
return_value=window,
) as built,
patch("python_pkg.wsg_grabber.cli.player.MpvPlayer") as made_player,
):
assert cli.open_reviewer() == 0
made_player.assert_called_once_with(4321, paths.ipc_socket_path())
window.attach_player.assert_called_once()
window.run.assert_called_once()
callbacks = built.call_args.args[2]
assert callbacks.commit == fake_session.commit
assert callbacks.shutdown == fake_session.shutdown
def test_log_level_flags_are_exposed() -> None:
parsed = cli.build_parser().parse_args(["--log-level", "debug", "--echo-log"])
assert parsed.log_level == "debug"
assert parsed.echo_log
assert cli.build_parser().parse_args([]).log_level == "info"
def test_running_a_command_writes_a_session_log() -> None:
assert cli.main(["stats"]) == 0
written = sorted(logs.logs_dir().glob("session-*.jsonl"))
assert written
assert "session.start" in written[-1].read_text(encoding="utf-8")
def test_logs_command_reports_where_logs_live(
capsys: pytest.CaptureFixture[str],
) -> None:
cli.main(["stats"])
capsys.readouterr()
assert cli.main(["logs"]) == 0
out = capsys.readouterr().out
assert str(logs.logs_dir()) in out
assert "newest session" in out
assert "jq" in out
def test_logs_command_before_anything_has_run(
capsys: pytest.CaptureFixture[str],
) -> None:
logs.logs_dir().mkdir(parents=True, exist_ok=True)
for stale in logs.logs_dir().glob("*.jsonl"):
stale.unlink()
assert cli.show_logs() == 0
assert "no session logs yet" in capsys.readouterr().out
def test_summarise_counts_events_and_skips_junk(tmp_path: Path) -> None:
sample = tmp_path / "s.jsonl"
sample.write_text(
'{"event":"a","level":"info"}\n'
"not json at all\n"
'{"event":"a","level":"error"}\n'
'{"event":"b","level":"info"}\n',
encoding="utf-8",
)
text = cli.summarise(sample)
assert "error=1" in text
assert "info=2" in text
assert "a" in text
def test_a_failed_player_start_does_not_strand_the_worker(
fake_session: app.Session,
) -> None:
"""The worker is non-daemon; without cleanup the process never exits."""
window = MagicMock()
window.video_wid.return_value = 1
with (
patch.object(app, "open_session", return_value=fake_session),
patch("python_pkg.wsg_grabber.cli.ui.ReviewWindow", return_value=window),
patch(
"python_pkg.wsg_grabber.cli.player.MpvPlayer",
side_effect=TimeoutError("mpv never came up"),
),
patch.object(fake_session, "shutdown") as stopped,
pytest.raises(TimeoutError),
):
cli.open_reviewer()
stopped.assert_called_once()

130
python_pkg/wsg_grabber/tests/test_db.py Normal file
View File

@ -0,0 +1,130 @@
"""Tests for connection setup and schema migration."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import db, paths
from python_pkg.wsg_grabber.constants import SCHEMA_VERSION
if TYPE_CHECKING:
from pathlib import Path
import sqlite3
def _fresh(tmp_path: Path) -> sqlite3.Connection:
"""Open a brand new index inside the sandbox.
Args:
tmp_path: Per-test temporary directory.
Returns:
sqlite3.Connection: Migrated connection.
"""
return db.open_index(tmp_path / "index.db")
def test_connect_enables_wal_and_row_access(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
mode = conn.execute("PRAGMA journal_mode").fetchone()[0]
assert str(mode).lower() == "wal"
conn.execute("CREATE TABLE t (a INTEGER)")
conn.execute("INSERT INTO t VALUES (1)")
assert conn.execute("SELECT a FROM t").fetchone()["a"] == 1
finally:
conn.close()
def test_ensure_schema_creates_tables_and_stamps_the_version(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
assert db.ensure_schema(conn) == SCHEMA_VERSION
names = {
row[0]
for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
assert {"files", "threads"} <= names
assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == SCHEMA_VERSION
finally:
conn.close()
def test_ensure_schema_is_idempotent(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
db.ensure_schema(conn)
assert db.ensure_schema(conn) == SCHEMA_VERSION
finally:
conn.close()
def test_a_newer_schema_is_refused_rather_than_downgraded(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
conn.execute(f"PRAGMA user_version={SCHEMA_VERSION + 1}")
with pytest.raises(db.SchemaTooNewError, match="upgrade the tool"):
db.ensure_schema(conn)
finally:
conn.close()
def test_open_index_returns_a_migrated_connection(tmp_path: Path) -> None:
conn = _fresh(tmp_path)
try:
assert conn.execute("SELECT COUNT(*) FROM files").fetchone()[0] == 0
finally:
conn.close()
def test_index_lands_inside_the_sandbox(tmp_path: Path) -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
try:
assert str(paths.db_path()).startswith(str(tmp_path))
assert paths.db_path().exists()
finally:
conn.close()
def test_a_version_1_index_is_migrated_in_place(tmp_path: Path) -> None:
"""An index written before undo existed must keep its rows and gain the column."""
path = tmp_path / "old.db"
conn = db.connect(path)
try:
conn.execute(
"""
CREATE TABLE files (
md5 TEXT PRIMARY KEY, tim INTEGER NOT NULL, ext TEXT NOT NULL,
orig_name TEXT NOT NULL, fsize INTEGER NOT NULL,
width INTEGER NOT NULL, height INTEGER NOT NULL,
thread_no INTEGER NOT NULL, post_no INTEGER NOT NULL,
state TEXT NOT NULL, attempts INTEGER NOT NULL DEFAULT 0,
bytes_done INTEGER NOT NULL DEFAULT 0, last_error TEXT,
local_name TEXT, first_seen TEXT NOT NULL,
downloaded_at TEXT, reviewed_at TEXT
)
""",
)
conn.execute(
"INSERT INTO files (md5,tim,ext,orig_name,fsize,width,height,"
"thread_no,post_no,state,first_seen) "
"VALUES ('OLD',1,'.webm','x',1,1,1,1,1,'kept','then')",
)
conn.execute("PRAGMA user_version=1")
finally:
conn.close()
conn = db.open_index(path)
try:
assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == SCHEMA_VERSION
columns = {row["name"] for row in conn.execute("PRAGMA table_info(files)")}
assert "reviewed_name" in columns
# the existing row survives, simply without an undo trail
row = conn.execute("SELECT state, reviewed_name FROM files").fetchone()
assert row["state"] == "kept"
assert row["reviewed_name"] is None
finally:
conn.close()

341
python_pkg/wsg_grabber/tests/test_downloader.py Normal file
View File

@ -0,0 +1,341 @@
"""Tests for the background worker.
Every case drives ``run_once`` directly. No real thread runs and no socket is
opened, so the suite stays deterministic and cannot leak either.
"""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net, store, store_threads
from python_pkg.wsg_grabber.models import (
Downloaded,
DownloadEvent,
DownloadResult,
FileReady,
Indexed,
JsonResponse,
Outcome,
RemoteFile,
TaskKind,
ThreadRef,
)
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_first_step_reads_the_thread_list(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
payload = [{"threads": [{"no": 7, "last_modified": 100}]}]
with patch.object(net, "get_json", return_value=_json(payload)):
assert worker.run_once() is TaskKind.SCAN
def test_scan_then_fetch_records_files(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
thread = {
"posts": [
{
"no": 1,
"tim": 1,
"ext": ".webm",
"md5": _MD5,
"fsize": 11,
"w": 4,
"h": 3,
"filename": "clip",
},
],
}
with patch.object(
net,
"get_json",
side_effect=[_json(listing), _json(thread, last_modified="stamp")],
):
worker.run_once()
worker.run_once()
assert store.known_md5s(deps.conn) == {_MD5}
assert store_threads.known_threads(deps.conn) == {7: 100}
assert store_threads.http_last_modified(deps.conn, 7) == "stamp"
assert any(isinstance(event, Indexed) for event in _drain(deps.events))
def test_an_unchanged_thread_costs_only_a_304(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store_threads.record_thread(deps.conn, ThreadRef(7, 50), "old-stamp")
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
with patch.object(
net,
"get_json",
side_effect=[_json(listing), _json(None, not_modified=True)],
) as fake:
worker.run_once()
worker.run_once()
assert fake.call_args.args[2] == "old-stamp"
assert store_threads.known_threads(deps.conn) == {7: 100}
assert store.known_md5s(deps.conn) == set()
def test_a_vanished_thread_is_marked_gone(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
with patch.object(
net,
"get_json",
side_effect=[_json(listing), _json(None, not_found=True)],
):
worker.run_once()
worker.run_once()
row = deps.conn.execute("SELECT status FROM threads WHERE thread_no=7").fetchone()
assert row["status"] == "gone"
def test_archive_threads_are_included_when_enabled(
deps: downloader.WorkerDeps,
) -> None:
deps.include_archive = True
worker = downloader.Worker(deps)
with patch.object(
net,
"get_json",
side_effect=[
_json([{"threads": [{"no": 7, "last_modified": 1}]}]),
_json([88]),
],
):
worker.run_once()
with patch.object(net, "get_json", side_effect=[_json({"posts": []})]) as fake:
worker.run_once()
assert "/thread/" in fake.call_args.args[1]
def test_a_board_without_an_archive_is_tolerated(
deps: downloader.WorkerDeps,
) -> None:
deps.include_archive = True
worker = downloader.Worker(deps)
with patch.object(
net,
"get_json",
side_effect=[_json([]), _json(None, not_found=True)],
):
assert worker.run_once() is TaskKind.SCAN
def test_a_completed_download_is_renamed_and_announced(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
def fake_download(_session: object, transfer: net.Transfer) -> DownloadResult:
transfer.part_path.write_bytes(_BODY)
return DownloadResult(outcome=Outcome.COMPLETED, bytes_done=len(_BODY))
with patch.object(net, "download", side_effect=fake_download):
assert worker.run_once() is TaskKind.DOWNLOAD
events = _drain(deps.events)
ready = [event for event in events if isinstance(event, FileReady)]
assert len(ready) == 1
assert ready[0].item.md5 == _MD5
assert ready[0].item.path.exists()
assert not ready[0].item.path.name.endswith(".part")
assert ready[0].item.orig_name == "clip.webm"
assert any(isinstance(event, Downloaded) for event in events)
assert worker.downloaded == 1
assert store.counts(deps.conn)[FileState.READY.value] == 1
def test_a_404_is_terminal_and_never_retried(deps: downloader.WorkerDeps) -> None:
"""This is what stops a fallen-off thread being re-fetched every run."""
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.NOT_FOUND, bytes_done=0),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.GONE.value] == 1
assert store.pending_downloads(deps.conn) == 0
def test_a_transient_failure_is_retried(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.TRANSIENT, bytes_done=0),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.FAILED.value] == 1
assert store.pending_downloads(deps.conn) == 1
def test_a_transient_failure_gives_up_after_the_attempt_budget(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.TRANSIENT, bytes_done=0),
):
for _ in range(3):
worker.run_once()
assert store.counts(deps.conn)[FileState.GONE.value] == 1
def test_a_retry_after_pauses_the_worker(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with (
patch.object(
net,
"download",
return_value=DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=0,
retry_after=5.0,
),
),
patch.object(deps.stop, "wait") as waited,
):
worker.run_once()
assert any(call.args and call.args[0] == 5.0 for call in waited.call_args_list)
def test_a_checksum_mismatch_marks_the_file_corrupt(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.CHECKSUM_MISMATCH, bytes_done=0),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.CORRUPT.value] == 1
def test_an_aborted_download_stays_resumable(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.ABORTED, bytes_done=4),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.FAILED.value] == 1
assert store.pending_downloads(deps.conn) == 1

333
python_pkg/wsg_grabber/tests/test_downloader_part2.py Normal file
View File

@ -0,0 +1,333 @@
"""More tests for the background worker.
Split from test_downloader.py to stay under this repo's 500-line-per-file cap;
the fixtures are duplicated because each test module must stand alone.
"""
from __future__ import annotations
import queue
import threading
import time
from typing import TYPE_CHECKING, cast
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net, store, store_threads
from python_pkg.wsg_grabber.constants import IDLE_RESCAN_S, THREADS_URL
from python_pkg.wsg_grabber.models import (
DownloadEvent,
DownloadResult,
JsonResponse,
Outcome,
RemoteFile,
ScanFinished,
TaskKind,
ThreadRef,
WorkerFailed,
)
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_download_progress_is_persisted(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
def fake_download(_session: object, transfer: net.Transfer) -> DownloadResult:
assert transfer.on_progress is not None
transfer.on_progress(7)
return DownloadResult(outcome=Outcome.ABORTED, bytes_done=7)
with patch.object(net, "download", side_effect=fake_download):
worker.run_once()
assert deps.conn.execute("SELECT bytes_done FROM files").fetchone()[0] == 7
def test_claim_losing_a_race_is_a_no_op(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with (
patch.object(store, "claim_next", return_value=None),
patch.object(
net,
"download",
) as never,
):
worker._download_one()
never.assert_not_called()
def test_deleted_upstream_files_are_written_off(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
thread = {"posts": [{"no": 1, "md5": _MD5, "filedeleted": 1}]}
# Driven directly: with a file queued, run_once would always pick DOWNLOAD
# over SCAN, so the thread fetch under test would never happen.
with patch.object(net, "get_json", return_value=_json(thread)):
worker._fetch_thread(ThreadRef(7, 100))
assert store.counts(deps.conn)[FileState.GONE.value] == 1
assert store.pending_downloads(deps.conn) == 0
def test_an_already_downloaded_file_survives_upstream_deletion(
deps: downloader.WorkerDeps,
) -> None:
"""Once the bytes are ours, the post being deleted is irrelevant."""
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
store.mark_downloaded(deps.conn, _MD5, "1-x.webm")
thread = {"posts": [{"no": 1, "md5": _MD5, "filedeleted": 1}]}
with patch.object(net, "get_json", return_value=_json(thread)):
worker._fetch_thread(ThreadRef(7, 100))
assert store.state_of(deps.conn, _MD5) is FileState.READY
def test_idle_announces_completion_exactly_once(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
with patch.object(net, "get_json", return_value=_json([])):
worker.run_once()
with patch.object(deps.stop, "wait"):
assert worker.run_once() is TaskKind.IDLE
assert worker.run_once() is TaskKind.IDLE
finished = [e for e in _drain(deps.events) if isinstance(e, ScanFinished)]
assert len(finished) == 1
def test_the_board_is_rescanned_after_the_quiet_period(
deps: downloader.WorkerDeps,
) -> None:
"""A finished sweep goes quiet, then polls again for new content."""
worker = downloader.Worker(deps)
with patch.object(net, "get_json", return_value=_json([])):
worker.run_once()
with patch.object(deps.stop, "wait"):
assert worker.run_once() is TaskKind.IDLE
with patch(
"python_pkg.wsg_grabber.downloader.time.monotonic",
return_value=time.monotonic() + IDLE_RESCAN_S + 1,
):
assert worker.run_once() is TaskKind.IDLE
# the quiet period has elapsed, so the next step walks the board again
assert worker.run_once() is TaskKind.SCAN
finished = [e for e in _drain(deps.events) if isinstance(e, ScanFinished)]
assert len(finished) == 1
def test_rate_limiting_waits_between_requests(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
with (
patch.object(
net,
"get_json",
side_effect=[_json(listing), _json({"posts": []})],
),
patch.object(deps.stop, "wait") as waited,
):
worker.run_once() # the first request is free
worker.run_once() # the second must be spaced out
delays = [call.args[0] for call in waited.call_args_list if call.args]
assert delays
assert max(delays) <= 1.0
def test_run_reports_a_crash_and_closes_resources(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
with patch.object(worker, "run_once", side_effect=RuntimeError("kaput")):
worker.run()
events = _drain(deps.events)
assert any(
isinstance(event, WorkerFailed) and event.message == "kaput" for event in events
)
cast("MagicMock", deps.session).close.assert_called_once()
def test_run_exits_cleanly_when_stopped(deps: downloader.WorkerDeps) -> None:
deps.stop.set()
worker = downloader.Worker(deps)
worker.run()
assert _drain(deps.events) == []
cast("MagicMock", deps.session).close.assert_called_once()
@pytest.mark.timeout(10)
def test_start_runs_and_joins_a_real_thread(tmp_path: Path) -> None:
"""The one test that starts a thread; it must stop on demand."""
incoming = tmp_path / "incoming"
incoming.mkdir()
stop = threading.Event()
stop.set()
def build() -> downloader.WorkerDeps:
"""Open the connection on the worker thread, which must own it."""
return downloader.WorkerDeps(
conn=db.open_index(tmp_path / "index.db"),
session=MagicMock(),
incoming=incoming,
events=queue.SimpleQueue(),
stop=stop,
include_archive=False,
)
thread = downloader.start(build)
thread.join(timeout=5)
assert not thread.is_alive()
def test_the_board_endpoints_are_fetched_conditionally(
deps: downloader.WorkerDeps,
) -> None:
"""Without this an idle reviewer re-fetches the board unconditionally forever."""
deps.include_archive = True
worker = downloader.Worker(deps)
stamp = "Mon, 27 Jul 2026 10:00:00 GMT"
with patch.object(
net,
"get_json",
return_value=_json([], last_modified=stamp),
) as fake:
worker.run_once()
sent_first = [call.args[2] for call in fake.call_args_list]
worker._catalog_fresh = False
fake.reset_mock()
worker.run_once()
sent_again = [call.args[2] for call in fake.call_args_list]
assert sent_first == [None, None] # nothing stored yet
assert sent_again == [stamp, stamp] # the stamp goes back
def test_a_304_on_the_board_keeps_the_stored_stamp(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
stamp = "Mon, 27 Jul 2026 10:00:00 GMT"
with patch.object(net, "get_json", return_value=_json([], last_modified=stamp)):
worker.run_once()
with patch.object(net, "get_json", return_value=_json(None, not_modified=True)):
worker._catalog_fresh = False
worker.run_once()
assert store_threads.cached_last_modified(deps.conn, THREADS_URL) == stamp
def test_a_worker_that_cannot_build_its_resources_reports_it(
tmp_path: Path,
) -> None:
"""Otherwise the reviewer waits on downloads that will never come."""
reported: list[str] = []
thread = downloader.start(
lambda: (_ for _ in ()).throw(OSError("no database")),
reported.append,
)
thread.join(timeout=5)
assert not thread.is_alive()
assert reported
assert "no database" in reported[0]

163
python_pkg/wsg_grabber/tests/test_logs.py Normal file
View File

@ -0,0 +1,163 @@
"""Tests for the structured event log."""
from __future__ import annotations
import json
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import logs
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
@pytest.fixture(autouse=True)
def _closed() -> Iterator[None]:
"""Make sure no test leaves the log file open.
Yields:
None: Control returns to the test.
"""
try:
yield
finally:
logs.stop()
def _records(path: Path) -> list[dict[str, object]]:
"""Parse a session log.
Args:
path: The JSONL file.
Returns:
list[dict[str, object]]: One dict per line.
"""
return [
json.loads(line)
for line in path.read_text(encoding="utf-8").splitlines()
if line
]
def test_starting_creates_a_log_and_announces_itself(tmp_path: Path) -> None:
path = logs.start()
assert path.parent == logs.logs_dir()
assert str(path).startswith(str(tmp_path))
assert logs.current_path() == path
first = _records(path)[0]
assert first["event"] == "session.start"
assert first["seq"] == 1
assert first["level"] == "info"
def test_every_record_carries_the_standard_fields() -> None:
path = logs.start()
logs.event("thing.happened", detail="x", count=3)
record = _records(path)[-1]
assert set(record) >= {"ts", "seq", "level", "event", "thread"}
assert record["event"] == "thing.happened"
assert record["detail"] == "x"
assert record["count"] == 3
def test_sequence_numbers_are_monotonic() -> None:
path = logs.start()
for index in range(5):
logs.event("tick", index=index)
seqs = [int(str(record["seq"])) for record in _records(path)]
assert seqs == sorted(seqs)
assert len(set(seqs)) == len(seqs)
def test_level_helpers_set_the_level() -> None:
path = logs.start("debug")
logs.debug("a")
logs.warning("b")
logs.error("c")
levels = [record["level"] for record in _records(path)[1:]]
assert levels == ["debug", "warning", "error"]
def test_records_below_the_threshold_are_dropped() -> None:
path = logs.start("warning")
logs.debug("dropped")
logs.event("also-dropped")
logs.warning("kept")
events = [record["event"] for record in _records(path)]
assert "dropped" not in events
assert "also-dropped" not in events
assert "kept" in events
def test_an_unknown_level_name_falls_back_to_info() -> None:
path = logs.start("nonsense")
logs.event("recorded", level="nonsense")
assert any(record["event"] == "recorded" for record in _records(path))
def test_values_that_cannot_be_serialised_are_stringified() -> None:
path = logs.start()
logs.event("odd", value=object(), where=tmp_marker())
record = _records(path)[-1]
assert isinstance(record["value"], str)
def tmp_marker() -> object:
"""Return a non-JSON-encodable value.
Returns:
object: Something json cannot encode directly.
"""
return {1, 2, 3}
def test_events_before_start_go_nowhere_and_do_not_raise() -> None:
logs.stop()
logs.event("ignored")
assert logs.current_path() is None
def test_echo_mirrors_records_to_stderr(
capsys: pytest.CaptureFixture[str],
) -> None:
logs.start(echo=True)
logs.event("mirrored", detail="yes")
captured = capsys.readouterr().err
assert "mirrored" in captured
assert json.loads(captured.strip().splitlines()[-1])["detail"] == "yes"
def test_starting_twice_rotates_to_a_new_file() -> None:
first = logs.start()
logs.event("one")
second = logs.start()
logs.event("two")
assert logs.current_path() == second
assert "one" in first.read_text(encoding="utf-8")
def test_stop_is_idempotent() -> None:
logs.start()
logs.stop()
logs.stop()
assert logs.current_path() is None
def test_the_log_lands_inside_the_sandbox(tmp_path: Path) -> None:
"""The suite must never write a log into the real data directory."""
assert str(logs.logs_dir()).startswith(str(tmp_path))
def test_each_session_numbers_its_own_records_from_one() -> None:
"""seq orders records within one file, so it must restart per session."""
first = logs.start()
logs.event("a")
logs.event("b")
second = logs.start()
assert _records(second)[0]["seq"] == 1
assert len(_records(first)) == 3

30
python_pkg/wsg_grabber/tests/test_main_entry.py Normal file
View File

@ -0,0 +1,30 @@
"""Tests for the ``python -m python_pkg.wsg_grabber`` entry point."""
from __future__ import annotations
import runpy
import sys
from unittest.mock import patch
import pytest
_MODULE = "python_pkg.wsg_grabber.__main__"
def test_running_as_a_module_exits_with_the_cli_status() -> None:
# pytest-randomly shuffles test order, so a previous import must not linger
sys.modules.pop(_MODULE, None)
with (
patch("python_pkg.wsg_grabber.cli.main", return_value=3) as entry,
pytest.raises(SystemExit) as exit_info,
):
runpy.run_module(_MODULE, run_name="__main__")
assert exit_info.value.code == 3
entry.assert_called_once_with()
def test_importing_the_module_has_no_side_effect() -> None:
sys.modules.pop(_MODULE, None)
with patch("python_pkg.wsg_grabber.cli.main") as entry:
runpy.run_module(_MODULE, run_name="not_main")
entry.assert_not_called()

370
python_pkg/wsg_grabber/tests/test_net.py Normal file
View File

@ -0,0 +1,370 @@
"""Tests for the HTTP layer.
No socket is opened: ``requests.Session`` is replaced with a mock whose
``iter_content`` yields real bytes, so the md5 verification path computes a
genuine digest rather than comparing two mocks.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
import pytest
import requests
from python_pkg.wsg_grabber import net
from python_pkg.wsg_grabber.constants import API_HEADERS, MEDIA_HEADERS
from python_pkg.wsg_grabber.models import Outcome
if TYPE_CHECKING:
from pathlib import Path
_BODY = b"fake webm bytes, but real bytes" * 40
def _response(
status: int = 200,
*,
body: bytes = b"",
headers: dict[str, str] | None = None,
text: str = "",
) -> MagicMock:
"""Build a stand-in for a requests Response.
Args:
status: HTTP status code.
body: Payload streamed by ``iter_content``.
headers: Response headers.
text: Body for JSON decoding.
Returns:
MagicMock: Response double.
"""
response = MagicMock()
response.status_code = status
response.headers = headers or {}
response.text = text
response.content = text.encode("utf-8")
response.iter_content.return_value = [body] if body else []
return response
def _transfer(tmp_path: Path, *, md5: str | None = None, size: int = 0) -> net.Transfer:
"""Build a Transfer pointing at a sandboxed part file.
Args:
tmp_path: Per-test temporary directory.
md5: Expected digest; defaults to the real digest of ``_BODY``.
size: Expected file size.
Returns:
net.Transfer: Transfer description.
"""
return net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5=md5 if md5 is not None else net.digest_of(_BODY),
expected_size=size,
should_stop=lambda: False,
)
def test_digest_matches_the_field_the_api_publishes() -> None:
"""Verified against a real /wsg/ file during design."""
assert net.digest_of(b"") == "1B2M2Y8AsgTpgAmY7PhCfg=="
assert len(net.digest_of(_BODY)) == 24
def test_build_session_carries_the_api_headers() -> None:
session = net.build_session()
try:
assert session.headers["User-Agent"] == API_HEADERS["User-Agent"]
assert session.headers["Referer"] == API_HEADERS["Referer"]
finally:
session.close()
def test_get_json_decodes_a_fresh_payload() -> None:
session = MagicMock()
session.get.return_value = _response(
200,
text='{"posts": [1]}',
headers={"Last-Modified": "Mon, 27 Jul 2026 08:00:00 GMT"},
)
result = net.get_json(session, "https://a.4cdn.org/wsg/threads.json")
assert result.payload == {"posts": [1]}
assert result.last_modified == "Mon, 27 Jul 2026 08:00:00 GMT"
assert not result.not_modified
assert session.get.call_args.kwargs["headers"] == {}
def test_get_json_sends_if_modified_since_and_honours_304() -> None:
"""An unchanged thread must cost a 304, not a re-parse."""
session = MagicMock()
session.get.return_value = _response(304)
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
result = net.get_json(session, "https://a.4cdn.org/wsg/thread/1.json", stamp)
assert result.not_modified
assert result.payload is None
assert result.last_modified == stamp
assert session.get.call_args.kwargs["headers"] == {"If-Modified-Since": stamp}
@pytest.mark.parametrize("status", [404, 410])
def test_get_json_reports_a_vanished_thread(status: int) -> None:
session = MagicMock()
session.get.return_value = _response(status)
result = net.get_json(session, "https://a.4cdn.org/wsg/thread/1.json")
assert result.not_found
assert result.payload is None
def test_get_json_raises_on_other_failures() -> None:
session = MagicMock()
response = _response(500)
response.raise_for_status.side_effect = requests.HTTPError("boom")
session.get.return_value = response
with pytest.raises(requests.HTTPError):
net.get_json(session, "https://a.4cdn.org/wsg/threads.json")
def test_download_writes_and_verifies_the_file(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = _transfer(tmp_path)
result = net.download(session, transfer)
assert result.outcome is Outcome.COMPLETED
assert result.bytes_done == len(_BODY)
assert transfer.part_path.read_bytes() == _BODY
assert (
session.get.call_args.kwargs["headers"]["Referer"] == MEDIA_HEADERS["Referer"]
)
assert "Range" not in session.get.call_args.kwargs["headers"]
def test_download_reports_progress(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
seen: list[int] = []
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5=net.digest_of(_BODY),
expected_size=0,
should_stop=lambda: False,
on_progress=seen.append,
)
net.download(session, transfer)
assert seen == [len(_BODY)]
def test_download_rejects_a_body_whose_digest_disagrees(tmp_path: Path) -> None:
"""A mangled transfer must never reach the reviewer."""
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = _transfer(tmp_path, md5="A" * 24)
result = net.download(session, transfer)
assert result.outcome is Outcome.CHECKSUM_MISMATCH
assert not transfer.part_path.exists()
@pytest.mark.parametrize("status", [404, 410])
def test_download_marks_a_missing_file_not_found(tmp_path: Path, status: int) -> None:
session = MagicMock()
session.get.return_value = _response(status)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.NOT_FOUND
def test_download_reads_retry_after_on_429(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(429, headers={"Retry-After": "45"})
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.retry_after == pytest.approx(45.0)
def test_download_treats_server_errors_as_transient(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(503)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.retry_after is None
def test_download_treats_an_unexpected_status_as_transient(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(302)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.message is not None
assert "302" in result.message
def test_download_turns_a_network_error_into_a_transient_result(
tmp_path: Path,
) -> None:
session = MagicMock()
session.get.side_effect = requests.ConnectionError("no route")
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.message == "no route"
def test_download_resumes_a_partial_file(tmp_path: Path) -> None:
part = tmp_path / "1.webm.part"
part.write_bytes(_BODY[:100])
session = MagicMock()
session.get.return_value = _response(206, body=_BODY[100:])
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert session.get.call_args.kwargs["headers"]["Range"] == "bytes=100-"
assert result.outcome is Outcome.COMPLETED
assert part.read_bytes() == _BODY
def test_a_server_ignoring_range_restarts_rather_than_appending(
tmp_path: Path,
) -> None:
"""Replying 200 to a Range request means the whole file is coming again."""
part = tmp_path / "1.webm.part"
part.write_bytes(_BODY[:100])
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.COMPLETED
assert part.read_bytes() == _BODY
def test_an_oversized_leftover_is_discarded_before_requesting(
tmp_path: Path,
) -> None:
part = tmp_path / "1.webm.part"
part.write_bytes(b"x" * 9999)
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert "Range" not in session.get.call_args.kwargs["headers"]
assert result.outcome is Outcome.COMPLETED
def test_a_rejected_range_restarts_from_zero(tmp_path: Path) -> None:
part = tmp_path / "1.webm.part"
part.write_bytes(_BODY[:100])
session = MagicMock()
session.get.return_value = _response(416)
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.TRANSIENT
assert not part.exists()
def test_shutdown_aborts_mid_stream_and_keeps_the_partial(tmp_path: Path) -> None:
"""Stopping must be immediate and must leave a resumable .part behind."""
session = MagicMock()
response = _response(200)
response.iter_content.return_value = [b"a" * 10, b"b" * 10, b"c" * 10]
session.get.return_value = response
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5="whatever" + "=" * 16,
expected_size=30,
should_stop=lambda: True,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.ABORTED
def test_empty_chunks_are_skipped(tmp_path: Path) -> None:
"""requests yields keep-alive chunks; they must not corrupt the digest."""
session = MagicMock()
response = _response(200)
response.iter_content.return_value = [b"", _BODY, b""]
session.get.return_value = response
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.COMPLETED
def test_size_of_missing_file_is_zero(tmp_path: Path) -> None:
assert net.size_of(tmp_path / "nope") == 0
def test_a_compressed_body_is_refused(tmp_path: Path) -> None:
"""We ask for identity; requests would silently expand a gzip bomb."""
session = MagicMock()
session.get.return_value = _response(
200,
body=_BODY,
headers={"Content-Encoding": "gzip"},
)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.message is not None
assert "Content-Encoding" in result.message
def test_a_body_larger_than_advertised_is_abandoned(tmp_path: Path) -> None:
"""A server that lies about its size must not fill the disk."""
session = MagicMock()
response = _response(200)
response.iter_content.return_value = [b"x" * (1024 * 1024)] * 4
session.get.return_value = response
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5=net.digest_of(_BODY),
expected_size=16,
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.TRANSIENT
assert not transfer.part_path.exists()
def test_an_unknown_size_is_not_treated_as_over_budget(tmp_path: Path) -> None:
"""fsize 0 means the API did not say; that must not block the download."""
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
result = net.download(session, _transfer(tmp_path, size=0))
assert result.outcome is Outcome.COMPLETED
def test_an_oversized_api_response_is_refused() -> None:
session = MagicMock()
response = _response(200)
response.content = b"[" + b"1," * 20_000_000 + b"1]"
session.get.return_value = response
with pytest.raises(ValueError, match="exceeded"):
net.get_json(session, "https://a.4cdn.org/wsg/threads.json")
def test_the_api_session_refuses_compression() -> None:
assert API_HEADERS["Accept-Encoding"] == "identity"

122
python_pkg/wsg_grabber/tests/test_paths.py Normal file
View File

@ -0,0 +1,122 @@
"""Tests for storage locations and for the isolation guarantee itself."""
from __future__ import annotations
import ast
import os
from pathlib import Path
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import paths
if TYPE_CHECKING:
import pytest
_HOME_CAPTURING_CALLS = frozenset({"home", "expanduser", "getenv", "expandvars"})
def test_data_dir_follows_xdg_data_home(tmp_path: Path) -> None:
assert paths.data_dir() == Path(os.environ["XDG_DATA_HOME"]) / "wsg_grabber"
assert str(paths.data_dir()).startswith(str(tmp_path))
def test_data_dir_falls_back_to_dot_local_share(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
monkeypatch.delenv("XDG_DATA_HOME", raising=False)
expected = Path.home() / ".local" / "share" / "wsg_grabber"
assert paths.data_dir() == expected
assert str(paths.data_dir()).startswith(str(tmp_path))
def test_derived_paths_live_under_the_data_dir() -> None:
root = paths.data_dir()
for candidate in (
paths.db_path(),
paths.incoming_dir(),
paths.keep_dir(),
paths.trash_dir(),
paths.ipc_socket_path(),
):
assert candidate.parent == root
def test_socket_path_names_the_board() -> None:
assert paths.ipc_socket_path().name == "mpv-wsg.sock"
def test_ensure_dirs_creates_every_directory() -> None:
paths.ensure_dirs()
for directory in (
paths.data_dir(),
paths.incoming_dir(),
paths.keep_dir(),
paths.trash_dir(),
):
assert directory.is_dir()
def test_ensure_dirs_is_idempotent() -> None:
paths.ensure_dirs()
paths.ensure_dirs()
assert paths.incoming_dir().is_dir()
def _module_level_home_calls(source: str) -> list[str]:
"""Return names of home-resolving calls made outside any function body.
Args:
source: Python source text of one module.
Returns:
list[str]: Offending attribute/function names, empty when the module is
clean.
"""
tree = ast.parse(source)
offenders: list[str] = []
for node in tree.body:
for inner in ast.walk(node):
if isinstance(inner, (ast.FunctionDef, ast.AsyncFunctionDef)):
break
else:
offenders.extend(_home_calls_in(node))
return offenders
def _home_calls_in(node: ast.AST) -> list[str]:
"""Collect home-resolving call names anywhere beneath *node*.
Args:
node: AST node to scan.
Returns:
list[str]: Names such as ``home`` or ``expanduser``.
"""
found: list[str] = []
for inner in ast.walk(node):
if not isinstance(inner, ast.Call):
continue
func = inner.func
name = func.attr if isinstance(func, ast.Attribute) else None
if name is None and isinstance(func, ast.Name):
name = func.id
if name in _HOME_CAPTURING_CALLS:
found.append(name)
return found
def test_no_module_captures_a_home_directory_at_import_time() -> None:
"""Guard the whole package, not just paths.py.
A constant like ``STATE = Path.home() / "x"`` is evaluated on import and
can never be redirected afterwards, which is exactly how a previous package
in this repo leaked test writes into real user data.
"""
package_root = Path(paths.__file__).parent
offenders: dict[str, list[str]] = {}
for source_file in sorted(package_root.glob("*.py")):
calls = _module_level_home_calls(source_file.read_text(encoding="utf-8"))
if calls:
offenders[source_file.name] = calls
assert offenders == {}

473
python_pkg/wsg_grabber/tests/test_player.py Normal file
View File

@ -0,0 +1,473 @@
"""Tests for the mpv subprocess driver.
No mpv binary is ever started: ``subprocess.Popen`` and ``socket.socket`` are
patched in the module's own namespace.
"""
from __future__ import annotations
import json
from pathlib import Path
import subprocess
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import logs, player
if TYPE_CHECKING:
from collections.abc import Iterator
def _socket_mock(**kwargs: object) -> MagicMock:
"""Build a socket stand-in whose reader thread terminates immediately.
An unconfigured MagicMock returns a MagicMock from recv(), which is truthy
and instant, so the drain loop spins as fast as the CPU allows -- and
MagicMock records every call, so memory grows without bound. That once
reached 21 GB and took the machine down, hence this helper: every socket
mock yields EOF unless a test deliberately says otherwise.
Args:
**kwargs: Attributes to set on the mock.
Returns:
MagicMock: Socket double.
"""
sock = MagicMock()
sock.recv.return_value = b""
for name, value in kwargs.items():
setattr(sock, name, value)
return sock
@pytest.fixture(autouse=True)
def _safe_sockets() -> Iterator[None]:
"""Stop a stray socket mock from turning the drain loop into a busy loop.
``MpvPlayer`` runs a reader thread that calls ``recv`` until EOF. An
unconfigured MagicMock returns a MagicMock -- truthy, instant, and it
records every call -- so the loop spins and memory grows without bound.
That once reached 21 GB and OOM-killed the machine. Defaulting the module's
socket factory to something that reports EOF means a test has to opt into
the hazard deliberately.
Yields:
None: Control returns to the test.
"""
with patch(
"python_pkg.wsg_grabber.player.socket.socket",
side_effect=lambda *_a, **_k: _socket_mock(),
):
yield
@pytest.fixture
def started(tmp_path: Path) -> Iterator[tuple[player.MpvPlayer, MagicMock, MagicMock]]:
"""Construct an MpvPlayer with its process and socket mocked out.
Args:
tmp_path: Per-test temporary directory.
Yields:
tuple: The player, its process mock and its socket mock.
"""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
def create_socket(_seconds: float) -> None:
"""Stand in for mpv finally creating its control socket."""
ipc.write_bytes(b"")
# __init__ unlinks any stale socket first, so it has to appear *during*
# the wait rather than being pre-created.
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch("python_pkg.wsg_grabber.player.time.sleep", side_effect=create_socket),
):
yield player.MpvPlayer(1234, ipc), process, sock
def test_argv_embeds_into_the_given_window(tmp_path: Path) -> None:
argv = player.build_argv(999, tmp_path / "s.sock")
assert argv[0] == "mpv"
assert "--wid=999" in argv
assert f"--input-ipc-server={tmp_path / 's.sock'}" in argv
def test_argv_declines_the_keyboard_so_tk_can_have_it(tmp_path: Path) -> None:
"""With --wid mpv's child window would otherwise swallow every keypress."""
argv = player.build_argv(1, tmp_path / "s.sock")
assert "--input-vo-keyboard=no" in argv
assert "--input-default-bindings=no" in argv
def test_argv_loops_and_stays_open(tmp_path: Path) -> None:
argv = player.build_argv(1, tmp_path / "s.sock")
assert "--loop-file=inf" in argv
assert "--idle=yes" in argv
assert "--keep-open=yes" in argv
def test_encode_produces_one_json_line() -> None:
payload = player.encode(["loadfile", "a.webm", "replace"])
assert payload.endswith(b"\n")
assert json.loads(payload) == {
"command": ["loadfile", "a.webm", "replace"],
}
def test_play_sends_loadfile(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
mpv, _process, sock = started
mpv.play(tmp_path / "clip.webm")
sock.sendall.assert_called_once_with(
player.encode(["loadfile", str(tmp_path / "clip.webm"), "replace"]),
)
def test_stop_blanks_the_video(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, _process, sock = started
mpv.stop()
assert sock.sendall.call_args.args[0] == player.encode(["stop"])
def test_a_dead_socket_does_not_raise(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
"""mpv can exit underneath us; is_alive reports that, sending must not blow up."""
mpv, _process, sock = started
sock.sendall.side_effect = BrokenPipeError("gone")
mpv.play(tmp_path / "clip.webm")
def test_is_alive_follows_the_process(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, process, _sock = started
assert mpv.is_alive()
process.poll.return_value = 0
assert not mpv.is_alive()
def test_close_quits_cleanly_and_removes_the_socket(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
mpv, process, sock = started
mpv.close()
assert sock.sendall.call_args.args[0] == player.encode(["quit"])
process.wait.assert_called_once()
process.kill.assert_not_called()
sock.close.assert_called_once()
assert not (tmp_path / "mpv.sock").exists()
def test_close_kills_an_mpv_that_ignores_quit(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""An orphaned mpv would leave a window on screen after the reviewer exits."""
mpv, process, _sock = started
process.wait.side_effect = [subprocess.TimeoutExpired("mpv", 1), None]
mpv.close()
process.kill.assert_called_once()
def test_a_socket_that_never_appears_is_a_timeout(tmp_path: Path) -> None:
ipc = tmp_path / "never.sock"
process = MagicMock()
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.time.sleep"),
patch(
"python_pkg.wsg_grabber.player.time.monotonic",
side_effect=[0.0, 0.0, 9999.0],
),
pytest.raises(TimeoutError, match="did not create"),
):
player.MpvPlayer(1, ipc)
process.kill.assert_called_once()
def test_the_socket_is_awaited_rather_than_assumed(tmp_path: Path) -> None:
"""mpv takes a moment to create the socket; connecting too early fails."""
ipc = tmp_path / "late.sock"
process = MagicMock()
sock = _socket_mock()
waits = {"n": 0}
def create_on_third_try(_seconds: float) -> None:
waits["n"] += 1
if waits["n"] >= 3:
ipc.write_bytes(b"")
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=create_on_third_try,
),
):
player.MpvPlayer(1, ipc)
assert waits["n"] == 3
def test_the_player_never_sets_geometry() -> None:
"""--wid makes mpv fill its parent; setting geometry shrinks the video."""
argv = player.build_argv(1, Path("sock"))
assert not any(arg.startswith("--geometry") for arg in argv)
assert not hasattr(player.MpvPlayer, "resize")
def test_probe_reports_what_mpv_is_doing(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
"""The answer that settles a 'it froze on one video' report."""
mpv, _process, _sock = started
reply = _socket_mock()
reply.recv.return_value = (
b'{"request_id": 9001, "data": "value", "error": "success"}\n'
)
reply.__enter__ = lambda self: self
reply.__exit__ = lambda *_a: False
with patch("python_pkg.wsg_grabber.player.socket.socket", return_value=reply):
result = mpv.probe()
assert set(result) == {
"path",
"pause",
"core-idle",
"eof-reached",
"playback-time",
"width",
"height",
}
assert result["path"] == "value"
def test_probe_records_a_failure_rather_than_raising(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, _process, _sock = started
with patch(
"python_pkg.wsg_grabber.player.socket.socket",
side_effect=ConnectionRefusedError("gone"),
):
result = mpv.probe()
assert all(str(value).startswith("error:") for value in result.values())
def test_probe_survives_a_socket_that_closes_early(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, _process, _sock = started
reply = _socket_mock()
reply.recv.return_value = b""
reply.__enter__ = lambda self: self
reply.__exit__ = lambda *_a: False
with patch("python_pkg.wsg_grabber.player.socket.socket", return_value=reply):
result = mpv.probe()
assert result["path"] == "error: socket closed"
def test_probe_ignores_unrelated_traffic_on_the_socket(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""mpv pushes async events down the socket; they must not be mistaken
for the reply, and a partial line must not crash the reader."""
mpv, _process, _sock = started
reply = _socket_mock()
reply.recv.side_effect = [
b'{"event": "property-change"}\nnot json\n',
b'{"request_id": 9001, "data": 42}\n',
] * 10
reply.__enter__ = lambda self: self
reply.__exit__ = lambda *_a: False
with patch("python_pkg.wsg_grabber.player.socket.socket", return_value=reply):
assert mpv.probe()["path"] == 42
def test_a_failed_send_is_logged_not_swallowed(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
"""A dead socket must not make a frozen reviewer look like a working one."""
mpv, _process, sock = started
sock.sendall.side_effect = BrokenPipeError("gone")
log = logs.start("debug")
try:
mpv.play(tmp_path / "clip.webm")
finally:
logs.stop()
recorded = [
json.loads(line)
for line in log.read_text(encoding="utf-8").splitlines()
if line
]
assert any(r["event"] == "player.send_failed" for r in recorded)
assert any(r["level"] == "error" for r in recorded)
def test_the_socket_is_drained_so_mpv_keeps_accepting_commands(
tmp_path: Path,
) -> None:
"""The regression that made the reviewer freeze on one video.
mpv pushes events down the same socket; a client that never reads lets the
buffer fill, at which point mpv stops reading commands while still looking
perfectly alive.
"""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
seen: list[int] = []
def recv(size: int) -> bytes:
seen.append(size)
return b'{"event":"property-change"}\n' if len(seen) < 3 else b""
sock.recv.side_effect = recv
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
):
mpv = player.MpvPlayer(1, ipc)
mpv._reader.join(timeout=5)
mpv.close()
assert seen, "the reader never read from the socket"
assert not mpv._reader.is_alive()
def test_the_reader_stops_on_a_socket_error(tmp_path: Path) -> None:
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
sock.recv.side_effect = ConnectionResetError("gone")
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
):
mpv = player.MpvPlayer(1, ipc)
mpv._reader.join(timeout=5)
assert not mpv._reader.is_alive()
mpv.close()
def test_the_reader_keeps_going_through_read_timeouts(tmp_path: Path) -> None:
"""A quiet socket must not be mistaken for a closed one."""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
calls = {"n": 0}
def recv(_size: int) -> bytes:
calls["n"] += 1
if calls["n"] < 3:
raise TimeoutError
return b""
sock.recv.side_effect = recv
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
):
mpv = player.MpvPlayer(1, ipc)
mpv._reader.join(timeout=5)
mpv.close()
assert calls["n"] >= 3
def test_mpv_errors_are_logged_rather_than_discarded(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""A file mpv cannot decode should be visible, not just look stuck."""
mpv, _process, _sock = started
log = logs.start("debug")
try:
mpv._note_errors(
b'{"error":"success"}\n'
b"\n"
b"garbage\n"
b'{"request_id":1,"error":"loadfile failed"}\n',
)
finally:
logs.stop()
recorded = [
json.loads(line)
for line in log.read_text(encoding="utf-8").splitlines()
if line
]
failures = [r for r in recorded if r["event"] == "player.reply_error"]
assert len(failures) == 1
assert failures[0]["detail"] == "loadfile failed"
def test_socket_stand_ins_always_report_eof() -> None:
"""The contract that keeps the reader thread from becoming a busy loop.
A mock whose recv() returns a MagicMock is truthy and instant, so the drain
loop spins and the mock's recorded-call list grows without bound -- 21 GB
and an OOM kill, in the incident this guards against.
"""
assert _socket_mock().recv(65536) == b""
def test_the_reader_returns_promptly_once_asked_to_stop(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""Shutdown must not wait on another read."""
mpv, _process, sock = started
sock.recv.reset_mock()
mpv._stopping.set()
mpv._drain()
sock.recv.assert_not_called()
def test_a_refused_connection_does_not_leak_the_socket(tmp_path: Path) -> None:
"""A stale socket file makes connect() fail; nothing may be left open."""
ipc = tmp_path / "stale.sock"
process = MagicMock()
sock = _socket_mock()
sock.connect.side_effect = ConnectionRefusedError("stale")
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
pytest.raises(ConnectionRefusedError),
):
player.MpvPlayer(1, ipc)
sock.close.assert_called_once()
process.kill.assert_called_once()
process.wait.assert_called_once() # reaped, not left a zombie

297
python_pkg/wsg_grabber/tests/test_review.py Normal file
View File

@ -0,0 +1,297 @@
"""Tests for the reviewer's behaviour.
Because every decision lives here rather than in the Tk module, the whole UI is
specified by these tests and none of them need a display.
"""
from __future__ import annotations
from pathlib import Path
from python_pkg.wsg_grabber import review
from python_pkg.wsg_grabber.models import (
Downloaded,
Emptiness,
FileReady,
Indexed,
ReviewedItem,
ReviewItem,
ScanFinished,
Verdict,
WorkerFailed,
)
def _item(name: str = "a") -> ReviewItem:
"""Build a ReviewItem.
Args:
name: Distinguishes items within a test.
Returns:
ReviewItem: Fixture value.
"""
return ReviewItem(
md5=f"{name}-md5",
path=Path(f"/incoming/{name}.webm"),
orig_name=f"{name}.webm",
fsize=2 * 1024 * 1024,
width=480,
height=360,
)
def test_the_first_download_goes_straight_on_screen() -> None:
"""Watching starts as soon as one file lands, not when all of them do."""
state = review.initial_state(indexed=9000, ready=[])
assert state.current is None
state = review.on_new_files(state, [_item("a")])
assert state.current is not None
assert state.current.md5 == "a-md5"
assert state.pending == ()
def test_later_downloads_queue_behind_the_current_one() -> None:
state = review.initial_state(9000, [_item("a")])
state = review.on_new_files(state, [_item("b"), _item("c")])
assert state.current is not None
assert state.current.md5 == "a-md5"
assert [item.md5 for item in state.pending] == ["b-md5", "c-md5"]
def test_new_files_with_nothing_to_add_is_a_no_op() -> None:
state = review.initial_state(1, [_item("a")])
assert review.on_new_files(state, []) is state
def test_a_previous_run_resumes_with_its_unreviewed_videos() -> None:
state = review.initial_state(50, [_item("a"), _item("b")])
assert state.current is not None
assert state.current.md5 == "a-md5"
assert len(state.pending) == 1
def test_keep_counts_and_advances() -> None:
state = review.initial_state(2, [_item("a"), _item("b")])
state = review.on_verdict(state, Verdict.KEEP)
assert state.kept == 1
assert state.passed == 0
assert state.current is not None
assert state.current.md5 == "b-md5"
def test_pass_counts_and_advances() -> None:
state = review.initial_state(2, [_item("a"), _item("b")])
state = review.on_verdict(state, Verdict.SKIP)
assert state.passed == 1
assert state.kept == 0
assert state.current is not None
assert state.current.md5 == "b-md5"
def test_the_last_verdict_empties_the_screen() -> None:
state = review.initial_state(1, [_item("a")])
state = review.on_verdict(state, Verdict.KEEP)
assert state.current is None
assert review.emptiness(state) is Emptiness.WAITING
def test_a_verdict_with_nothing_showing_is_ignored() -> None:
state = review.initial_state(0, [])
assert review.on_verdict(state, Verdict.KEEP) is state
def test_a_vanished_file_is_skipped() -> None:
state = review.initial_state(2, [_item("a"), _item("b")])
state = review.on_missing_locally(state)
assert state.current is not None
assert state.current.md5 == "b-md5"
assert state.kept == 0
assert state.passed == 0
def test_missing_with_nothing_showing_is_ignored() -> None:
state = review.initial_state(0, [])
assert review.on_missing_locally(state) is state
def test_events_fold_into_the_state() -> None:
state = review.initial_state(0, [])
state = review.on_event(state, Indexed(known=9028))
assert state.indexed == 9028
state = review.on_event(state, FileReady(item=_item("a")))
assert state.current is not None
state = review.on_event(state, Downloaded(count=7))
assert state.downloaded == 7
state = review.on_event(state, ScanFinished())
assert state.scan_complete
state = review.on_event(state, WorkerFailed(message="no route to host"))
assert state.error == "no route to host"
def test_emptiness_distinguishes_waiting_from_exhausted() -> None:
state = review.initial_state(0, [])
assert review.emptiness(state) is Emptiness.WAITING
state = review.on_event(state, ScanFinished())
assert review.emptiness(state) is Emptiness.EXHAUSTED
state = review.on_new_files(state, [_item("a")])
assert review.emptiness(state) is Emptiness.NOT_EMPTY
def test_status_line_reports_progress() -> None:
state = review.initial_state(9028, [_item("a")])
state = review.on_event(state, Downloaded(count=12))
state = review.on_verdict(state, Verdict.KEEP)
line = review.status_line(state)
assert "kept 1" in line
assert "passed 0" in line
assert "downloaded 12/9028" in line
def test_status_line_omits_the_total_before_indexing() -> None:
state = review.initial_state(0, [_item("a")])
assert "/0" not in review.status_line(state)
def test_status_line_shows_the_two_empty_moods() -> None:
state = review.initial_state(0, [])
assert "waiting for downloads" in review.status_line(state)
assert "all caught up" in review.status_line(
review.on_event(state, ScanFinished()),
)
def test_a_worker_failure_takes_over_the_status_line() -> None:
state = review.on_event(review.initial_state(0, []), WorkerFailed(message="boom"))
assert review.status_line(state) == "downloader stopped: boom"
def test_filename_line_describes_the_current_video() -> None:
state = review.initial_state(1, [_item("a")])
line = review.filename_line(state)
assert "a.webm" in line
assert "480x360" in line
assert "2.0 MiB" in line
def test_filename_line_has_a_placeholder_when_idle() -> None:
assert review.filename_line(review.initial_state(0, [])) == "—"
def test_title_line_carries_the_tally() -> None:
state = review.initial_state(1, [_item("a")])
state = review.on_verdict(state, Verdict.KEEP)
assert review.title_line(state) == "/wsg/ — kept 1, passed 0"
def test_render_drives_the_window_completely() -> None:
state = review.initial_state(5, [_item("a")])
command = review.render(state)
assert command.play == Path("/incoming/a.webm")
assert not command.stop
assert command.verdicts_enabled
assert not command.quit_app
assert command.status == review.status_line(state)
assert command.title == review.title_line(state)
assert command.filename == review.filename_line(state)
def test_render_stops_playback_when_nothing_is_queued() -> None:
command = review.render(review.initial_state(0, []))
assert command.play is None
assert command.stop
assert not command.verdicts_enabled
def test_render_signals_quit_and_disables_the_buttons() -> None:
state = review.on_quit(review.initial_state(1, [_item("a")]))
command = review.render(state)
assert command.quit_app
assert not command.verdicts_enabled
def _reviewed(item: ReviewItem, choice: Verdict = Verdict.SKIP) -> ReviewedItem:
"""Build the record a verdict would have persisted.
Args:
item: The reviewed video.
choice: What was decided.
Returns:
ReviewedItem: Reversible verdict.
"""
return ReviewedItem(item=item, choice=choice, reviewed_name=item.path.name)
def test_undo_availability_comes_from_the_index_not_memory() -> None:
"""The count is seeded from the index, so undo survives a restart."""
resumed = review.initial_state(5, [_item("a")], undoable=3)
assert review.can_undo(resumed)
assert "u undoes (3)" in review.status_line(resumed)
assert not review.can_undo(review.initial_state(5, [_item("a")]))
def test_a_recorded_verdict_can_be_undone() -> None:
first = _item("a")
state = review.initial_state(2, [first, _item("b")])
state = review.on_verdict(state, Verdict.SKIP)
assert review.can_undo(state)
state = review.on_undo(state, _reviewed(first))
assert state.current is not None
assert state.current.md5 == "a-md5"
assert state.passed == 0
assert [item.md5 for item in state.pending] == ["b-md5"]
assert not review.can_undo(state)
def test_undo_restores_the_keep_counter() -> None:
first = _item("a")
state = review.initial_state(1, [first])
state = review.on_verdict(state, Verdict.KEEP)
assert state.kept == 1
assert review.on_undo(state, _reviewed(first, Verdict.KEEP)).kept == 0
def test_undo_works_when_the_queue_has_run_dry() -> None:
first = _item("a")
state = review.initial_state(1, [first])
state = review.on_verdict(state, Verdict.SKIP)
assert state.current is None
state = review.on_undo(state, _reviewed(first))
assert state.current is not None
assert state.current.md5 == "a-md5"
def test_forget_last_drops_one_undo_without_restoring() -> None:
first = _item("a")
state = review.initial_state(1, [first])
state = review.on_verdict(state, Verdict.SKIP)
dropped = review.forget_last(state)
assert not review.can_undo(dropped)
assert dropped.passed == 1
assert review.forget_last(dropped).undoable == 0
def test_the_status_line_advertises_undo_once_it_is_possible() -> None:
first = _item("a")
state = review.initial_state(1, [first])
assert "u undoes" not in review.status_line(state)
state = review.on_verdict(state, Verdict.SKIP)
assert "u undoes (1)" in review.status_line(state)
def test_render_enables_undo_only_when_something_can_be_undone() -> None:
first = _item("a")
state = review.initial_state(1, [first])
assert not review.render(state).undo_enabled
state = review.on_verdict(state, Verdict.KEEP)
assert review.render(state).undo_enabled
assert not review.render(review.on_quit(state)).undo_enabled

103
python_pkg/wsg_grabber/tests/test_scanner.py Normal file
View File

@ -0,0 +1,103 @@
"""Tests for the pure scheduling and retry helpers."""
from __future__ import annotations
import pytest
from python_pkg.wsg_grabber import scanner
from python_pkg.wsg_grabber.constants import (
BACKOFF_BASE_S,
BACKOFF_CAP_S,
MIN_REQUEST_INTERVAL_S,
)
from python_pkg.wsg_grabber.models import ResumePlan, TaskKind, ThreadRef
def test_downloads_are_preferred_so_a_video_appears_early() -> None:
"""The user watches while the rest load; fetching must outrank scanning."""
assert scanner.choose_task(5, 10, catalog_fresh=True) is TaskKind.DOWNLOAD
assert scanner.choose_task(1, 0, catalog_fresh=False) is TaskKind.DOWNLOAD
def test_scanning_happens_when_nothing_is_queued() -> None:
assert scanner.choose_task(0, 3, catalog_fresh=True) is TaskKind.SCAN
assert scanner.choose_task(0, 0, catalog_fresh=False) is TaskKind.SCAN
def test_idle_only_when_the_board_is_exhausted() -> None:
assert scanner.choose_task(0, 0, catalog_fresh=True) is TaskKind.IDLE
def test_stale_threads_picks_unseen_and_bumped_threads() -> None:
known = {1: 100, 2: 200}
live = [ThreadRef(1, 100), ThreadRef(2, 250), ThreadRef(3, 10)]
assert [ref.thread_no for ref in scanner.stale_threads(known, live)] == [2, 3]
def test_stale_threads_is_empty_on_an_unchanged_board() -> None:
known = {1: 100}
assert scanner.stale_threads(known, [ThreadRef(1, 100)]) == []
def test_next_delay_enforces_one_request_per_second() -> None:
assert scanner.next_delay(0.0, 0.0) == pytest.approx(MIN_REQUEST_INTERVAL_S)
assert scanner.next_delay(10.0, 10.5) == pytest.approx(0.5)
def test_next_delay_never_goes_negative() -> None:
assert scanner.next_delay(10.0, 99.0) == 0.0
def test_backoff_doubles_and_then_caps() -> None:
assert scanner.backoff_seconds(1) == pytest.approx(BACKOFF_BASE_S)
assert scanner.backoff_seconds(2) == pytest.approx(BACKOFF_BASE_S * 2)
assert scanner.backoff_seconds(3) == pytest.approx(BACKOFF_BASE_S * 4)
assert scanner.backoff_seconds(99) == pytest.approx(BACKOFF_CAP_S)
def test_backoff_treats_attempt_zero_as_the_first_failure() -> None:
assert scanner.backoff_seconds(0) == pytest.approx(BACKOFF_BASE_S)
def test_an_explicit_retry_after_wins_but_is_still_capped() -> None:
assert scanner.backoff_seconds(1, retry_after=7.0) == pytest.approx(7.0)
assert scanner.backoff_seconds(1, retry_after=9999.0) == pytest.approx(
BACKOFF_CAP_S,
)
def test_a_zero_retry_after_falls_back_to_exponential_backoff() -> None:
assert scanner.backoff_seconds(2, retry_after=0.0) == pytest.approx(
BACKOFF_BASE_S * 2,
)
def test_parse_retry_after_handles_the_forms_we_accept() -> None:
assert scanner.parse_retry_after("30") == pytest.approx(30.0)
assert scanner.parse_retry_after(" 12 ") == pytest.approx(12.0)
assert scanner.parse_retry_after(None) is None
assert scanner.parse_retry_after("Wed, 21 Oct 2026 07:28:00 GMT") is None
assert scanner.parse_retry_after("-5") is None
def test_resume_plan_starts_from_scratch_when_nothing_is_on_disk() -> None:
plan = scanner.resume_plan(0, 1000)
assert plan == ResumePlan(offset=0, discard=False)
def test_resume_plan_continues_a_genuine_partial() -> None:
plan = scanner.resume_plan(400, 1000)
assert plan.offset == 400
assert not plan.discard
def test_resume_plan_bins_an_oversized_leftover() -> None:
"""A .part at or beyond the expected size is corrupt, not resumable."""
assert scanner.resume_plan(1000, 1000).discard
assert scanner.resume_plan(1500, 1000).discard
def test_resume_plan_with_unknown_expected_size_still_resumes() -> None:
plan = scanner.resume_plan(400, 0)
assert plan.offset == 400
assert not plan.discard

90
python_pkg/wsg_grabber/tests/test_states.py Normal file
View File

@ -0,0 +1,90 @@
"""Tests for the file lifecycle transition function."""
from __future__ import annotations
import pytest
from python_pkg.wsg_grabber.constants import MAX_ATTEMPTS
from python_pkg.wsg_grabber.states import (
CLAIMABLE,
TERMINAL,
FileEvent,
FileState,
is_terminal,
next_state,
)
@pytest.mark.parametrize("state", sorted(CLAIMABLE))
def test_claiming_starts_a_download(state: FileState) -> None:
assert next_state(state, FileEvent.CLAIMED, 0) is FileState.DOWNLOADING
@pytest.mark.parametrize("state", sorted(CLAIMABLE))
def test_upstream_deletion_before_download_is_terminal(state: FileState) -> None:
assert next_state(state, FileEvent.DELETED_UPSTREAM, 0) is FileState.GONE
def test_completed_download_becomes_reviewable() -> None:
assert next_state(FileState.DOWNLOADING, FileEvent.COMPLETED, 1) is FileState.READY
def test_missing_upstream_file_is_never_retried() -> None:
assert next_state(FileState.DOWNLOADING, FileEvent.NOT_FOUND, 1) is FileState.GONE
def test_transient_error_retries_until_the_attempt_budget_runs_out() -> None:
assert (
next_state(FileState.DOWNLOADING, FileEvent.TRANSIENT_ERROR, 1)
is FileState.FAILED
)
assert (
next_state(FileState.DOWNLOADING, FileEvent.TRANSIENT_ERROR, MAX_ATTEMPTS)
is FileState.GONE
)
def test_checksum_mismatch_retries_until_the_attempt_budget_runs_out() -> None:
assert (
next_state(FileState.DOWNLOADING, FileEvent.CHECKSUM_MISMATCH, 1)
is FileState.CORRUPT
)
assert (
next_state(FileState.DOWNLOADING, FileEvent.CHECKSUM_MISMATCH, MAX_ATTEMPTS)
is FileState.GONE
)
def test_verdicts_are_terminal() -> None:
assert next_state(FileState.READY, FileEvent.KEPT, 1) is FileState.KEPT
assert next_state(FileState.READY, FileEvent.PASSED, 1) is FileState.PASSED
def test_a_vanished_local_file_is_written_off() -> None:
assert next_state(FileState.READY, FileEvent.MISSING_LOCALLY, 1) is FileState.GONE
def test_ready_ignores_upstream_deletion() -> None:
"""The bytes are already ours; the post being deleted is irrelevant."""
with pytest.raises(ValueError, match="ready"):
next_state(FileState.READY, FileEvent.DELETED_UPSTREAM, 1)
def test_illegal_transition_out_of_downloading_raises() -> None:
with pytest.raises(ValueError, match="downloading"):
next_state(FileState.DOWNLOADING, FileEvent.KEPT, 1)
def test_illegal_transition_from_a_terminal_state_raises() -> None:
with pytest.raises(ValueError, match="illegal transition"):
next_state(FileState.GONE, FileEvent.CLAIMED, 0)
def test_terminal_states_are_exactly_the_three_end_points() -> None:
assert {FileState.KEPT, FileState.PASSED, FileState.GONE} == TERMINAL
for state in TERMINAL:
assert is_terminal(state)
for state in CLAIMABLE:
assert not is_terminal(state)
assert not is_terminal(FileState.READY)
assert not is_terminal(FileState.DOWNLOADING)

235
python_pkg/wsg_grabber/tests/test_store.py Normal file
View File

@ -0,0 +1,235 @@
"""Tests for the typed index operations."""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
import pytest
from python_pkg.wsg_grabber import db, store, store_threads, store_verdicts
from python_pkg.wsg_grabber.models import RemoteFile, ThreadRef
from python_pkg.wsg_grabber.states import CLAIMABLE, FileState
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
import sqlite3
_EXPECTED_CLAIMABLE_STATES = 3
@pytest.fixture
def conn(tmp_path: Path) -> Iterator[sqlite3.Connection]:
"""Open a migrated index for one test and close it afterwards.
Closing matters: pytest runs with ``filterwarnings = ["error", ...]`` and a
leaked connection surfaces as a ResourceWarning, i.e. a test failure.
Args:
tmp_path: Per-test temporary directory.
Yields:
sqlite3.Connection: Ready connection.
"""
connection = db.open_index(tmp_path / "index.db")
try:
yield connection
finally:
connection.close()
def _file(md5: str = "AAAA", tim: int = 111, thread_no: int = 9) -> RemoteFile:
"""Build a RemoteFile with sensible defaults.
Args:
md5: Identity.
tim: CDN filename stem.
thread_no: Owning thread.
Returns:
RemoteFile: Test fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=1234,
width=480,
height=360,
thread_no=thread_no,
post_no=tim,
)
def test_claimable_placeholder_count_matches_the_state_set() -> None:
"""The SQL spells out one '?' per claimable state; keep them in step."""
assert len(CLAIMABLE) == _EXPECTED_CLAIMABLE_STATES
assert len(store.claimable_values()) == _EXPECTED_CLAIMABLE_STATES
def test_local_name_ignores_the_poster_supplied_filename() -> None:
name = store.local_name("ab/cd+ef==", 555, ".webm")
assert name.startswith("555-")
assert name.endswith(".webm")
assert "/" not in name
assert "+" not in name
def test_record_files_counts_only_new_rows(conn: sqlite3.Connection) -> None:
assert store.record_files(conn, [_file("A"), _file("B", tim=2)]) == 2
assert store.record_files(conn, [_file("B", tim=2), _file("C", tim=3)]) == 1
assert store.known_md5s(conn) == {"A", "B", "C"}
def test_record_files_with_nothing_to_do(conn: sqlite3.Connection) -> None:
assert store.record_files(conn, []) == 0
def test_a_repost_of_identical_bytes_never_becomes_a_second_row(
conn: sqlite3.Connection,
) -> None:
"""Same md5 in another thread must not cost a second download."""
store.record_files(conn, [_file("SAME", tim=1, thread_no=100)])
added = store.record_files(conn, [_file("SAME", tim=999, thread_no=200)])
assert added == 0
assert store.pending_downloads(conn) == 1
def test_claim_next_takes_the_oldest_and_marks_it_in_flight(
conn: sqlite3.Connection,
) -> None:
store.record_files(conn, [_file("A", tim=1)])
claimed = store.claim_next(conn)
assert claimed is not None
assert claimed.md5 == "A"
assert store.attempts_for(conn, "A") == 1
assert store.counts(conn)[FileState.DOWNLOADING.value] == 1
def test_claim_next_returns_none_when_idle(conn: sqlite3.Connection) -> None:
assert store.claim_next(conn) is None
def test_claim_next_rolls_back_on_failure() -> None:
"""A mid-transaction failure must not leave the claim half-applied.
``sqlite3.Connection.execute`` is read-only so it cannot be patched on a
real connection; a mock stands in purely to inject the failure.
"""
fake = MagicMock()
# BEGIN IMMEDIATE, then the SELECT blows up, then ROLLBACK must still run.
fake.execute.side_effect = [MagicMock(), RuntimeError("simulated"), MagicMock()]
with pytest.raises(RuntimeError, match="simulated"):
store.claim_next(fake)
executed = [call.args[0] for call in fake.execute.call_args_list]
assert executed[0] == "BEGIN IMMEDIATE"
assert executed[-1] == "ROLLBACK"
def test_attempts_for_unknown_file_is_zero(conn: sqlite3.Connection) -> None:
assert store.attempts_for(conn, "nope") == 0
def test_set_state_records_the_error(conn: sqlite3.Connection) -> None:
store.record_files(conn, [_file("A")])
store.set_state(conn, "A", FileState.GONE, error="404")
row = conn.execute("SELECT state, last_error FROM files").fetchone()
assert row["state"] == FileState.GONE.value
assert row["last_error"] == "404"
def test_mark_downloaded_makes_the_file_reviewable(
conn: sqlite3.Connection,
tmp_path: Path,
) -> None:
store.record_files(conn, [_file("A", tim=42)])
store.mark_downloaded(conn, "A", "42-x.webm")
items = store.ready_items(conn, tmp_path)
assert len(items) == 1
assert items[0].md5 == "A"
assert items[0].path == tmp_path / "42-x.webm"
assert items[0].orig_name == "clip.webm"
def test_ready_items_is_empty_before_any_download(
conn: sqlite3.Connection,
tmp_path: Path,
) -> None:
store.record_files(conn, [_file("A")])
assert store.ready_items(conn, tmp_path) == []
def test_record_progress_persists_partial_bytes(conn: sqlite3.Connection) -> None:
store.record_files(conn, [_file("A")])
store.record_progress(conn, "A", 4096)
assert conn.execute("SELECT bytes_done FROM files").fetchone()[0] == 4096
def test_record_verdict_is_terminal_and_stamps_the_time(
conn: sqlite3.Connection,
tmp_path: Path,
) -> None:
store.record_files(conn, [_file("A")])
store.mark_downloaded(conn, "A", "a.webm")
store_verdicts.record_verdict(conn, "A", FileState.PASSED)
assert store.ready_items(conn, tmp_path) == []
assert conn.execute("SELECT reviewed_at FROM files").fetchone()[0] is not None
assert store.pending_downloads(conn) == 0
def test_a_reviewed_file_is_still_remembered_so_it_never_returns(
conn: sqlite3.Connection,
) -> None:
store.record_files(conn, [_file("A")])
store_verdicts.record_verdict(conn, "A", FileState.PASSED)
assert "A" in store.known_md5s(conn)
assert store.record_files(conn, [_file("A")]) == 0
def test_reset_in_flight_recovers_an_interrupted_download(
conn: sqlite3.Connection,
) -> None:
store.record_files(conn, [_file("A")])
store.claim_next(conn)
assert store.reset_in_flight(conn) == 1
assert store.pending_downloads(conn) == 1
assert store.reset_in_flight(conn) == 0
def test_counts_reports_every_state_and_a_total(conn: sqlite3.Connection) -> None:
store.record_files(conn, [_file("A"), _file("B", tim=2)])
store.set_state(conn, "A", FileState.GONE)
result = store.counts(conn)
assert result["total"] == 2
assert result[FileState.GONE.value] == 1
assert result[FileState.NEW.value] == 1
def test_thread_bookkeeping_round_trip(conn: sqlite3.Connection) -> None:
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
store_threads.record_thread(conn, ThreadRef(7, 1000), stamp)
assert store_threads.known_threads(conn) == {7: 1000}
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
assert store_threads.http_last_modified(conn, 7) == stamp
store_threads.record_thread(conn, ThreadRef(7, 2000), None)
assert store_threads.known_threads(conn) == {7: 2000}
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
assert store_threads.http_last_modified(conn, 7) == stamp
def test_http_last_modified_absent_cases(conn: sqlite3.Connection) -> None:
assert store_threads.http_last_modified(conn, 404) is None
store_threads.record_thread(conn, ThreadRef(8, 1), None)
assert store_threads.http_last_modified(conn, 8) is None
def test_mark_thread_gone(conn: sqlite3.Connection) -> None:
store_threads.record_thread(conn, ThreadRef(7, 1000), None)
store_threads.mark_thread_gone(conn, 7)
row = conn.execute("SELECT status FROM threads WHERE thread_no = 7").fetchone()
assert row["status"] == "gone"

453
python_pkg/wsg_grabber/tests/test_ui.py Normal file
View File

@ -0,0 +1,453 @@
"""Tests for the Tk window.
``tkinter`` is replaced wholesale in the module's namespace, so every widget is
a mock and no real ``Tk()`` is ever constructed. That matters beyond speed: CI
has no display, and instantiating Tk there would fail outright.
There is deliberately little to test here, because the window contains no
decisions -- those all live in ``review`` and are covered by ``test_review``.
"""
from __future__ import annotations
import queue
from typing import TYPE_CHECKING, cast
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import review, ui
from python_pkg.wsg_grabber.constants import (
KEEP_KEYS,
PASS_KEYS,
QUIT_KEYS,
UNDO_KEYS,
)
from python_pkg.wsg_grabber.models import (
ReviewCommand,
ReviewedItem,
ReviewItem,
Verdict,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.review import SessionState
@pytest.fixture
def toolkit() -> Iterator[MagicMock]:
"""Replace tkinter in the ui module's namespace.
Yields:
MagicMock: The stand-in toolkit.
"""
fake = MagicMock()
# a fresh mock per widget, else status and filename are the same object
fake.Frame.side_effect = lambda *_a, **_k: MagicMock()
fake.Label.side_effect = lambda *_a, **_k: MagicMock()
fake.Button.side_effect = lambda *_a, **_k: MagicMock()
fake.BOTH = "both"
fake.X = "x"
fake.LEFT = "left"
fake.RIGHT = "right"
fake.BOTTOM = "bottom"
with patch.object(ui, "tk", fake):
yield fake
def _mock(widget: object) -> MagicMock:
"""View a mocked widget as the MagicMock it really is.
The Widgets dataclass is typed with real Tk classes, so mypy will not let
the tests reach ``call_args`` without this.
Args:
widget: The stand-in widget.
Returns:
MagicMock: The same object, typed for assertions.
"""
return cast("MagicMock", widget)
def _item(tmp_path: Path, name: str = "a") -> ReviewItem:
"""Build a ReviewItem backed by a real file.
Args:
tmp_path: Per-test temporary directory.
name: Distinguishes items.
Returns:
ReviewItem: Fixture value.
"""
path = tmp_path / f"{name}.webm"
path.write_bytes(b"video")
return ReviewItem(
md5=f"{name}-md5",
path=path,
orig_name=f"{name}.webm",
fsize=5,
width=4,
height=3,
)
def _callbacks() -> tuple[ui.Callbacks, MagicMock]:
"""Build callbacks that record what the window asked for.
Returns:
tuple: The callbacks and the recorder behind them.
"""
recorder = MagicMock()
_undone: dict[str, ReviewedItem | None] = {"item": None}
def commit(state: SessionState, choice: Verdict) -> SessionState:
recorder.commit(choice)
if state.current is not None:
_undone["item"] = ReviewedItem(
item=state.current,
choice=choice,
reviewed_name=state.current.path.name,
)
return review.on_verdict(state, choice)
def missing(state: SessionState) -> SessionState:
recorder.missing()
return review.on_missing_locally(state)
def undo(state: SessionState) -> SessionState:
recorder.undo()
action = _undone["item"]
if action is None or not review.can_undo(state):
return state
return review.on_undo(state, action)
return (
ui.Callbacks(
commit=commit,
missing=missing,
undo=undo,
shutdown=recorder.shutdown,
),
recorder,
)
def _window(
state: SessionState,
events: queue.SimpleQueue[DownloadEvent] | None = None,
) -> tuple[ui.ReviewWindow, MagicMock]:
"""Build a window over a mocked toolkit.
Args:
state: Starting state.
events: Optional event queue.
Returns:
tuple: The window and the callback recorder.
"""
callbacks, recorder = _callbacks()
window = ui.ReviewWindow(state, events or queue.SimpleQueue(), callbacks)
return window, recorder
def test_the_window_has_a_video_area_and_a_control_bar(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
assert toolkit.Tk.called
assert toolkit.Button.call_count == 3 # keep, pass, undo
# two frames: the video area and the control bar below it
assert toolkit.Frame.call_count == 2
# the video area expands; the bar sits under it rather than over it, because
# mpv's --wid child window covers whatever shares its frame
assert _mock(window.widgets.video).pack.call_args.kwargs["expand"]
def test_every_shortcut_is_bound(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
calls = _mock(window.widgets.root).bind.call_args_list
bound = {call.args[0] for call in calls}
assert set(KEEP_KEYS) <= bound
assert set(PASS_KEYS) <= bound
assert set(QUIT_KEYS) <= bound
assert set(UNDO_KEYS) <= bound
def test_keep_and_pass_shortcuts_reach_the_callbacks(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
state = review.initial_state(2, [_item(tmp_path, "a"), _item(tmp_path, "b")])
window, recorder = _window(state)
window.attach_player(MagicMock())
bound = _mock(window.widgets.root).bind.call_args_list
handlers = {call.args[0]: call.args[1] for call in bound}
handlers[KEEP_KEYS[0]](object())
first: object = recorder.commit.call_args.args[0]
assert first is Verdict.KEEP
handlers[PASS_KEYS[0]](object())
second: object = recorder.commit.call_args.args[0]
assert second is Verdict.SKIP
def test_the_quit_shortcut_shuts_down(toolkit: MagicMock) -> None:
window, recorder = _window(review.initial_state(0, []))
bound = _mock(window.widgets.root).bind.call_args_list
handlers = {call.args[0]: call.args[1] for call in bound}
handlers[QUIT_KEYS[0]](object())
recorder.shutdown.assert_called_once()
_mock(window.widgets.root).destroy.assert_called_once()
def test_the_buttons_call_the_same_handlers(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
state = review.initial_state(1, [_item(tmp_path)])
window, recorder = _window(state)
window.attach_player(MagicMock())
window.on_keep()
assert recorder.commit.call_args.args[0] is Verdict.KEEP
def test_apply_plays_and_labels(toolkit: MagicMock, tmp_path: Path) -> None:
window, _ = _window(review.initial_state(0, []))
player = MagicMock()
window.attach_player(player)
path = tmp_path / "x.webm"
path.write_bytes(b"v")
window.apply(
ReviewCommand(
play=path,
stop=False,
status="status text",
title="title text",
filename="file text",
verdicts_enabled=True,
undo_enabled=False,
quit_app=False,
),
)
player.play.assert_called_once_with(path)
_mock(window.widgets.status).configure.assert_called_with(text="status text")
_mock(window.widgets.filename).configure.assert_called_with(text="file text")
_mock(window.widgets.root).title.assert_called_with("title text")
_mock(window.widgets.keep).configure.assert_called_with(state="normal")
def test_the_same_video_is_only_loaded_once(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
"""refresh() runs every tick; re-sending loadfile restarts the video."""
window, _ = _window(review.initial_state(1, [_item(tmp_path)]))
player = MagicMock()
window.attach_player(player)
assert player.play.call_count == 1
for _ in range(10):
window.tick()
assert player.play.call_count == 1
def test_advancing_loads_the_next_video(toolkit: MagicMock, tmp_path: Path) -> None:
state = review.initial_state(2, [_item(tmp_path, "a"), _item(tmp_path, "b")])
window, _ = _window(state)
player = MagicMock()
window.attach_player(player)
window.on_keep()
assert player.play.call_count == 2
assert player.play.call_args.args[0].name == "b.webm"
def test_stopping_lets_the_same_video_load_again(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
item = _item(tmp_path)
window, _ = _window(review.initial_state(1, [item]))
player = MagicMock()
window.attach_player(player)
window.apply(
ReviewCommand(
play=None,
stop=True,
status="s",
title="t",
filename="f",
verdicts_enabled=False,
undo_enabled=False,
quit_app=False,
),
)
player.stop.assert_called_once()
window.refresh()
assert player.play.call_count == 2
def test_apply_stops_and_disables_when_idle(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
player = MagicMock()
window.attach_player(player)
window.apply(
ReviewCommand(
play=None,
stop=True,
status="s",
title="t",
filename="f",
verdicts_enabled=False,
undo_enabled=False,
quit_app=False,
),
)
# nothing was playing, so there is nothing to stop; the buttons still go dead
player.stop.assert_not_called()
_mock(window.widgets.keep).configure.assert_called_with(state="disabled")
def test_apply_quits_when_told_to(toolkit: MagicMock) -> None:
window, recorder = _window(review.initial_state(0, []))
window.apply(
ReviewCommand(
play=None,
stop=False,
status="s",
title="t",
filename="f",
verdicts_enabled=False,
undo_enabled=False,
quit_app=True,
),
)
recorder.shutdown.assert_called_once()
def test_playing_without_a_player_is_a_no_op(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
window, _ = _window(review.initial_state(1, [_item(tmp_path)]))
window.refresh()
def test_a_video_that_vanished_is_skipped(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
"""A file can disappear between download and review; don't hang on it."""
first = _item(tmp_path, "a")
state = review.initial_state(2, [first, _item(tmp_path, "b")])
window, recorder = _window(state)
first.path.unlink()
player = MagicMock()
window.attach_player(player)
recorder.missing.assert_called_once()
assert window.state.current is not None
assert window.state.current.md5 == "b-md5"
def test_tick_drains_events_and_reschedules(toolkit: MagicMock) -> None:
from python_pkg.wsg_grabber.models import Indexed
events: queue.SimpleQueue[DownloadEvent] = queue.SimpleQueue()
events.put(Indexed(known=42))
window, _ = _window(review.initial_state(0, []), events)
window.tick()
assert window.state.indexed == 42
_mock(window.widgets.root).after.assert_called_once()
assert _mock(window.widgets.root).after.call_args.args[1] == window.tick
def test_drain_on_an_empty_queue_is_harmless(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
window.drain()
assert window.state.indexed == 0
def test_shutdown_cancels_the_tick_and_closes_the_player(
toolkit: MagicMock,
) -> None:
"""A leaked after-callback or an orphan mpv would outlive the window."""
window, recorder = _window(review.initial_state(0, []))
player = MagicMock()
window.attach_player(player)
window.tick()
window.shutdown()
_mock(window.widgets.root).after_cancel.assert_called_once()
player.close.assert_called_once()
recorder.shutdown.assert_called_once()
_mock(window.widgets.root).destroy.assert_called_once()
def test_shutdown_without_a_pending_tick(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
window.shutdown()
_mock(window.widgets.root).after_cancel.assert_not_called()
def test_video_wid_realises_the_window_first(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
_mock(window.widgets.video).winfo_id.return_value = 4321
assert window.video_wid() == 4321
_mock(window.widgets.root).update.assert_called()
def test_run_starts_ticking_then_enters_the_loop(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
window.run()
_mock(window.widgets.root).mainloop.assert_called_once()
_mock(window.widgets.root).after.assert_called_once()
def test_undo_takes_back_the_last_verdict(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
"""The misclick case: passed something you meant to keep."""
state = review.initial_state(2, [_item(tmp_path, "a"), _item(tmp_path, "b")])
window, recorder = _window(state)
player = MagicMock()
window.attach_player(player)
window.on_pass()
assert window.state.current is not None
assert window.state.current.md5 == "b-md5"
assert window.state.passed == 1
window.on_undo()
recorder.undo.assert_called_once()
assert window.state.current is not None
assert window.state.current.md5 == "a-md5"
assert window.state.passed == 0
# and the video that was on screen is not lost
assert [item.md5 for item in window.state.pending] == ["b-md5"]
def test_the_undo_shortcut_is_wired(toolkit: MagicMock, tmp_path: Path) -> None:
state = review.initial_state(1, [_item(tmp_path)])
window, recorder = _window(state)
window.attach_player(MagicMock())
window.on_keep()
bound = _mock(window.widgets.root).bind.call_args_list
handlers = {call.args[0]: call.args[1] for call in bound}
handlers[UNDO_KEYS[0]](object())
recorder.undo.assert_called_once()
def test_the_undo_button_is_disabled_until_there_is_something_to_undo(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
state = review.initial_state(1, [_item(tmp_path)])
window, _ = _window(state)
window.attach_player(MagicMock())
assert _mock(window.widgets.undo).configure.call_args.kwargs["state"] == "disabled"
window.on_keep()
assert _mock(window.widgets.undo).configure.call_args.kwargs["state"] == "normal"

126
python_pkg/wsg_grabber/tests/test_verdict.py Normal file
View File

@ -0,0 +1,126 @@
"""Tests for verdict planning and the one module that moves files."""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import files, verdict
from python_pkg.wsg_grabber.models import FileMove, ReviewItem, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from pathlib import Path
def _item(path: Path) -> ReviewItem:
"""Build a ReviewItem pointing at *path*.
Args:
path: Location of the video.
Returns:
ReviewItem: Fixture value.
"""
return ReviewItem(
md5="md5",
path=path,
orig_name="clip.webm",
fsize=10,
width=4,
height=3,
)
def test_verdicts_map_to_terminal_states() -> None:
assert verdict.target_state(Verdict.KEEP) is FileState.KEPT
assert verdict.target_state(Verdict.SKIP) is FileState.PASSED
def test_verdicts_map_to_directories(tmp_path: Path) -> None:
keep, trash = tmp_path / "keep", tmp_path / "trash"
assert verdict.target_dir(Verdict.KEEP, keep, trash) == keep
assert verdict.target_dir(Verdict.SKIP, keep, trash) == trash
def test_unique_destination_passes_through_when_free(tmp_path: Path) -> None:
assert verdict.unique_destination(tmp_path, "a.webm", set()) == tmp_path / "a.webm"
def test_unique_destination_suffixes_a_collision(tmp_path: Path) -> None:
result = verdict.unique_destination(tmp_path, "a.webm", {"a.webm"})
assert result == tmp_path / "a-2.webm"
def test_unique_destination_keeps_counting(tmp_path: Path) -> None:
taken = {"a.webm", "a-2.webm", "a-3.webm"}
assert (
verdict.unique_destination(tmp_path, "a.webm", taken) == tmp_path / "a-4.webm"
)
def test_unique_destination_handles_a_name_without_an_extension(
tmp_path: Path,
) -> None:
assert verdict.unique_destination(tmp_path, "a", {"a"}) == tmp_path / "a-2"
def test_plan_move_sends_a_keep_to_the_keep_directory(tmp_path: Path) -> None:
keep, trash = tmp_path / "keep", tmp_path / "trash"
move = verdict.plan_move(
_item(tmp_path / "in" / "x.webm"), Verdict.KEEP, keep, trash
)
assert move.dst == keep / "x.webm"
assert move.src == tmp_path / "in" / "x.webm"
assert move.md5 == "md5"
def test_plan_move_sends_a_pass_to_the_trash_directory(tmp_path: Path) -> None:
keep, trash = tmp_path / "keep", tmp_path / "trash"
move = verdict.plan_move(
_item(tmp_path / "in" / "x.webm"), Verdict.SKIP, keep, trash
)
assert move.dst == trash / "x.webm"
def test_apply_move_relocates_rather_than_deleting(tmp_path: Path) -> None:
"""A pass must never destroy the file; trash is a destination, not a bin."""
source = tmp_path / "in" / "x.webm"
source.parent.mkdir()
source.write_bytes(b"data")
move = FileMove(md5="md5", src=source, dst=tmp_path / "trash" / "x.webm")
assert files.apply_move(move)
assert not source.exists()
assert move.dst.read_bytes() == b"data"
def test_apply_move_reports_a_source_that_has_vanished(tmp_path: Path) -> None:
move = FileMove(md5="md5", src=tmp_path / "gone.webm", dst=tmp_path / "t.webm")
assert not files.apply_move(move)
def test_existing_names_lists_the_directory(tmp_path: Path) -> None:
# a subdirectory, because the isolation fixture puts a fake home in tmp_path
keep = tmp_path / "keep"
keep.mkdir()
(keep / "a.webm").write_bytes(b"")
(keep / "b.webm").write_bytes(b"")
assert files.existing_names(keep) == {"a.webm", "b.webm"}
def test_existing_names_of_a_missing_directory_is_empty(tmp_path: Path) -> None:
assert files.existing_names(tmp_path / "nope") == set()
def test_apply_move_refuses_to_clobber_the_destination(tmp_path: Path) -> None:
"""The structural backstop: no move may ever destroy an existing file."""
source = tmp_path / "in" / "x.webm"
source.parent.mkdir()
source.write_bytes(b"new")
victim = tmp_path / "keep" / "x.webm"
victim.parent.mkdir()
victim.write_bytes(b"ALREADY HERE")
assert not files.apply_move(FileMove(md5="m", src=source, dst=victim))
assert victim.read_bytes() == b"ALREADY HERE"
assert source.read_bytes() == b"new"

398
python_pkg/wsg_grabber/ui.py Normal file
View File

@ -0,0 +1,398 @@
"""The Tk window: video on top, keep/pass bar underneath.
The only module in this package that imports ``tkinter``. Almost every callback
hands off to :mod:`python_pkg.wsg_grabber.review`, gets a
:class:`ReviewCommand` back and applies it verbatim, so the behaviour is
specified and covered without a display ever being opened.
Two decisions do live here, both because a bug proved they had to: ``_play``
skips a reload when the file is already playing, and drops an item whose file
has vanished. Everything else is wiring.
The control bar sits *below* the video rather than over it: with ``--wid`` mpv
reparents its own child window onto the frame and covers it completely, so a
widget drawn on top would be hidden.
Nothing here tracks the window size. mpv resizes itself to fill the frame it was
given, and telling it otherwise via ``geometry`` leaves the video drawn at the
wrong size -- which is exactly what happened the first time this ran.
"""
from __future__ import annotations
from dataclasses import dataclass
import tkinter as tk
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import logs, review
from python_pkg.wsg_grabber.constants import (
CONTROL_BAR_HEIGHT_PX,
KEEP_KEYS,
PASS_KEYS,
POLL_INTERVAL_MS,
QUIT_KEYS,
UNDO_KEYS,
WINDOW_HEIGHT_PX,
WINDOW_WIDTH_PX,
)
from python_pkg.wsg_grabber.models import ReviewCommand, Verdict
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
import queue
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.player import Player
from python_pkg.wsg_grabber.review import SessionState
_BACKGROUND = "#1B1D21"
_ACCENT = "#B8862E"
_TEXT = "#E8E6E3"
@dataclass(frozen=True, slots=True)
class Callbacks:
"""What the window asks the rest of the application to do."""
commit: Callable[[SessionState, Verdict], SessionState]
missing: Callable[[SessionState], SessionState]
undo: Callable[[SessionState], SessionState]
shutdown: Callable[[], None]
@dataclass
class Widgets:
"""The widgets, grouped so the window stays within the attribute limit."""
root: tk.Tk
video: tk.Frame
status: tk.Label
filename: tk.Label
keep: tk.Button
skip: tk.Button
undo: tk.Button
class ReviewWindow:
"""Presents downloaded videos one at a time for a keep/pass decision."""
def __init__(
self,
state: SessionState,
events: queue.SimpleQueue[DownloadEvent],
callbacks: Callbacks,
) -> None:
"""Build the window.
The player is attached afterwards with :meth:`attach_player`, because
it needs the X11 id of a frame that must exist first.
Args:
state: Starting session state.
events: Queue the download worker publishes to.
callbacks: Hooks back into the application.
"""
self._state = state
self._events = events
self._callbacks = callbacks
self._player: Player | None = None
self._after_id: str | None = None
self._playing: Path | None = None
self._widgets = self._build()
self._bind_keys()
@property
def widgets(self) -> Widgets:
"""Return the constructed widgets.
Returns:
Widgets: The window's parts.
"""
return self._widgets
@property
def state(self) -> SessionState:
"""Return the current session state.
Returns:
SessionState: What the window is showing.
"""
return self._state
def _build(self) -> Widgets:
"""Construct the window and its widgets.
Returns:
Widgets: The constructed parts.
"""
root = tk.Tk()
root.title("/wsg/")
root.geometry(f"{WINDOW_WIDTH_PX}x{WINDOW_HEIGHT_PX}")
root.configure(bg=_BACKGROUND)
video = tk.Frame(root, bg="black")
video.pack(fill=tk.BOTH, expand=True)
controls = tk.Frame(root, bg=_BACKGROUND, height=CONTROL_BAR_HEIGHT_PX)
controls.pack(fill=tk.X, side=tk.BOTTOM)
keep = tk.Button(
controls,
text="✔ KEEP (k)",
bg=_ACCENT,
fg=_BACKGROUND,
command=self.on_keep,
)
keep.pack(side=tk.LEFT, padx=16, pady=10)
skip = tk.Button(
controls,
text="✘ PASS (j)",
bg=_BACKGROUND,
fg=_TEXT,
command=self.on_pass,
)
skip.pack(side=tk.RIGHT, padx=16, pady=10)
undo = tk.Button(
controls,
text="↶ UNDO (u)",
bg=_BACKGROUND,
fg=_TEXT,
command=self.on_undo,
)
undo.pack(side=tk.LEFT, padx=8, pady=10)
status = tk.Label(controls, text="", bg=_BACKGROUND, fg=_ACCENT)
status.pack()
filename = tk.Label(controls, text="", bg=_BACKGROUND, fg=_TEXT)
filename.pack()
root.protocol("WM_DELETE_WINDOW", self.on_quit)
return Widgets(
root=root,
video=video,
status=status,
filename=filename,
keep=keep,
skip=skip,
undo=undo,
)
def _bind_keys(self) -> None:
"""Bind the keyboard shortcuts onto the toplevel."""
for sequence in KEEP_KEYS:
self._widgets.root.bind(sequence, self._key_keep)
for sequence in PASS_KEYS:
self._widgets.root.bind(sequence, self._key_pass)
for sequence in UNDO_KEYS:
self._widgets.root.bind(sequence, self._key_undo)
for sequence in QUIT_KEYS:
self._widgets.root.bind(sequence, self._key_quit)
def video_wid(self) -> int:
"""Return the X11 id of the video frame.
Returns:
int: Window id to hand to mpv's ``--wid``.
"""
self._widgets.root.update()
return int(self._widgets.video.winfo_id())
def attach_player(self, player: Player) -> None:
"""Adopt a player and show the first video.
Args:
player: The video player to drive.
"""
self._player = player
self.refresh()
def _key_keep(self, _event: object) -> None:
"""Handle a keep keypress.
Args:
_event: Tk event, unused.
"""
self.on_keep()
def _key_pass(self, _event: object) -> None:
"""Handle a pass keypress.
Args:
_event: Tk event, unused.
"""
self.on_pass()
def _key_undo(self, _event: object) -> None:
"""Handle an undo keypress.
Args:
_event: Tk event, unused.
"""
self.on_undo()
def _key_quit(self, _event: object) -> None:
"""Handle a quit keypress.
Args:
_event: Tk event, unused.
"""
self.on_quit()
def on_keep(self) -> None:
"""Keep the current video."""
self._verdict(Verdict.KEEP)
def on_pass(self) -> None:
"""Pass on the current video."""
self._verdict(Verdict.SKIP)
def _verdict(self, choice: Verdict) -> None:
"""Apply a verdict and redraw.
Args:
choice: What the user decided.
"""
before = self._state.current
self._state = self._callbacks.commit(self._state, choice)
after = self._state.current
logs.event(
"review.verdict",
choice=choice.value,
on=before.orig_name if before else None,
on_file=before.path.name if before else None,
next_file=after.path.name if after else None,
queued=len(self._state.pending),
kept=self._state.kept,
passed=self._state.passed,
)
self.refresh()
def on_undo(self) -> None:
"""Take back the last verdict and show that video again."""
before = self._state.current
self._state = self._callbacks.undo(self._state)
after = self._state.current
logs.event(
"review.undo",
was_showing=before.orig_name if before else None,
now_showing=after.orig_name if after else None,
kept=self._state.kept,
passed=self._state.passed,
remaining_undos=self._state.undoable,
)
self.refresh()
def on_quit(self) -> None:
"""Begin shutting down."""
self._state = review.on_quit(self._state)
self.refresh()
def tick(self) -> None:
"""Drain the worker's events, redraw, and schedule the next tick."""
self.drain()
self.refresh()
self._after_id = self._widgets.root.after(POLL_INTERVAL_MS, self.tick)
def drain(self) -> None:
"""Fold every queued worker event into the session state."""
while not self._events.empty():
self._state = review.on_event(self._state, self._events.get_nowait())
def refresh(self) -> None:
"""Re-render from the current state."""
self.apply(review.render(self._state))
def apply(self, command: ReviewCommand) -> None:
"""Carry out a rendered command.
Args:
command: What to show and do.
"""
if command.play is not None:
self._play(command.play)
if command.stop and self._player is not None and self._playing is not None:
self._playing = None
self._player.stop()
self._widgets.status.configure(text=command.status)
self._widgets.filename.configure(text=command.filename)
self._widgets.root.title(command.title)
if command.verdicts_enabled:
self._widgets.keep.configure(state="normal")
self._widgets.skip.configure(state="normal")
else:
self._widgets.keep.configure(state="disabled")
self._widgets.skip.configure(state="disabled")
if command.undo_enabled:
self._widgets.undo.configure(state="normal")
else:
self._widgets.undo.configure(state="disabled")
if command.quit_app:
self.shutdown()
def _play(self, path: Path) -> None:
"""Show a video, unless it is already the one playing.
The already-playing guard is essential rather than an optimisation:
``refresh`` runs on every tick, so without it ``loadfile`` would be sent
several times a second and the video would restart continuously instead
of ever playing.
A file can also vanish between being downloaded and being reviewed.
Handing mpv a dead path would leave the reviewer stuck on a black
frame, so the item is dropped and the next one shown instead.
Args:
path: Video to play.
"""
if self._player is None:
logs.debug("review.play_skipped", reason="no player", file=path.name)
return
if path == self._playing:
logs.debug("review.play_skipped", reason="already playing", file=path.name)
return
if not path.exists():
logs.warning("review.file_missing", file=path.name, path=str(path))
self._state = self._callbacks.missing(self._state)
self.refresh()
return
# Recorded BEFORE the new load, so a frozen-looking session shows what
# mpv was doing with the file it was supposedly already playing.
logs.event(
"review.show",
file=path.name,
previous=self._playing.name if self._playing else None,
player_alive=self._player.is_alive(),
mpv_before=self._player.probe(),
)
self._playing = path
self._player.play(path)
def shutdown(self) -> None:
"""Cancel the tick, stop the worker and destroy the window."""
logs.event(
"review.shutdown",
kept=self._state.kept,
passed=self._state.passed,
queued=len(self._state.pending),
)
try:
if self._after_id is not None:
self._widgets.root.after_cancel(self._after_id)
self._after_id = None
self._callbacks.shutdown()
player, self._player = self._player, None
if player is not None:
player.close()
finally:
# Whatever failed above -- a Ctrl-C landing in the worker join, an
# mpv that will not die -- the window must not be left up and
# frozen with its tick already cancelled.
self._widgets.root.destroy()
def run(self) -> None:
"""Start ticking and hand control to Tk."""
self.tick()
self._widgets.root.mainloop()

99
python_pkg/wsg_grabber/verdict.py Normal file
View File

@ -0,0 +1,99 @@
"""What a keep/pass decision means, expressed as data.
Planning a move and performing it are split deliberately: everything here is
pure and exhaustively testable, and :mod:`python_pkg.wsg_grabber.files` is the
only code that touches the filesystem.
Nothing in this package deletes a video. A pass relocates it to ``trash/`` and
leaves clearing that directory to the user.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import FileMove, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Container
from pathlib import Path
from python_pkg.wsg_grabber.models import ReviewItem
def target_state(choice: Verdict) -> FileState:
"""Return the terminal state a verdict leads to.
Args:
choice: The user's decision.
Returns:
FileState: ``KEPT`` or ``PASSED``.
"""
return FileState.KEPT if choice is Verdict.KEEP else FileState.PASSED
def target_dir(choice: Verdict, keep: Path, trash: Path) -> Path:
"""Return the directory a verdict sends a file to.
Args:
choice: The user's decision.
keep: Directory for kept videos.
trash: Directory for passed videos.
Returns:
Path: Destination directory.
"""
return keep if choice is Verdict.KEEP else trash
def unique_destination(directory: Path, name: str, taken: Container[str]) -> Path:
"""Return a collision-free destination inside *directory*.
``taken`` is supplied by the caller rather than read from disk, keeping this
function pure and its collision handling directly testable.
Args:
directory: Destination directory.
name: Preferred filename.
taken: Names already present.
Returns:
Path: A path whose name is not in *taken*.
"""
if name not in taken:
return directory / name
stem, _, suffix = name.rpartition(".")
base, ext = (stem, f".{suffix}") if stem else (name, "")
index = 2
while f"{base}-{index}{ext}" in taken:
index += 1
return directory / f"{base}-{index}{ext}"
def plan_move(
item: ReviewItem,
choice: Verdict,
keep: Path,
trash: Path,
taken: Container[str] = (),
) -> FileMove:
"""Work out where a reviewed video should end up.
Args:
item: The video under review.
choice: The user's decision.
keep: Directory for kept videos.
trash: Directory for passed videos.
taken: Names already used in the destination directory.
Returns:
FileMove: Source, destination and the file's identity.
"""
destination = target_dir(choice, keep, trash)
return FileMove(
md5=item.md5,
src=item.path,
dst=unique_destination(destination, item.path.name, taken),
)