kuhyx/testsAndMisc
1
0
Fork 0
mirror of https://github.com/kuhyx/testsAndMisc.git synced 2026-11-10 18:47:04 +01:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

Remove wsg_grabber: extracted to kuhyx/wsg-grabber; bump hypothesis
Some checks failed
dependency freshness / dependencies on newest stable (push) Has been cancelled
Details
md-naming / markdown naming convention (push) Has been cancelled
Details
Pre-commit checks / pre-commit (${{ matrix.group }}) (mypy, mypy) (push) Has been cancelled
Details
Pre-commit checks / pre-commit (${{ matrix.group }}) (pylint, pylint) (push) Has been cancelled
Details
Pre-commit checks / pre-commit (${{ matrix.group }}) (rest, ) (push) Has been cancelled
Details
Pre-commit checks / pre-commit (${{ matrix.group }}) (shellcheck, shellcheck) (push) Has been cancelled
Details
Python tests / test (push) Has been cancelled
Details
Shell tests / shell-tests (push) Has been cancelled
Details
Shell tests / lib-tests (push) Has been cancelled
Details
Shell tests / arch-tests (push) Has been cancelled
Details

Browse Source 
The package now lives in its own private repo at ~/src/wsg-grabber, with
the 14 commits that touched it. /usr/local/bin/wsg-grabber was regenerated
by that repo's install.sh, so nothing here runs it any more.

hypothesis 6.168.2 -> 6.168.3: an upstream release turned the Pre-commit
checks and dependency freshness jobs red on 33f0fee3.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F8G3BNETGmbr6aM5cc18g4
This commit is contained in:
Krzysztof Rudnicki 2026-09-28 12:44:45 +02:00
parent 33f0fee342
commit efe343dac4
83 changed files with 56 additions and 11686 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
AGENTS.md
View File

@ -41,6 +41,8 @@ Extracted to their own repos: [`steam-backlog-enforcer`](https://github.com/kuhy
`~/android-guardian`),
[`phone-focus-mode`](https://github.com/kuhyx/phone-focus-mode) (with
`python_pkg/focus_policy`),
[`wsg-grabber`](https://github.com/kuhyx/wsg-grabber) (private; lives at
`~/src/wsg-grabber`),
[`hosts-blocker`](https://github.com/kuhyx/hosts-blocker) (lives at
`~/src/hosts-blocker`),
[`digital-wellbeing`](https://github.com/kuhyx/digital-wellbeing) (lives at

16
docs/superpowers/contracts/remove-wsg-grabber-2026-09-28.json Normal file
View File

@ -0,0 +1,16 @@
{
"title": "Remove wsg_grabber from the monorepo after extracting it",
"objective": "wsg_grabber lives only in kuhyx/wsg-grabber (private, ~/src/wsg-grabber); nothing in testsAndMisc still carries or runs it, and main's CI is green.",
"acceptance_criteria": [
"python_pkg/wsg_grabber/ no longer exists in the tree.",
"kuhyx/wsg-grabber holds the package's history (14 commits) plus the standalone setup, and its suite passes at 100% branch coverage.",
"/usr/local/bin/wsg-grabber runs from ~/src/wsg-grabber and `wsg-grabber stats` reads the live index.",
"AGENTS.md lists wsg-grabber among the extracted repos.",
"Pre-commit checks and dependency freshness are green on the pushed commit."
],
"out_of_scope": [
"The metadata, duplicate-detection and report features; they are built in the new repo.",
"Rewriting historical docs that mention the old path."
],
"verifier": "ls python_pkg/wsg_grabber fails; git -C ~/src/wsg-grabber log --oneline | wc -l; cat /usr/local/bin/wsg-grabber; wsg-grabber stats; gh run list --branch main"
}

37
docs/superpowers/evidence/remove-wsg-grabber-2026-09-28.json Normal file
View File

@ -0,0 +1,37 @@
{
"intent": "Remove python_pkg/wsg_grabber from the monorepo now that it lives in its own private repo, kuhyx/wsg-grabber, with its full history; and turn main's CI green again (hypothesis released 6.168.3).",
"scope": [
"python_pkg/wsg_grabber/ deleted (package, tests, README, install.sh)",
"AGENTS.md (CLAUDE.md symlink): wsg-grabber added to the extracted-repos list",
"meta/requirements.txt: hypothesis 6.168.2 -> 6.168.3, the only item the Pre-commit checks and dependency freshness jobs reported behind on 33f0fee3",
"Non-goal: the historical mentions in docs/DOCS-python-split-recipes.md and docs/DOCS-restructure-live-state.md stay; they record what happened here"
],
"changes": [
"git filter-repo kept the 14 commits that touched python_pkg/wsg_grabber; the standalone setup is kuhyx/wsg-grabber 6b6d4b3",
"/usr/local/bin/wsg-grabber regenerated by ~/src/wsg-grabber/install.sh, so the launcher no longer depends on this repo"
],
"verification": [
{
"command": "~/src/wsg-grabber: .venv/bin/python -m pytest -p no:cacheprovider -q",
"result": "pass",
"evidence": "303 passed, TOTAL 1645 stmts / 342 branches, 100.00% coverage; pre-commit run --all-files clean there"
},
{
"command": "vm run wsg 'sudo bash ~/install.sh' then on the host: sudo bash ~/src/wsg-grabber/install.sh && wsg-grabber stats",
"result": "pass",
"evidence": "vmbox: packages installed, launcher written with PYTHONPATH=/home/arch, --uninstall removed it. Host: launcher PYTHONPATH=/home/kuhy/src/wsg-grabber; stats reads the live index (26898 known, 706 kept, 200 passed, 25992 purged)"
},
{
"command": "gh run view 36399404250 --log-failed",
"result": "pass",
"evidence": "the only freshness finding was pypi:hypothesis 6.168.2 -> 6.168.3 at meta/requirements.txt:42, bumped here"
}
],
"risks": [
"Anything outside git that pointed at ~/src/testsAndMisc/python_pkg/wsg_grabber would break; the only such artefact found was /usr/local/bin/wsg-grabber, already repointed."
],
"rollback": [
"git revert this commit to restore the package, then rerun its install.sh to point the launcher back here",
"Validate with `wsg-grabber stats`"
]
}

44
docs/superpowers/evidence/wire-shell-coverage-ratchet-2026-08-23.json
View File

@ -1,44 +0,0 @@
{
"intent": "Wire check_shell_coverage.sh into pre-commit so the shell ratchet cannot silently rot, and give the coverage tooling itself the suite it was missing.",
"scope": [
".pre-commit-config.yaml (new shell-coverage-ratchet hook)",
"meta/scripts/check_shell_coverage.sh (stale message referencing a removed constant)",
"meta/scripts/lib/tests/ (new suite: harness + 3 test files + run_all.sh)"
],
"changes": [
"Added the shell-coverage-ratchet hook, types: [shell], excluding third_party/. It was previously never wired, so a new untested shell library could land freely.",
"FIX: check_shell_coverage.sh line 177 interpolated ${COVERAGE_BAR}, which no longer exists after 578c4a6c, producing 'COVERAGE_BAR: unbound variable' on every failure path. The message now describes the presence bar it actually enforces.",
"New suite for the three meta/scripts/lib coverage-tooling libs, which are the one corner of the repo the jail cannot measure itself: 48 assertions over is_covered, build_jail, write_cases, build_mount_script, reconcile_pass_failures and write_case_runner.",
"Assertions on unexpanded ${BASH_SOURCE}/$1 literals compose the dollar sign from an exported DOLLAR constant rather than writing single-quoted literals, so the files are SC2016-clean with no suppressions."
],
"verification": [
{
"command": "bash meta/scripts/lib/tests/run_all.sh",
"result": "pass",
"evidence": "48 assertions across 3 files, 0 failed (5 + 26 + 17)"
},
{
"command": "pre-commit run shell-coverage-ratchet --all-files",
"result": "pass",
"evidence": "Passed in 0.16s; check_shell_coverage.sh --all now reports 0 BLOCK entries, down from 3, and runs in 0.31s"
},
{
"command": "Created an untested lib at mtk_root/lib/zz_probe.sh and ran the hook on it",
"result": "pass",
"evidence": "Hook failed with 'needs: linux_configuration/scripts/mtk_root/lib/tests/run_all.sh'. Probe file removed; the ratchet is not toothless."
},
{
"command": "shellcheck -x and shfmt -l over the new files",
"result": "pass",
"evidence": "clean, no noqa/disable directives added"
}
],
"risks": [
"The hook now runs on every commit touching a shell file. Measured at 0.16s repo-wide, so the cost is negligible.",
"A lib added to a directory that has no suite is now blocked at commit time. That is the intended ratchet behaviour, and all 8 such directories are already allowlisted."
],
"rollback": [
"Remove the shell-coverage-ratchet block from .pre-commit-config.yaml; the gate returns to being unenforced.",
"After rollback, confirm `pre-commit run --all-files` no longer lists the hook."
]
}

2
meta/requirements.txt
View File

@ -39,7 +39,7 @@ gatelock @ git+https://github.com/kuhyx/utils@gatelock-v0.8.2#subdirectory=gatel
genanki==0.13.1
geopandas==1.1.4
howlongtobeatpy==1.0.23
hypothesis==6.168.2
hypothesis==6.168.3
importlib-metadata==9.0.1
interrogate==1.7.0
# isort is deliberately not pinned: nothing here imports it (ruff's isort

199
python_pkg/wsg_grabber/README.md
View File

@ -1,199 +0,0 @@
# wsg_grabber
Scrapes 4chan's /wsg/ board for videos, downloads every one it has not seen
before, and shows them one at a time for a keep/pass decision — mpv embedded in
a Tk window with the buttons directly underneath.
```bash
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber # review
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber scrape # no UI
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber stats
```
`k` / `→` keeps, `j` / `space` / `←` passes, `u` / `Backspace` undoes, `q` /
`Esc` quits. The buttons do the same thing for the mouse.
## Undo
Misclicked? `u` takes the last verdict back: the file returns from the keep
folder or `trash/` to `incoming/`, the index row goes back to _ready_, the
counter is corrected, and that video is on screen again. Whatever was showing
when you hit undo goes to the front of the queue, so nothing is skipped.
It **survives quitting** — the trail lives in the index, not in memory, so
reopening the reviewer can still undo what you decided yesterday. The status
bar shows how many are left (`u undoes (17)`), and the button greys out when
there is nothing to take back. Keeps are undoable for as long as the file is
still in any place a keep can end up (`housekeeping.locate_reviewed` looks in
this year's cloud folder, earlier years', `~/Downloads`, the old `keep/`, and
wherever the media sync swept it); passes are undoable while they are among
the newest `TRASH_RETAIN` (200). If a file has gone anyway, undo says so in
the log and drops that entry rather than failing on it repeatedly.
## Storage
```
~/data/cloud/Media/<year>/wsg/ you kept these (falls back to ~/Downloads
on a machine without the cloud folder)
~/.local/share/wsg_grabber/
├── index.db sqlite, remembers every file ever seen
├── incoming/ downloaded, awaiting your verdict
└── trash/ you passed on these — only the newest 200 passes stay
```
A keep goes where you actually look: the dufs cloud folder, so it shows up in
the gallery and on the phone. The first start after this change moved the old
`keep/` contents across (`_migrate_keep.py`).
A pass goes into `trash/`, and `_prune.py` deletes everything there except the
200 most recent passes — at startup and after every pass. Their index rows
become _purged_: still known, so the file is never downloaded again, but no
longer undoable. Before this cap existed 23k files / 77 GB had piled up.
`files.apply_move` still refuses to overwrite an existing destination, so a
move can never destroy a video you already decided on.
## Why it never shows you the same video twice
Every 4chan post that carries a file also carries `md5` — the base64 MD5 of the
whole file, published by the API before you fetch anything. That is the primary
key of the `files` table, so:
- a video you have already reviewed is never downloaded again, at a cost of
zero bytes;
- the same clip reposted in another thread collapses onto the existing row,
which on /wsg/ is the common case;
- a truncated or mangled download is caught, because the bytes are checked
against that same value before the file is offered for review.
Verified against a live file during development: the API's
`Nz9OEKdMuMZEdYE6eLTKmA==` matched `base64(md5(bytes))` exactly.
## Being a good citizen
- One request per second to `a.4cdn.org`, which is what the API rules ask for.
- `If-Modified-Since` on every API request — thread bodies, `threads.json` and
`archive.json` alike — plus a `last_modified` comparison against the stored
value, so an unchanged thread costs a 304 and nothing else. A steady-state
rescan is a couple of conditional requests that return 304.
- `Accept-Encoding: identity` on both hosts, and hard caps on response and
download size. A compressed response is refused outright: a few hundred KB on
the wire can decode to gigabytes, and the md5 check does not help when the
same party publishes both the bytes and the digest.
- One long-lived `requests.Session`. This is required rather than merely tidy:
`i.4cdn.org` answers with 429 under load and sets `__cf_bm` / `_cfuvid`
cookies that have to persist across requests.
- 429 and 5xx back off exponentially, honouring `Retry-After`.
## Downloading is prioritised over indexing
While anything is queued the worker downloads rather than walking more threads.
That is deliberate: it is what puts a playable video on screen seconds after
launch instead of after the whole board has been catalogued. The consequence is
that `stats` shows the known-file count climbing in bursts rather than smoothly
— the board is fully indexed once downloads drain. A first full run is roughly
9,000 files and 20–25 GB, so expect a couple of hours; it is resumable and
restarting picks up mid-file via HTTP `Range`.
## Logs
Every run writes one JSON object per line to
`~/.local/share/wsg_grabber/logs/session-<time>-<pid>.jsonl`.
```bash
wsg-grabber logs # where they are, plus a tally of the newest
wsg-grabber --log-level debug # record every IPC command as well
wsg-grabber --echo-log # mirror records to stderr while running
```
The point is after-the-fact diagnosis. If the reviewer ever looks stuck on one
video while the filenames keep advancing, one query settles whether mpv was
handed the new file and was actually decoding it:
```bash
LOG=$(ls -t ~/.local/share/wsg_grabber/logs/*.jsonl | head -1)
jq -r 'select(.event=="review.show")
| "show \(.file) mpv_was: \(.mpv_before.path) t=\(.mpv_before["playback-time"]) \(.mpv_before.width)x\(.mpv_before.height)"' "$LOG"
```
A healthy run shows `playback-time` advancing and the dimensions changing per
video. A frozen one shows them static, which points at the player rather than
at the queue. Other useful queries:
```bash
jq -c 'select(.level=="error")' "$LOG" # what went wrong
jq -r 'select(.event=="review.verdict")|"\(.choice) \(.on_file)"' "$LOG"
jq -c 'select(.event|startswith("player."))' "$LOG"
```
Field names are stable; every record carries `ts`, `seq`, `level`, `event` and
`thread`.
## Debugging without touching your queue
`paths` honours `XDG_DATA_HOME`, so a throwaway instance costs nothing and can
neither consume real videos nor record real verdicts:
```bash
XDG_DATA_HOME=/tmp/wsg-sandbox python3 -m python_pkg.wsg_grabber stats
```
## Layout
| Module | Responsibility |
| ------------------- | ----------------------------------------------------------------- |
| `catalog.py` | Parses the board's JSON. Pure. |
| `scanner.py` | Scheduling, backoff, resume decisions. Pure. |
| `review.py` | Everything the reviewer does: queue, verdicts, status text. Pure. |
| `verdict.py` | What keep/pass means, as a planned file move. Pure. |
| `net.py` | The only module that speaks HTTP. |
| `db.py` | sqlite schema, connection setup, migrations. |
| `store.py` | The download lifecycle of a file. |
| `store_threads.py` | Which threads were visited, and their HTTP stamps. |
| `store_verdicts.py` | The review trail that undo walks back. |
| `downloader.py` | The background worker thread. |
| `files.py` | The only module that moves or deletes files. |
| `housekeeping.py` | Around a verdict: keep migration, trash prune, finding a keep. |
| `_migrate_keep.py` | One-off move of the old `keep/` into the cloud folder. |
| `_prune.py` | Caps `trash/` at the newest 200 passes. |
| `player.py` | mpv as a subprocess over its JSON IPC socket. |
| `ui.py` | The only module that imports `tkinter`. |
| `logs.py` | Structured JSONL event log. |
| `app.py` / `cli.py` | Composition and argument parsing. |
The split is what makes 100% branch coverage achievable: `ui.py` and
`player.py` are close to pure wiring, and every decision they would otherwise
make lives in a pure function tested from plain values with no display and no
network. Two decisions do remain in `ui.py`, both because a bug proved they had
to: the already-playing guard (without it `loadfile` fires every tick and the
video restarts continuously) and dropping an item whose file has vanished.
## mpv is a subprocess, not `python-mpv`
`python-mpv` resolves `libmpv` at _import_ time and raises `OSError` wherever it
is missing — including this repo's CI runner, so it would pass locally and fail
on push. Driving `mpv --wid=<frame> --input-ipc-server=<socket>` over a unix
socket needs only `subprocess`, `socket` and `json`.
**The socket must be drained.** mpv pushes an asynchronous event stream plus a
reply per command down the same connection. A client that only ever writes lets
that backlog fill the kernel buffer, at which point mpv's writer for that
connection blocks and it stops reading commands. Nothing errors: `sendall` still
succeeds, mpv stays alive and keeps playing whatever it had, so the reviewer
advances through filenames while the picture never changes. Measured: commands
were silently ignored from about the thirty-fifth video onwards, and the very
same command sent on a fresh connection worked instantly. `player.py` therefore
runs a reader thread whose only job is to drain — deleting it reintroduces the
freeze roughly half an hour into a session.
Two more mpv details worth knowing if you touch `ui.py`:
- `--wid` does not draw into your widget. mpv creates its own X window and
reparents it onto the frame, covering it completely — which is why the
control bar is packed _below_ the video rather than over it.
- `--input-vo-keyboard=no` is what lets the Tk key bindings fire at all. Without
it mpv's child window takes keyboard focus and Tk never sees a keypress.
## Requirements
`mpv` and `tk` (system packages, installed by `install.sh`) and `requests`,
which the repo already declares. Nothing else.

13
python_pkg/wsg_grabber/__init__.py
View File

@ -1,13 +0,0 @@
"""Scrape 4chan's /wsg/ board and triage its videos with a keep/pass reviewer.
The board exposes a read-only JSON API whose post records carry ``md5`` --
``base64(md5(file_bytes))`` over the whole file. That single field is why this
package can honour "never show me the same video twice" without spending a byte:
dedupe happens against the catalogue, before any download starts, and reposts of
an identical file across threads collapse onto one row for free.
Nothing here ever deletes a file. A "pass" verdict moves the video to
``trash/`` and leaves it for the user to clear by hand.
"""
from __future__ import annotations

14
python_pkg/wsg_grabber/__main__.py
View File

@ -1,14 +0,0 @@
"""Module entry point.
Usage:
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber --help
"""
from __future__ import annotations
import sys
from python_pkg.wsg_grabber.cli import main
if __name__ == "__main__":
sys.exit(main())

87
python_pkg/wsg_grabber/_coerce.py
View File

@ -1,87 +0,0 @@
"""Defensive accessors for the loosely-typed /wsg/ JSON payloads.
The API omits file fields entirely on posts without an attachment and is fed by
an anonymous imageboard, so nothing here trusts the shape of what it is handed:
every accessor answers "not usable" rather than raising.
"""
from __future__ import annotations
import re
from typing import Any
# base64 of a 16-byte digest is always 22 characters plus "==". Matching the
# exact shape -- not merely the length -- is what keeps a NUL byte out of a
# filename: one such value poisons the index permanently, because the crash
# happens before the row can ever be marked gone.
_MD5_SHAPE = re.compile(r"[A-Za-z0-9+/]{22}==")
# sqlite stores signed 64-bit integers; anything wider raises OverflowError deep
# inside executemany, which used to take the worker thread down with it.
_INT64_MIN = -(2**63)
_INT64_MAX = 2**63 - 1
def get(container: object, key: str) -> object:
"""Read *key* from *container* when it is a mapping.
Args:
container: Anything; only dicts yield a value.
key: Key to read.
Returns:
object: The value, or None.
"""
if isinstance(container, dict):
return container.get(key)
return None
def as_list(value: object) -> list[Any]:
"""Return *value* when it is a list, else an empty list.
Args:
value: Candidate.
Returns:
list[Any]: Safe list to iterate.
"""
return value if isinstance(value, list) else []
def as_int(value: object) -> int | None:
"""Coerce *value* to int when it is a real integer.
Booleans are rejected: ``True`` is an int in Python but never a valid post
number or timestamp. Values outside sqlite's signed 64-bit range are
rejected too -- they raise OverflowError on insert, which is not in the
worker's recoverable set and would kill it silently.
Args:
value: Candidate.
Returns:
int | None: The integer, or None.
"""
if isinstance(value, bool) or not isinstance(value, int):
return None
return value if _INT64_MIN <= value <= _INT64_MAX else None
def as_md5(value: object) -> str | None:
"""Validate the API's base64 md5 field.
The charset check is not decoration: the tail of this value ends up in a
filename via ``store.local_name``, and it arrives from an anonymous
imageboard. Length alone would let through a NUL byte, which raises deep
inside the filesystem call instead of being rejected here.
Args:
value: Candidate.
Returns:
str | None: The 24-character digest, or None when malformed.
"""
if not isinstance(value, str):
return None
return value if _MD5_SHAPE.fullmatch(value) else None

179
python_pkg/wsg_grabber/_display.py
View File

@ -1,179 +0,0 @@
"""Applying a rendered command to the screen.
Split out of :mod:`python_pkg.wsg_grabber.ui` to keep it under the 250-line
cap. Note what is *not* here: creating widgets. ``ui`` is the only module that
imports ``tkinter``, and ``test_ui.py`` fakes the toolkit with
``patch.object(ui, "tk", ...)``, so construction has to stay there. These two
methods only ever reach widgets through ``self._widgets``, so they move freely.
A mixin rather than free functions because both read and write the window's
playback bookkeeping and ``apply`` is part of ReviewWindow's public surface.
"""
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import logs, review
from python_pkg.wsg_grabber.constants import POLL_INTERVAL_MS
if TYPE_CHECKING:
from pathlib import Path
import queue
from python_pkg.wsg_grabber._callbacks import Callbacks
from python_pkg.wsg_grabber.models import (
DownloadEvent,
ReviewCommand,
Verdict,
)
from python_pkg.wsg_grabber.player import Player
from python_pkg.wsg_grabber.review import SessionState
from python_pkg.wsg_grabber.ui import Widgets
class DisplayMixin(ABC):
"""The screen-updating half of :class:`~.ui.ReviewWindow`."""
_widgets: Widgets
_callbacks: Callbacks
_player: Player | None
_playing: Path | None
_state: SessionState
_after_id: str | None
_events: queue.SimpleQueue[DownloadEvent]
@abstractmethod
def shutdown(self) -> None:
"""Close the player and tear the window down."""
def apply(self, command: ReviewCommand) -> None:
"""Carry out a rendered command.
Args:
command: What to show and do.
"""
if command.play is not None:
self._play(command.play)
if command.stop and self._player is not None and self._playing is not None:
self._playing = None
self._player.stop()
self._widgets.status.configure(text=command.status)
self._widgets.filename.configure(text=command.filename)
self._widgets.root.title(command.title)
if command.verdicts_enabled:
self._widgets.keep.configure(state="normal")
self._widgets.skip.configure(state="normal")
else:
self._widgets.keep.configure(state="disabled")
self._widgets.skip.configure(state="disabled")
if command.undo_enabled:
self._widgets.undo.configure(state="normal")
else:
self._widgets.undo.configure(state="disabled")
if command.quit_app:
self.shutdown()
def _play(self, path: Path) -> None:
"""Show a video, unless it is already the one playing.
The already-playing guard is essential rather than an optimisation:
``refresh`` runs on every tick, so without it ``loadfile`` would be sent
several times a second and the video would restart continuously instead
of ever playing.
A file can also vanish between being downloaded and being reviewed.
Handing mpv a dead path would leave the reviewer stuck on a black
frame, so the item is dropped and the next one shown instead.
Args:
path: Video to play.
"""
if self._player is None:
logs.debug("review.play_skipped", reason="no player", file=path.name)
return
if path == self._playing:
logs.debug("review.play_skipped", reason="already playing", file=path.name)
return
if not path.exists():
logs.warning("review.file_missing", file=path.name, path=str(path))
self._state = self._callbacks.missing(self._state)
self.refresh()
return
# Recorded BEFORE the new load, so a frozen-looking session shows what
# mpv was doing with the file it was supposedly already playing.
logs.event(
"review.show",
file=path.name,
previous=self._playing.name if self._playing else None,
player_alive=self._player.is_alive(),
mpv_before=self._player.probe(),
)
self._playing = path
self._player.play(path)
def _verdict(self, choice: Verdict) -> None:
"""Apply a verdict and redraw.
Args:
choice: What the user decided.
"""
before = self._state.current
self._state = self._callbacks.commit(self._state, choice)
after = self._state.current
logs.event(
"review.verdict",
choice=choice.value,
on=before.orig_name if before else None,
on_file=before.path.name if before else None,
next_file=after.path.name if after else None,
queued=len(self._state.pending),
kept=self._state.kept,
passed=self._state.passed,
)
self.refresh()
def on_undo(self) -> None:
"""Take back the last verdict and show that video again."""
before = self._state.current
self._state = self._callbacks.undo(self._state)
after = self._state.current
logs.event(
"review.undo",
was_showing=before.orig_name if before else None,
now_showing=after.orig_name if after else None,
kept=self._state.kept,
passed=self._state.passed,
remaining_undos=self._state.undoable,
)
self.refresh()
def on_quit(self) -> None:
"""Begin shutting down."""
self._state = review.on_quit(self._state)
self.refresh()
def tick(self) -> None:
"""Drain the worker's events, redraw, and schedule the next tick."""
self.drain()
self.refresh()
self._after_id = self._widgets.root.after(POLL_INTERVAL_MS, self.tick)
def drain(self) -> None:
"""Fold every queued worker event into the session state."""
while not self._events.empty():
self._state = review.on_event(self._state, self._events.get_nowait())
def refresh(self) -> None:
"""Re-render from the current state."""
self.apply(review.render(self._state))
def attach_player(self, player: Player) -> None:
"""Adopt a player and show the first video.
Args:
player: The video player to drive.
"""
self._player = player
self.refresh()

205
python_pkg/wsg_grabber/_download.py
View File

@ -1,205 +0,0 @@
"""Streaming one media file to disk, with resume and verification.
Split out of :mod:`python_pkg.wsg_grabber.net` to keep that module under the
250-line cap. ``net`` still owns the session and the JSON endpoints; everything
about pulling bytes from ``i.4cdn.org`` and deciding whether to trust them
lives here.
"""
from __future__ import annotations
import base64
from dataclasses import dataclass
import hashlib
from typing import TYPE_CHECKING
import requests
from python_pkg.wsg_grabber._reject import classify, size_of
from python_pkg.wsg_grabber.constants import (
CHUNK_BYTES,
CONNECT_TIMEOUT_S,
MEDIA_HEADERS,
READ_TIMEOUT_S,
SIZE_SLACK_BYTES,
)
from python_pkg.wsg_grabber.models import DownloadResult, Outcome
from python_pkg.wsg_grabber.scanner import resume_plan
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
_TIMEOUT = (CONNECT_TIMEOUT_S, READ_TIMEOUT_S)
_OK = 200
_PARTIAL = 206
_NOT_FOUND = 404
_GONE = 410
_RANGE_NOT_SATISFIABLE = 416
_TOO_MANY = 429
_SERVER_ERROR = 500
def digest_of(payload: bytes) -> str:
"""Return the base64 MD5 the API uses to identify a file.
MD5 here is 4chan's content identifier, not a security control, hence
``usedforsecurity=False``.
Args:
payload: Full file contents.
Returns:
str: 24-character base64 digest.
"""
digest = hashlib.md5(payload, usedforsecurity=False).digest()
return base64.b64encode(digest).decode("ascii")
@dataclass(frozen=True, slots=True)
class Transfer:
"""Everything one download needs, bundled to stay within the arg limit."""
url: str
part_path: Path
expected_md5: str
expected_size: int
should_stop: Callable[[], bool]
on_progress: Callable[[int], None] | None = None
def download(session: requests.Session, transfer: Transfer) -> DownloadResult:
"""Stream a file to disk, resuming where possible and verifying the result.
The bytes are checked against the API's md5 before the download counts as
finished, so a truncated or mangled transfer can never reach the reviewer.
Args:
session: Shared session.
transfer: What to fetch and where to put it.
Returns:
DownloadResult: Outcome plus the byte count now on disk.
"""
plan = resume_plan(size_of(transfer.part_path), transfer.expected_size)
if plan.discard:
transfer.part_path.unlink(missing_ok=True)
headers = dict(MEDIA_HEADERS)
if plan.offset:
headers["Range"] = f"bytes={plan.offset}-"
try:
return _stream(session, transfer, headers, plan.offset)
except requests.RequestException as exc:
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
message=str(exc),
)
def _stream(
session: requests.Session,
transfer: Transfer,
headers: dict[str, str],
offset: int,
) -> DownloadResult:
"""Perform the request and write the body out.
Args:
session: Shared session.
transfer: What to fetch and where to put it.
headers: Media headers, possibly carrying a ``Range``.
offset: Byte offset requested; zero for a fresh download.
Returns:
DownloadResult: Outcome plus the byte count now on disk.
"""
response = session.get(
transfer.url,
headers=headers,
timeout=_TIMEOUT,
stream=True,
)
rejection = classify(response, transfer)
if rejection is not None:
return rejection
# A server that ignores Range replies 200 with the whole file, so anything
# already on disk has to be thrown away rather than appended to.
append = offset > 0 and response.status_code == _PARTIAL
written = _write_body(response, transfer, append=append)
if written is None:
return DownloadResult(
outcome=Outcome.ABORTED,
bytes_done=size_of(transfer.part_path),
)
if _over_budget(written, transfer.expected_size):
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=0,
message=f"body exceeded the advertised {transfer.expected_size} bytes",
)
payload = transfer.part_path.read_bytes()
if digest_of(payload) != transfer.expected_md5:
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.CHECKSUM_MISMATCH,
bytes_done=0,
message="md5 did not match the value published by the API",
)
return DownloadResult(outcome=Outcome.COMPLETED, bytes_done=len(payload))
def _over_budget(written: int, expected: int) -> bool:
"""Report whether a transfer outgrew the size the API advertised.
Without this a server that lies about Content-Length can write until the
disk is full; the md5 check alone does not help, because whoever serves the
bytes also publishes the digest.
Args:
written: Bytes now on disk.
expected: Size the API declared; zero means it did not say.
Returns:
bool: True when the transfer should be abandoned.
"""
if expected <= 0:
return False
return written > expected + SIZE_SLACK_BYTES
def _write_body(
response: requests.Response,
transfer: Transfer,
*,
append: bool,
) -> int | None:
"""Write the response body to the part file.
Args:
response: Streamed response.
transfer: What is being fetched.
append: Whether to continue an existing part file.
Returns:
int | None: Bytes written, or None when shutdown interrupted it.
"""
mode = "ab" if append else "wb"
done = size_of(transfer.part_path) if append else 0
with transfer.part_path.open(mode) as handle:
for chunk in response.iter_content(chunk_size=CHUNK_BYTES):
if transfer.should_stop():
return None
if not chunk:
continue
handle.write(chunk)
done += len(chunk)
if transfer.on_progress is not None:
transfer.on_progress(done)
if _over_budget(done, transfer.expected_size):
return done
return done

98
python_pkg/wsg_grabber/_fetch.py
View File

@ -1,98 +0,0 @@
"""Fetching one claimed file and recording how it went.
Split out of :mod:`python_pkg.wsg_grabber.downloader` to keep it under the
250-line cap. A mixin rather than free functions because the tests drive
``worker._download_one()`` as a bound method and it shares the worker's
download tally.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import catalog, net, scanner, store
from python_pkg.wsg_grabber._worker_base import WorkerBase
from python_pkg.wsg_grabber.models import Downloaded, FileReady, Outcome
from python_pkg.wsg_grabber.states import FileEvent, FileState, next_state
if TYPE_CHECKING:
from pathlib import Path
from python_pkg.wsg_grabber.models import RemoteFile
class DownloadMixin(WorkerBase):
"""The file-fetching half of :class:`~.downloader.Worker`."""
def _download_one(self) -> None:
"""Claim the next pending file and fetch it."""
claimed = store.claim_next(self._deps.conn)
if claimed is None:
return
part = (
self._deps.incoming
/ f"{store.local_name(claimed.md5, claimed.tim, claimed.ext)}.part"
)
self._wait_for_slot()
result = net.download(
self._deps.session,
net.Transfer(
url=catalog.media_url(claimed.tim, claimed.ext),
part_path=part,
expected_md5=claimed.md5,
expected_size=claimed.fsize,
should_stop=self._deps.stop.is_set,
on_progress=lambda done: store.record_progress(
self._deps.conn,
claimed.md5,
done,
),
),
)
self._apply_result(claimed, part, result.outcome, result.retry_after)
def _apply_result(
self,
claimed: RemoteFile,
part: Path,
outcome: Outcome,
retry_after: float | None,
) -> None:
"""Translate a download outcome into index state and an event.
Args:
claimed: The file that was attempted.
part: Its partial-file path.
outcome: How the download ended.
retry_after: Server-requested pause, when supplied.
"""
if outcome is Outcome.ABORTED:
store.set_state(self._deps.conn, claimed.md5, FileState.FAILED)
return
if outcome is Outcome.COMPLETED:
final = part.with_suffix("")
part.replace(final)
store.mark_downloaded(self._deps.conn, claimed.md5, final.name)
self._downloaded += 1
self._publish(
FileReady(
item=store.ready_item(claimed, final),
),
)
self._publish(Downloaded(count=self._downloaded))
return
event = {
Outcome.NOT_FOUND: FileEvent.NOT_FOUND,
Outcome.CHECKSUM_MISMATCH: FileEvent.CHECKSUM_MISMATCH,
Outcome.TRANSIENT: FileEvent.TRANSIENT_ERROR,
}[outcome]
attempts = store.attempts_for(self._deps.conn, claimed.md5)
store.set_state(
self._deps.conn,
claimed.md5,
next_state(FileState.DOWNLOADING, event, attempts),
error=outcome.value,
)
if retry_after is not None:
self._deps.stop.wait(scanner.backoff_seconds(attempts, retry_after))

76
python_pkg/wsg_grabber/_logs_report.py
View File

@ -1,76 +0,0 @@
"""The ``logs`` subcommand: where session logs live and what is in the newest.
Split out of :mod:`python_pkg.wsg_grabber.cli` to keep that module under the
250-line cap. This is read-only reporting over the JSONL session logs and
shares nothing with the scrape/review flow beyond the log directory itself.
"""
from __future__ import annotations
from collections import Counter
import json
import sys
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import logs
if TYPE_CHECKING:
from pathlib import Path
def _emit(message: str) -> None:
"""Write a line to stdout.
Args:
message: Line to write, without a trailing newline.
"""
sys.stdout.write(f"{message}\n")
def show_logs() -> int:
"""Print where logs live and summarise the most recent session.
Returns:
int: Exit status.
"""
directory = logs.logs_dir()
_emit(f"log directory {directory}")
sessions = sorted(directory.glob("session-*.jsonl"))
if not sessions:
_emit("no session logs yet")
return 0
newest = sessions[-1]
_emit(f"newest session {newest}")
_emit(f"sessions kept {len(sessions)}")
_emit("")
_emit(summarise(newest))
_emit("")
_emit("Useful queries:")
_emit(f" jq -c 'select(.level==\"error\")' {newest}")
_emit(f" jq -r 'select(.event==\"review.show\")|.file' {newest}")
_emit(f" jq -c 'select(.event|startswith(\"player.\"))' {newest}")
return 0
def summarise(path: Path) -> str:
"""Count events by name and level in one session log.
Args:
path: A session JSONL file.
Returns:
str: A short human-readable tally.
"""
by_event: Counter[str] = Counter()
by_level: Counter[str] = Counter()
for line in path.read_text(encoding="utf-8").splitlines():
try:
record = json.loads(line)
except ValueError:
continue
by_event[str(record.get("event"))] += 1
by_level[str(record.get("level"))] += 1
levels = " ".join(f"{name}={count}" for name, count in sorted(by_level.items()))
lines = [f"records by level: {levels or 'none'}", "events:"]
lines.extend(f" {name:<24} {count}" for name, count in by_event.most_common(15))
return "\n".join(lines)

74
python_pkg/wsg_grabber/_migrate_keep.py
View File

@ -1,74 +0,0 @@
"""One-off move of kept videos out of the data dir.
Kept videos used to live in ``~/.local/share/wsg_grabber/keep``, which is
where nobody looks. They now go to the cloud folder (or ``~/Downloads``), and
this module carries the old directory's contents across on the first start
after the change. Idempotent: an empty or missing ``keep/`` is a no-op, and a
partial run just resumes.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import files, logs, paths, store_verdicts, verdict
from python_pkg.wsg_grabber.models import FileMove
if TYPE_CHECKING:
from pathlib import Path
import sqlite3
def migrate_keep(conn: sqlite3.Connection) -> int:
"""Move everything in the legacy ``keep/`` into today's keep directory.
A file already present at the destination with the same size is a
duplicate of an earlier hand-copy and the legacy one is dropped. A name
clash with a different size gets the usual ``-2`` suffix, and the index is
told the new name so the verdict stays undoable.
Args:
conn: Open connection.
Returns:
int: Files moved.
"""
legacy = paths.legacy_keep_dir()
if not legacy.is_dir():
return 0
target = paths.keep_dir()
target.mkdir(parents=True, exist_ok=True)
taken = files.existing_names(target)
moved = 0
for entry in sorted(legacy.iterdir()):
if not entry.is_file():
continue
if _is_duplicate(entry, target / entry.name):
files.remove(entry)
logs.event("keep.migrate_duplicate", file=entry.name)
continue
destination = verdict.unique_destination(target, entry.name, taken)
if not files.apply_move(FileMove(md5="", src=entry, dst=destination)):
continue
taken.add(destination.name)
moved += 1
if destination.name != entry.name:
store_verdicts.rename_kept(conn, entry.name, destination.name)
if not any(legacy.iterdir()):
legacy.rmdir()
if moved:
logs.event("keep.migrated", moved=moved, to=str(target))
return moved
def _is_duplicate(source: Path, candidate: Path) -> bool:
"""Report whether *candidate* is already a copy of *source*.
Args:
source: Legacy file.
candidate: Same name at the destination.
Returns:
bool: True when both exist with the same size.
"""
return candidate.is_file() and candidate.stat().st_size == source.stat().st_size

158
python_pkg/wsg_grabber/_mpv_ipc.py
View File

@ -1,158 +0,0 @@
"""Talking to mpv: the command line, the wire format, and the reply reader.
Split out of :mod:`python_pkg.wsg_grabber.player` to keep it under the 250-line
cap. Everything here is stateless -- it does not own the process or the socket,
only the shapes of what goes over them.
"""
from __future__ import annotations
import json
import socket
from typing import TYPE_CHECKING, Final
from python_pkg.wsg_grabber import logs
if TYPE_CHECKING:
from collections.abc import Sequence
from pathlib import Path
_PROBE_ID = 9001
_PROBE_TIMEOUT_S = 2.0
# What to record when diagnosing an apparently frozen reviewer: which file mpv
# holds, whether it is actually advancing, and the size it configured its video
# output to (which changes per video, so a static value is itself a symptom).
_PROBE_PROPERTIES: Final[tuple[str, ...]] = (
"path",
"pause",
"core-idle",
"eof-reached",
"playback-time",
"width",
"height",
)
def read_reply(conn: socket.socket) -> object:
"""Read until mpv answers the probe request.
Args:
conn: A freshly connected control socket.
Returns:
object: The ``data`` field of the reply, or an ``error:`` string.
"""
buffer = b""
while True:
chunk = conn.recv(65536)
if not chunk:
return "error: socket closed"
buffer += chunk
for line in buffer.split(b"\n"):
if not line:
continue
try:
message = json.loads(line)
except ValueError:
continue
if message.get("request_id") == _PROBE_ID:
return message.get("data")
def build_argv(wid: int, ipc_path: Path) -> list[str]:
"""Build the mpv command line.
``--input-vo-keyboard=no`` matters: with ``--wid`` mpv reparents a child
window over the Tk frame, and if mpv claims the keyboard the Tk bindings
never see a keypress. Declining it leaves X delivering keys to the focused
toplevel, which is the Tk window.
Args:
wid: X11 window id of the frame to draw into.
ipc_path: Where to create the control socket.
Returns:
list[str]: Argument vector for :func:`subprocess.Popen`.
"""
return [
"mpv",
f"--wid={wid}",
f"--input-ipc-server={ipc_path}",
"--idle=yes",
"--loop-file=inf",
"--keep-open=yes",
"--no-terminal",
"--osc=no",
"--input-default-bindings=no",
"--input-vo-keyboard=no",
"--no-config",
]
def encode(command: Sequence[object]) -> bytes:
"""Encode one mpv IPC command.
Args:
command: Command name followed by its arguments.
Returns:
bytes: Newline-terminated JSON, ready for the socket.
"""
return json.dumps({"command": list(command)}).encode("utf-8") + b"\n"
def note_errors(chunk: bytes) -> None:
"""Log any failure mpv reported back.
Args:
chunk: Raw bytes just read from the socket.
"""
for line in chunk.split(b"\n"):
if not line.strip():
continue
try:
message = json.loads(line)
except ValueError:
continue
failure = message.get("error")
if isinstance(failure, str) and failure != "success":
logs.warning("player.reply_error", detail=failure, reply=message)
def probe_properties(ipc_path: Path) -> dict[str, object]:
"""Ask mpv for each diagnostic property in turn.
Args:
ipc_path: Where mpv's control socket lives.
Returns:
dict[str, object]: Property name to value; a property that could not
be read maps to a string starting with ``error:``.
"""
return {name: query(ipc_path, name) for name in _PROBE_PROPERTIES}
def query(ipc_path: Path, name: str) -> object:
"""Read one mpv property over a short-lived connection.
A separate connection avoids interleaving with the asynchronous events mpv
pushes down the main socket, which would otherwise make replies easy to
mis-attribute.
Args:
ipc_path: Where mpv's control socket lives.
name: Property name.
Returns:
object: The value, or an ``error:`` string.
"""
request = json.dumps({"command": ["get_property", name], "request_id": _PROBE_ID})
try:
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as probe:
probe.settimeout(_PROBE_TIMEOUT_S)
probe.connect(str(ipc_path))
probe.sendall(request.encode("utf-8") + b"\n")
return read_reply(probe)
except OSError as exc:
return f"error: {exc}"

49
python_pkg/wsg_grabber/_player_api.py
View File

@ -1,49 +0,0 @@
"""What the reviewer window requires of a video player.
Split out of :mod:`python_pkg.wsg_grabber.player` to keep it under the 250-line
cap. ``ui`` depends on this protocol rather than on :class:`~.player.MpvPlayer`,
so the window can be driven by a stub in the tests without mpv present.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Protocol
if TYPE_CHECKING:
from pathlib import Path
class Player(Protocol):
"""What the reviewer needs from a video player."""
def play(self, path: Path) -> None:
"""Show *path*, replacing whatever is on screen.
Args:
path: Video file to play.
"""
... # pragma: no cover
def stop(self) -> None:
"""Blank the video area."""
... # pragma: no cover
def is_alive(self) -> bool:
"""Report whether the player is still running.
Returns:
bool: True while the process lives.
"""
... # pragma: no cover
def probe(self) -> dict[str, object]:
"""Report what the player is currently doing, for diagnostics.
Returns:
dict[str, object]: Property name to value.
"""
... # pragma: no cover
def close(self) -> None:
"""Shut the player down and release its resources."""
... # pragma: no cover

56
python_pkg/wsg_grabber/_prune.py
View File

@ -1,56 +0,0 @@
"""Keeping ``trash/`` bounded.
A pass used to be a move into a directory nobody ever emptied; 23k files and
77 GB later that was clearly wrong. The trail is now capped at the newest
``TRASH_RETAIN`` passes, which is still far more than anyone undoes. Runs at
startup and after every pass, so the directory never drifts above the cap.
The survivor set comes from the index, not from mtimes: a verdict's
``reviewed_at`` is the fact, the file's timestamp is a side-effect of it.
Anything in ``trash/`` the index does not vouch for -- orphans from before the
``reviewed_name`` column existed -- is deleted along with the old passes.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import files, logs, paths, store_verdicts
from python_pkg.wsg_grabber.constants import TRASH_RETAIN
if TYPE_CHECKING:
import sqlite3
def prune_trash(conn: sqlite3.Connection, retain: int = TRASH_RETAIN) -> int:
"""Delete every trashed file except the *retain* most recent passes.
Files go first, rows second: a crash in between leaves rows that still say
``PASSED`` for files that are gone, which undo already copes with by
forgetting the verdict. The reverse order would leave undoable-looking
files the index had already written off.
Args:
conn: Open connection.
retain: How many passes stay on disk and undoable.
Returns:
int: Files deleted.
"""
trash = paths.trash_dir()
survivors = store_verdicts.recent_passed_names(conn, retain)
removed = 0
if trash.is_dir():
for entry in trash.iterdir():
if entry.name in survivors or not entry.is_file():
continue
removed += files.remove(entry)
purged = store_verdicts.purge_older_passes(conn, retain)
if removed or purged:
logs.event(
"trash.pruned",
removed=removed,
purged_rows=purged,
retained=len(survivors),
)
return removed

96
python_pkg/wsg_grabber/_reject.py
View File

@ -1,96 +0,0 @@
"""Deciding whether a media response can be trusted at all.
Split out of :mod:`python_pkg.wsg_grabber._download` to keep it under the
250-line cap. This is pure classification of a status line and headers into an
:class:`Outcome`; it never touches the body.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import DownloadResult, Outcome
from python_pkg.wsg_grabber.scanner import parse_retry_after
if TYPE_CHECKING:
from pathlib import Path
import requests
from python_pkg.wsg_grabber._download import Transfer
_OK = 200
_PARTIAL = 206
_NOT_FOUND = 404
_GONE = 410
_RANGE_NOT_SATISFIABLE = 416
_TOO_MANY = 429
_SERVER_ERROR = 500
def size_of(path: Path) -> int:
"""Return the size of *path*, or zero when it does not exist.
Args:
path: File to measure.
Returns:
int: Size in bytes.
"""
return path.stat().st_size if path.exists() else 0
def classify(
response: requests.Response,
transfer: Transfer,
) -> DownloadResult | None:
"""Classify a non-success status, if any.
Args:
response: The (streamed) response.
transfer: The transfer being attempted.
Returns:
DownloadResult | None: None when the response is usable.
"""
status = response.status_code
if status in (_NOT_FOUND, _GONE):
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.NOT_FOUND,
bytes_done=0,
message=f"HTTP {status}",
)
encoding = response.headers.get("Content-Encoding", "identity")
if encoding != "identity":
# We asked for identity. A compressed body means the far side is not
# playing by the rules, and requests would silently expand it -- a
# 200 KB response can decode to 200 MB, and the md5 would still match
# because the same party published it.
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
message=f"refused Content-Encoding: {encoding}",
)
if status == _RANGE_NOT_SATISFIABLE:
# The leftover part is longer than the file actually is; start over.
transfer.part_path.unlink(missing_ok=True)
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=0,
message="range rejected; restarting",
)
if status == _TOO_MANY or status >= _SERVER_ERROR:
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
retry_after=parse_retry_after(response.headers.get("Retry-After")),
message=f"HTTP {status}",
)
if status not in (_OK, _PARTIAL):
return DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=size_of(transfer.part_path),
message=f"unexpected HTTP {status}",
)
return None

107
python_pkg/wsg_grabber/_render.py
View File

@ -1,107 +0,0 @@
"""Turning reviewer state into the text and commands the window applies.
Split out of :mod:`python_pkg.wsg_grabber.review` to keep that module under the
250-line cap. Everything here is a pure function of :class:`SessionState`: no
toolkit, no I/O, no state transitions -- those stay in ``review``.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import Emptiness, ReviewCommand
if TYPE_CHECKING:
from python_pkg.wsg_grabber.review import SessionState
_BYTES_PER_MIB = 1024 * 1024
def emptiness(state: SessionState) -> Emptiness:
"""Explain why there is nothing to show, if that is the case.
Args:
state: Current state.
Returns:
Emptiness: Whether the queue is merely waiting or genuinely exhausted.
"""
if state.current is not None:
return Emptiness.NOT_EMPTY
return Emptiness.EXHAUSTED if state.scan_complete else Emptiness.WAITING
def status_line(state: SessionState) -> str:
"""Build the text under the video.
Args:
state: Current state.
Returns:
str: One line summarising progress.
"""
if state.error is not None:
return f"downloader stopped: {state.error}"
counts = (
f"kept {state.kept} · passed {state.passed} · "
f"queued {len(state.pending)} · downloaded {state.downloaded}"
)
if state.indexed:
counts = f"{counts}/{state.indexed}"
undo = f" · u undoes ({state.undoable})" if state.undoable else ""
mood = {
Emptiness.NOT_EMPTY: "",
Emptiness.WAITING: " · waiting for downloads…",
Emptiness.EXHAUSTED: " · all caught up",
}[emptiness(state)]
return f"{counts}{mood}{undo}"
def filename_line(state: SessionState) -> str:
"""Describe the video currently on screen.
Args:
state: Current state.
Returns:
str: Original filename with size and dimensions, or a placeholder.
"""
item = state.current
if item is None:
return "—"
size = item.fsize / _BYTES_PER_MIB
return f"{item.orig_name} · {item.width}x{item.height} · {size:.1f} MiB"
def title_line(state: SessionState) -> str:
"""Build the window title.
Args:
state: Current state.
Returns:
str: Title including the running verdict tally.
"""
return f"/wsg/ — kept {state.kept}, passed {state.passed}"
def render(state: SessionState) -> ReviewCommand:
"""Turn the state into instructions the window applies verbatim.
Args:
state: Current state.
Returns:
ReviewCommand: Everything the GUI should show or do.
"""
item = state.current
return ReviewCommand(
play=None if item is None else item.path,
stop=item is None,
status=status_line(state),
title=title_line(state),
filename=filename_line(state),
verdicts_enabled=item is not None and not state.quitting,
undo_enabled=state.undoable > 0 and not state.quitting,
quit_app=state.quitting,
)

134
python_pkg/wsg_grabber/_scan.py
View File

@ -1,134 +0,0 @@
"""Walking the board: the thread list, the archive, and each changed thread.
Split out of :mod:`python_pkg.wsg_grabber.downloader` to keep it under the
250-line cap. This is a mixin rather than a set of free functions because the
tests drive these as bound methods (``worker._fetch_thread(...)``), and because
they share the worker's scan bookkeeping.
The bare annotations below declare what ``Worker.__init__`` actually sets, so
mypy can see the attributes without the mixin duplicating any of that state.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import catalog, scanner, store, store_threads
from python_pkg.wsg_grabber._worker_base import WorkerBase
from python_pkg.wsg_grabber.constants import (
API_HOST,
ARCHIVE_URL,
BOARD,
THREADS_URL,
)
from python_pkg.wsg_grabber.models import Indexed, ThreadRef
from python_pkg.wsg_grabber.states import CLAIMABLE, FileState
if TYPE_CHECKING:
from python_pkg.wsg_grabber.models import JsonResponse
class ScanMixin(WorkerBase):
"""The board-walking half of :class:`~.downloader.Worker`."""
def _scan_step(self) -> None:
"""Read the thread list, or fetch the next thread that changed."""
if not self._catalog_fresh:
self._refresh_thread_list()
return
ref = self._pending_threads.pop(0)
self._fetch_thread(ref)
def _refresh_thread_list(self) -> None:
"""Fetch ``threads.json`` and work out which threads to visit."""
response = self._conditional_get(THREADS_URL)
live = catalog.parse_thread_list(response.payload)
if self._deps.include_archive:
live = live + self._archive_refs()
known = store_threads.known_threads(self._deps.conn)
self._pending_threads = scanner.stale_threads(known, live)
self._catalog_fresh = True
self._finished_announced = False
def _archive_refs(self) -> list[ThreadRef]:
"""Fetch the archive listing, tolerating boards without one.
Returns:
list[ThreadRef]: Archived threads, empty when unavailable.
"""
response = self._conditional_get(ARCHIVE_URL)
if response.not_found:
return []
return catalog.parse_archive(response.payload)
def _conditional_get(self, url: str) -> JsonResponse:
"""Fetch a board-level endpoint, sending back the stamp we hold.
Without this an idle reviewer re-fetches threads.json and archive.json
unconditionally every cycle. With it an unchanged board costs a 304.
Args:
url: Absolute API URL.
Returns:
JsonResponse: The response; on 304 the payload is None and the
caller keeps whatever it already knew.
"""
stamp = store_threads.cached_last_modified(self._deps.conn, url)
response = self._get_json(url, stamp)
if not response.not_modified:
store_threads.record_last_modified(
self._deps.conn,
url,
response.last_modified,
)
return response
def _fetch_thread(self, ref: ThreadRef) -> None:
"""Read one thread and record any files it introduces.
Args:
ref: Thread to fetch.
"""
url = f"{API_HOST}/{BOARD}/thread/{ref.thread_no}.json"
stamp = store_threads.http_last_modified(self._deps.conn, ref.thread_no)
response = self._get_json(url, stamp)
if response.not_found:
store_threads.mark_thread_gone(self._deps.conn, ref.thread_no)
return
if response.not_modified:
store_threads.record_thread(self._deps.conn, ref, stamp)
return
parsed = catalog.parse_thread(ref.thread_no, response.payload)
known = store.known_md5s(self._deps.conn)
added = store.record_files(
self._deps.conn,
catalog.new_files(parsed, known),
)
self._write_off_deleted(catalog.deleted_md5s(response.payload), known)
store_threads.record_thread(self._deps.conn, ref, response.last_modified)
if added:
self._publish(Indexed(known=len(known) + added))
def _write_off_deleted(self, gone: set[str], known: set[str]) -> None:
"""Mark not-yet-fetched files terminal when the board drops them.
Only files still waiting to download are affected. Once the bytes are
on disk the post being deleted upstream is irrelevant, so an already
downloaded or reviewed file keeps its state -- the same rule the
transition table encodes by rejecting ``READY + DELETED_UPSTREAM``.
Args:
gone: md5s the board reports as deleted.
known: md5s already in the index.
"""
for digest in gone & known:
if store.state_of(self._deps.conn, digest) not in CLAIMABLE:
continue
store.set_state(
self._deps.conn,
digest,
FileState.GONE,
error="deleted upstream",
)

232
python_pkg/wsg_grabber/_store_queries.py
View File

@ -1,232 +0,0 @@
"""Read-only projections over the file index, plus the atomic claim.
Split out of :mod:`python_pkg.wsg_grabber.store` to keep it under the 250-line
cap. ``store`` owns the writes that move a file through its lifecycle; this
module asks the questions -- what is reviewable, how many of each state there
are, how much work is left -- and owns ``claim_next``, which is the one write
that has to be a single atomic transaction against those same reads.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import RemoteFile, ReviewItem
from python_pkg.wsg_grabber.states import CLAIMABLE, FileState
if TYPE_CHECKING:
from collections.abc import Mapping, Sequence
from pathlib import Path
import sqlite3
def scalar(
conn: sqlite3.Connection,
sql: str,
params: Sequence[object] | Mapping[str, object] | None = None,
) -> int:
"""Run a single-value query and return it as an int.
Args:
conn: Open connection.
sql: Query yielding exactly one column.
params: Optional bound parameters.
Returns:
int: The first column of the first row.
"""
cursor = conn.execute(sql) if params is None else conn.execute(sql, params)
return int(cursor.fetchone()[0])
def claimable_values() -> list[str]:
"""Return the state values a download worker may pick up, in a stable order.
The queries below spell out one placeholder per member rather than building
the ``IN`` list at runtime, so the SQL stays a literal. ``test_store``
pins the member count so adding a state cannot silently desync the two.
Returns:
list[str]: Sorted state values.
"""
return [state.value for state in sorted(CLAIMABLE)]
def claim_next(conn: sqlite3.Connection) -> RemoteFile | None:
"""Atomically take the oldest downloadable file and mark it in flight.
Args:
conn: Open connection.
Returns:
RemoteFile | None: The claimed file, or None when nothing is pending.
"""
conn.execute("BEGIN IMMEDIATE")
try:
row = conn.execute(
"""
SELECT md5, tim, ext, orig_name, fsize, width, height,
thread_no, post_no
FROM files
WHERE state IN (?, ?, ?)
ORDER BY first_seen, md5
LIMIT 1
""",
claimable_values(),
).fetchone()
if row is None:
conn.execute("COMMIT")
return None
conn.execute(
"UPDATE files SET state = ?, attempts = attempts + 1 WHERE md5 = ?",
(FileState.DOWNLOADING.value, row["md5"]),
)
conn.execute("COMMIT")
except Exception:
conn.execute("ROLLBACK")
raise
return RemoteFile(
md5=row["md5"],
tim=row["tim"],
ext=row["ext"],
orig_name=row["orig_name"],
fsize=row["fsize"],
width=row["width"],
height=row["height"],
thread_no=row["thread_no"],
post_no=row["post_no"],
)
def ready_items(conn: sqlite3.Connection, directory: Path) -> list[ReviewItem]:
"""Return downloaded files still awaiting a verdict, oldest first.
Args:
conn: Open connection.
directory: Where downloaded files live.
Returns:
list[ReviewItem]: Reviewable videos.
"""
rows = conn.execute(
"""
SELECT md5, local_name, orig_name, ext, fsize, width, height
FROM files
WHERE state = ? AND local_name IS NOT NULL
ORDER BY downloaded_at, md5
""",
(FileState.READY.value,),
).fetchall()
return [
ReviewItem(
md5=row["md5"],
path=directory / row["local_name"],
orig_name=f"{row['orig_name']}{row['ext']}",
fsize=row["fsize"],
width=row["width"],
height=row["height"],
)
for row in rows
]
def ready_item(source: RemoteFile, path: Path) -> ReviewItem:
"""Build the reviewer's view of a file that just finished downloading.
Args:
source: The catalogue record it came from.
path: Where the verified bytes now live.
Returns:
ReviewItem: Value handed to the UI thread.
"""
return ReviewItem(
md5=source.md5,
path=path,
orig_name=f"{source.orig_name}{source.ext}",
fsize=source.fsize,
width=source.width,
height=source.height,
)
def counts(conn: sqlite3.Connection) -> dict[str, int]:
"""Return the number of files in each state.
Args:
conn: Open connection.
Returns:
dict[str, int]: State value mapped to row count, plus ``total``.
"""
result = {
str(row["state"]): int(row["n"])
for row in conn.execute(
"SELECT state, COUNT(*) AS n FROM files GROUP BY state",
)
}
result["total"] = scalar(conn, "SELECT COUNT(*) FROM files")
return result
def pending_downloads(conn: sqlite3.Connection) -> int:
"""Return how many files are still waiting to be fetched.
Args:
conn: Open connection.
Returns:
int: Count of claimable rows.
"""
return scalar(
conn,
"SELECT COUNT(*) FROM files WHERE state IN (?, ?, ?)",
claimable_values(),
)
def known_md5s(conn: sqlite3.Connection) -> set[str]:
"""Return every md5 the index has ever seen.
Args:
conn: Open connection.
Returns:
set[str]: Base64 md5 values, including terminal ones, so a reviewed or
written-off file is never offered again.
"""
return {str(row[0]) for row in conn.execute("SELECT md5 FROM files")}
def state_of(conn: sqlite3.Connection, md5_b64: str) -> FileState | None:
"""Return a file's current state.
Args:
conn: Open connection.
md5_b64: File identity.
Returns:
FileState | None: The state, or None when the file is unknown.
"""
row = conn.execute(
"SELECT state FROM files WHERE md5 = ?",
(md5_b64,),
).fetchone()
return None if row is None else FileState(row["state"])
def attempts_for(conn: sqlite3.Connection, md5_b64: str) -> int:
"""Return how many download attempts a file has had.
Args:
conn: Open connection.
md5_b64: File identity.
Returns:
int: Attempt count, zero when the file is unknown.
"""
row = conn.execute(
"SELECT attempts FROM files WHERE md5 = ?",
(md5_b64,),
).fetchone()
return 0 if row is None else int(row["attempts"])

74
python_pkg/wsg_grabber/_undo.py
View File

@ -1,74 +0,0 @@
"""Taking verdicts back.
Split out of :mod:`python_pkg.wsg_grabber.review` to keep that module under the
250-line cap. Undo is its own concern: it has to correct the running tally,
put the video that was on screen back at the front of the queue, and cope with
a file that has since moved -- none of which the forward path cares about.
"""
from __future__ import annotations
from dataclasses import replace
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import Verdict
if TYPE_CHECKING:
from python_pkg.wsg_grabber.models import ReviewedItem
from python_pkg.wsg_grabber.review import SessionState
def can_undo(state: SessionState) -> bool:
"""Report whether there is a verdict to take back.
Args:
state: Current state.
Returns:
bool: True when the index holds at least one reversible verdict.
"""
return state.undoable > 0
def on_undo(state: SessionState, action: ReviewedItem) -> SessionState:
"""Show a restored video again after its verdict was reversed.
The video currently on screen goes to the front of the queue, so changing
your mind never skips anything.
Args:
state: Current state.
action: The verdict that was just reversed on disk and in the index.
Returns:
SessionState: Updated state showing the restored video.
"""
queue = list(state.pending)
if state.current is not None:
queue.insert(0, state.current)
corrected = (
replace(state, kept=max(0, state.kept - 1))
if action.choice is Verdict.KEEP
else replace(state, passed=max(0, state.passed - 1))
)
return replace(
corrected,
undoable=max(0, state.undoable - 1),
pending=tuple(queue),
current=action.item,
)
def forget_last(state: SessionState) -> SessionState:
"""Drop one verdict from the undo count without reversing it.
Used when the file is no longer where the verdict left it, so a second
attempt does not keep failing on the same entry.
Args:
state: Current state.
Returns:
SessionState: Updated state.
"""
return replace(state, undoable=max(0, state.undoable - 1))

52
python_pkg/wsg_grabber/_worker_base.py
View File

@ -1,52 +0,0 @@
"""The state and collaborators the worker's two halves both rely on.
:class:`~python_pkg.wsg_grabber.downloader.Worker` is assembled from two mixins
-- :mod:`._scan` and :mod:`._fetch` -- so that no single module exceeds the
250-line cap. Each half touches bookkeeping the other half owns, so both
inherit from here rather than declaring the same attributes twice.
The annotations carry no values: ``Worker.__init__`` remains the only place
this state is created. The two methods are abstract because ``Worker`` supplies
them; declaring them gives mypy and pylint something real to resolve against,
which a bare mixin does not.
"""
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from python_pkg.wsg_grabber.downloader import WorkerDeps
from python_pkg.wsg_grabber.models import (
DownloadEvent,
JsonResponse,
ThreadRef,
)
class WorkerBase(ABC):
"""State and collaborators shared by the worker's two halves."""
_deps: WorkerDeps
_pending_threads: list[ThreadRef]
_catalog_fresh: bool
_finished_announced: bool
# Given a real class-level default rather than a bare annotation: _fetch
# increments it (`self._downloaded += 1`), and pylint only treats an
# attribute as existing for assignment when there is something to assign
# to. Worker.__init__ still sets the instance value on construction.
_downloaded: int = 0
@abstractmethod
def _get_json(self, url: str, stamp: str | None = None) -> JsonResponse:
"""Fetch a JSON endpoint, pacing requests and counting the last one."""
@abstractmethod
def _publish(self, event: DownloadEvent) -> None:
"""Hand an event to the UI thread."""
@abstractmethod
def _wait_for_slot(self) -> None:
"""Block until the polite inter-request delay has elapsed."""

249
python_pkg/wsg_grabber/app.py
View File

@ -1,249 +0,0 @@
"""Wires the pieces together and owns an orderly shutdown.
Kept separate from :mod:`python_pkg.wsg_grabber.cli` so the composition can be
exercised without argument parsing, and separate from
:mod:`python_pkg.wsg_grabber.ui` so nothing here needs a display.
"""
from __future__ import annotations
from dataclasses import dataclass
import queue
import threading
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import (
db,
downloader,
files,
housekeeping,
logs,
net,
paths,
review,
store,
store_verdicts,
verdict,
)
from python_pkg.wsg_grabber.constants import JOIN_TIMEOUT_S
from python_pkg.wsg_grabber.models import FileMove, Verdict, WorkerFailed
from python_pkg.wsg_grabber.states import TERMINAL, FileState
if TYPE_CHECKING:
import sqlite3
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.review import SessionState
@dataclass
class Session:
"""A running review session and everything it must clean up."""
conn: sqlite3.Connection
events: queue.SimpleQueue[DownloadEvent]
stop: threading.Event
worker: threading.Thread | None = None
def commit(self, state: SessionState, choice: Verdict) -> SessionState:
"""Apply a verdict: move the file, record it, advance the queue.
Args:
state: Current session state.
choice: What the user decided.
Returns:
SessionState: Updated state.
"""
item = state.current
if item is None:
return state
move = verdict.plan_move(
item,
choice,
paths.keep_dir(),
paths.trash_dir(),
files.existing_names(
verdict.target_dir(choice, paths.keep_dir(), paths.trash_dir()),
),
)
if not files.apply_move(move):
logs.warning(
"verdict.source_gone",
file=move.src.name,
choice=choice.value,
)
return self.missing(state)
logs.event(
"verdict.moved",
choice=choice.value,
src=str(move.src),
dst=str(move.dst),
md5=item.md5,
)
store_verdicts.record_verdict(
self.conn,
item.md5,
verdict.target_state(choice),
move.dst.name,
)
housekeeping.after_verdict(self.conn, choice)
return review.on_verdict(state, choice)
def undo(self, state: SessionState) -> SessionState:
"""Take back the last verdict, putting the file back in the queue.
Args:
state: Current session state.
Returns:
SessionState: Updated state showing the restored video.
"""
action = store_verdicts.newest_verdict(self.conn, paths.incoming_dir())
if action is None:
return state
source = housekeeping.locate_reviewed(action)
# Mirror what commit does: never assume the old name is free again.
destination = verdict.unique_destination(
paths.incoming_dir(),
action.item.path.name,
files.existing_names(paths.incoming_dir()),
)
back = FileMove(md5=action.item.md5, src=source, dst=destination)
if not files.apply_move(back):
# The file is no longer where the verdict left it -- most likely the
# user cleared trash/ by hand. Drop it from the undo trail so a
# second attempt does not keep failing on the same entry.
logs.warning(
"undo.file_gone",
expected=str(source),
md5=action.item.md5,
)
store_verdicts.forget_verdict(self.conn, action.item.md5)
return review.forget_last(state)
store_verdicts.restore_for_review(
self.conn,
action.item.md5,
destination.name,
)
logs.event(
"undo.applied",
choice=action.choice.value,
file=destination.name,
restored_from=str(source),
remaining=store_verdicts.reviewed_count(self.conn),
)
return review.on_undo(state, action)
def missing(self, state: SessionState) -> SessionState:
"""Write off a video whose file has disappeared.
Args:
state: Current session state.
Returns:
SessionState: Updated state showing the next video.
"""
item = state.current
if item is None:
return state
# A row that already carries a verdict must not be forced to GONE: that
# would drop it out of the undo trail while the in-memory count still
# believes it is there, and the verdict itself would be unrecoverable.
# Reachable when the same item is queued twice, e.g. two reviewers
# running against one index.
if store.state_of(self.conn, item.md5) in TERMINAL:
logs.warning("missing.already_reviewed", md5=item.md5)
return review.on_missing_locally(state)
store.set_state(
self.conn,
item.md5,
FileState.GONE,
error="file missing locally",
)
return review.on_missing_locally(state)
def shutdown(self) -> None:
"""Stop the worker and close the index.
Joining matters: the worker owns a database connection and an HTTP
session, and abandoning it leaks both. Note the suite does NOT catch
that for us -- ``meta/pyproject.toml`` downgrades
``PytestUnraisableExceptionWarning`` to a warning -- so this has to be
right by construction rather than by test pressure.
"""
self.stop.set()
if self.worker is not None:
self.worker.join(timeout=JOIN_TIMEOUT_S)
if self.worker.is_alive():
# A worker blocked in a socket read can outlive the join, since
# READ_TIMEOUT_S exceeds JOIN_TIMEOUT_S. Keep the handle so the
# failure is visible rather than silently discarded.
logs.warning("shutdown.worker_still_running")
else:
self.worker = None
self.conn.close()
def open_session(*, include_archive: bool = True) -> Session:
"""Prepare storage and start the background worker.
Args:
include_archive: Whether to walk the board's archive as well.
Returns:
Session: A running session; call :meth:`Session.shutdown` when done.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
store.reset_in_flight(conn)
housekeeping.on_startup(conn)
events: queue.SimpleQueue[DownloadEvent] = queue.SimpleQueue()
stop = threading.Event()
session = Session(conn=conn, events=events, stop=stop)
incoming = paths.incoming_dir()
db_file = paths.db_path()
def build() -> downloader.WorkerDeps:
"""Open the worker's own resources on the worker's own thread.
Returns:
downloader.WorkerDeps: Dependencies owned by the worker.
"""
return downloader.WorkerDeps(
conn=db.open_index(db_file),
session=net.build_session(),
incoming=incoming,
events=events,
stop=stop,
include_archive=include_archive,
)
try:
session.worker = downloader.start(
build,
lambda message: events.put(WorkerFailed(message=message)),
)
except BaseException:
conn.close()
raise
return session
def initial_state(session: Session) -> SessionState:
"""Build the state a window should open with.
Args:
session: The running session.
Returns:
SessionState: Starting state, including anything left unreviewed by a
previous run.
"""
counts = store.counts(session.conn)
return review.initial_state(
counts.get("total", 0),
store.ready_items(session.conn, paths.incoming_dir()),
store_verdicts.reviewed_count(session.conn),
)

175
python_pkg/wsg_grabber/catalog.py
View File

@ -1,175 +0,0 @@
"""Parsing of the /wsg/ JSON payloads. No network, no filesystem.
The API hands back loosely-typed JSON where file fields are simply absent on
posts without an attachment, so every accessor here treats a missing or
wrong-typed field as "not a video" rather than trusting the shape.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber._coerce import as_int, as_list, as_md5, get
from python_pkg.wsg_grabber.constants import MEDIA_HOST, VIDEO_EXTENSIONS
from python_pkg.wsg_grabber.models import RemoteFile, ThreadRef
if TYPE_CHECKING:
from collections.abc import Iterable, Sequence
def media_url(tim: int, ext: str) -> str:
"""Return the CDN address of an attachment.
Args:
tim: The post's ``tim`` field, which is the CDN filename stem.
ext: Extension including the leading dot.
Returns:
str: Absolute URL on the media host.
"""
return f"{MEDIA_HOST}/wsg/{tim}{ext}"
def parse_thread_list(payload: object) -> list[ThreadRef]:
"""Flatten ``threads.json`` into thread references.
Args:
payload: Decoded JSON: a list of pages, each with a ``threads`` list.
Returns:
list[ThreadRef]: Every advertised thread, in page order.
"""
refs: list[ThreadRef] = []
for page in as_list(payload):
for entry in as_list(get(page, "threads")):
thread_no = as_int(get(entry, "no"))
if thread_no is None:
continue
refs.append(
ThreadRef(
thread_no=thread_no,
api_last_modified=as_int(get(entry, "last_modified")) or 0,
),
)
return refs
def parse_archive(payload: object) -> list[ThreadRef]:
"""Turn ``archive.json`` into thread references.
The archive is a bare list of thread numbers with no timestamps, so each
gets a zero stamp; they are immutable anyway and only need fetching once.
Args:
payload: Decoded JSON list of integers.
Returns:
list[ThreadRef]: Archived threads.
"""
refs: list[ThreadRef] = []
for entry in as_list(payload):
thread_no = as_int(entry)
if thread_no is not None:
refs.append(ThreadRef(thread_no=thread_no, api_last_modified=0))
return refs
def parse_thread(thread_no: int, payload: object) -> list[RemoteFile]:
"""Extract every video attachment from one thread payload.
Posts whose file was deleted upstream are skipped: there is nothing left to
fetch, and they must not be recorded as pending work.
Args:
thread_no: The thread being parsed, used to stamp each file.
payload: Decoded ``thread/<no>.json``.
Returns:
list[RemoteFile]: Attachments with a video extension.
"""
files: list[RemoteFile] = []
for post in as_list(get(payload, "posts")):
parsed = _parse_post(thread_no, post)
if parsed is not None:
files.append(parsed)
return files
def deleted_md5s(payload: object) -> set[str]:
"""Return md5s of posts whose attachment was removed upstream.
Args:
payload: Decoded ``thread/<no>.json``.
Returns:
set[str]: Identities that should be written off rather than retried.
"""
gone: set[str] = set()
for post in as_list(get(payload, "posts")):
if not as_int(get(post, "filedeleted")):
continue
digest = as_md5(get(post, "md5"))
if digest is not None:
gone.add(digest)
return gone
def new_files(
parsed: Iterable[RemoteFile],
known: Sequence[str] | set[str] | frozenset[str],
) -> list[RemoteFile]:
"""Drop attachments already present in the index.
This is where "never download the same video twice" is enforced, and it
happens before a single byte is transferred because the API supplies the
md5 up front.
Args:
parsed: Candidates from a thread.
known: md5s already recorded, in any state including terminal ones.
Returns:
list[RemoteFile]: Genuinely new attachments, deduped within the batch.
"""
seen = set(known)
fresh: list[RemoteFile] = []
for item in parsed:
if item.md5 in seen:
continue
seen.add(item.md5)
fresh.append(item)
return fresh
def _parse_post(thread_no: int, post: object) -> RemoteFile | None:
"""Turn one post into a RemoteFile when it carries a usable video.
Args:
thread_no: Owning thread.
post: Decoded post object.
Returns:
RemoteFile | None: None when the post has no video attachment.
"""
if as_int(get(post, "filedeleted")):
return None
tim = as_int(get(post, "tim"))
ext = get(post, "ext")
digest = as_md5(get(post, "md5"))
post_no = as_int(get(post, "no"))
if tim is None or post_no is None or digest is None:
return None
if not isinstance(ext, str) or ext not in VIDEO_EXTENSIONS:
return None
name = get(post, "filename")
return RemoteFile(
md5=digest,
tim=tim,
ext=ext,
orig_name=name if isinstance(name, str) else str(tim),
fsize=as_int(get(post, "fsize")) or 0,
width=as_int(get(post, "w")) or 0,
height=as_int(get(post, "h")) or 0,
thread_no=thread_no,
post_no=post_no,
)

248
python_pkg/wsg_grabber/cli.py
View File

@ -1,248 +0,0 @@
"""Command-line entry point for the /wsg/ grabber.
Usage:
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber scrape --seconds 60
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber stats
"""
from __future__ import annotations
import argparse
import sys
import time
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import (
app,
db,
logs,
paths,
player,
store,
store_threads,
ui,
)
from python_pkg.wsg_grabber._logs_report import show_logs, summarise
# Re-exported so `cli.show_logs` / `cli.summarise` keep working for callers and
# tests; the implementation lives in _logs_report to keep this module under the
# 250-line cap. __all__ is what marks them as re-exports -- the `X as X` alias
# form is rejected here (ruff PLC0414) and then stripped as unused.
from python_pkg.wsg_grabber.constants import TRASH_RETAIN
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Callable
__all__ = [
"build_parser",
"main",
"open_reviewer",
"progress",
"scrape",
"show_logs",
"stats",
"summarise",
]
_REPORT_EVERY_S = 15.0
_DESCRIPTION = (
"Scrape 4chan /wsg/ for videos and triage them with a keep/pass reviewer. "
"A keep goes to the cloud folder (or ~/Downloads); a pass goes to trash, "
"where only the newest passes are kept."
)
def _emit(message: str) -> None:
"""Write a line to stdout.
Args:
message: Line to write, without a trailing newline.
"""
sys.stdout.write(f"{message}\n")
def build_parser() -> argparse.ArgumentParser:
"""Construct the argument parser.
Returns:
argparse.ArgumentParser: Parser with the ``review``, ``scrape`` and
``stats`` subcommands. ``review`` is the default.
"""
parser = argparse.ArgumentParser(
prog="python -m python_pkg.wsg_grabber",
description=_DESCRIPTION,
)
parser.add_argument(
"--log-level",
choices=logs.LEVELS,
default="info",
help="detail written to the session log (default: info)",
)
parser.add_argument(
"--echo-log",
action="store_true",
help="also print log records to stderr as they happen",
)
subparsers = parser.add_subparsers(dest="command")
subparsers.add_parser(
"review",
help="download in the background and review on screen (default)",
)
subparsers.add_parser("stats", help="print index counts and exit")
subparsers.add_parser(
"logs",
help="show where session logs live and summarise the newest one",
)
scrape_parser = subparsers.add_parser(
"scrape",
help="index and download without opening the reviewer",
)
scrape_parser.add_argument(
"--seconds",
type=float,
default=0.0,
help="stop after this long (0 means run until interrupted)",
)
return parser
def main(argv: list[str] | None = None) -> int:
"""Run the requested subcommand.
Args:
argv: Argument list, defaulting to ``sys.argv[1:]``.
Returns:
int: Process exit status.
"""
args = build_parser().parse_args(argv)
logs.start(args.log_level, echo=args.echo_log)
try:
return _dispatch(args)
finally:
logs.stop()
def _dispatch(args: argparse.Namespace) -> int:
"""Run the subcommand named in *args*.
Args:
args: Parsed arguments.
Returns:
int: Process exit status.
"""
if args.command == "logs":
return show_logs()
if args.command == "stats":
return stats()
if args.command == "scrape":
return scrape(args.seconds)
return open_reviewer()
def stats() -> int:
"""Print what the index knows.
Returns:
int: Exit status.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
try:
counts = store.counts(conn)
_emit(f"index {paths.db_path()}")
_emit(f"known files {counts.get('total', 0)}")
for state in FileState:
_emit(f" {state.value:<12} {counts.get(state.value, 0)}")
_emit(f"threads {len(store_threads.known_threads(conn))}")
_emit(f"keep dir {paths.keep_dir()}")
_emit(f"trash dir {paths.trash_dir()} (newest {TRASH_RETAIN} passes kept)")
finally:
conn.close()
return 0
def scrape(seconds: float, sleeper: Callable[[float], None] | None = None) -> int:
"""Index and download without opening a window.
Args:
seconds: Stop after this long; 0 runs until interrupted.
sleeper: Optional stand-in for :func:`time.sleep`, used by tests to
avoid actually waiting.
Returns:
int: Exit status.
"""
rest = time.sleep if sleeper is None else sleeper
session = app.open_session()
started = time.monotonic()
reported = started
try:
_emit("scraping /wsg/ — Ctrl-C to stop")
while True:
rest(1.0)
now = time.monotonic()
if now - reported >= _REPORT_EVERY_S:
reported = now
_emit(progress(session))
if seconds and now - started >= seconds:
break
except KeyboardInterrupt:
_emit("stopping")
finally:
_emit(progress(session))
session.shutdown()
return 0
def progress(session: app.Session) -> str:
"""Summarise index progress.
Args:
session: The running session.
Returns:
str: One-line summary.
"""
counts = store.counts(session.conn)
return (
f"known {counts.get('total', 0)} · "
f"ready {counts.get(FileState.READY.value, 0)} · "
f"pending {store.pending_downloads(session.conn)}"
)
def open_reviewer() -> int:
"""Open the reviewer window and hand control to it.
Returns:
int: Exit status.
"""
session = app.open_session()
try:
window = ui.ReviewWindow(
app.initial_state(session),
session.events,
ui.Callbacks(
commit=session.commit,
missing=session.missing,
undo=session.undo,
shutdown=session.shutdown,
),
)
window.attach_player(
player.MpvPlayer(window.video_wid(), paths.ipc_socket_path()),
)
window.run()
except BaseException:
# The worker is a non-daemon thread, so anything raised between opening
# the session and entering the Tk loop -- mpv missing, a stale socket --
# would otherwise leave the process alive forever with no window.
logs.error("review.startup_failed")
session.shutdown()
raise
return 0

109
python_pkg/wsg_grabber/constants.py
View File

@ -1,109 +0,0 @@
"""Fixed configuration for the /wsg/ grabber.
Values here are deliberately plain data so the pure planning helpers in
:mod:`python_pkg.wsg_grabber.scanner` can be exercised without touching the
network, the filesystem or a display.
"""
from __future__ import annotations
from types import MappingProxyType
from typing import TYPE_CHECKING, Final
if TYPE_CHECKING:
from collections.abc import Mapping
BOARD: Final[str] = "wsg"
# The only board this tool targets.
API_HOST: Final[str] = "https://a.4cdn.org"
MEDIA_HOST: Final[str] = "https://i.4cdn.org"
BOARDS_URL: Final[str] = "https://boards.4chan.org/"
THREADS_URL: Final[str] = f"{API_HOST}/{BOARD}/threads.json"
ARCHIVE_URL: Final[str] = f"{API_HOST}/{BOARD}/archive.json"
VIDEO_EXTENSIONS: Final[frozenset[str]] = frozenset({".webm", ".mp4", ".gif"})
# /wsg/ serves all three. A live catalogue sample was 55% webm, 37% mp4, 8%
# gif, so filtering to {".webm", ".gif"} would silently drop a third of it.
# i.4cdn.org answers a bare request with 429 once it has seen a burst, so the
# session presents itself as a browser following a link from the board. The
# cookies 4chan sets in response (``__cf_bm``, ``_cfuvid``) are why a single
# long-lived Session is mandatory rather than merely efficient.
USER_AGENT: Final[str] = (
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"
)
API_HEADERS: Final[Mapping[str, str]] = MappingProxyType(
{
"User-Agent": USER_AGENT,
"Accept": "application/json,text/plain,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
# Ask for no compression so a hostile or compromised host cannot turn a
# few hundred KB on the wire into gigabytes of decoded JSON.
"Accept-Encoding": "identity",
"Referer": BOARDS_URL,
},
)
MEDIA_HEADERS: Final[Mapping[str, str]] = MappingProxyType(
{
"User-Agent": USER_AGENT,
"Accept": "video/webm,video/mp4,image/gif,video/*;q=0.9,*/*;q=0.5",
"Accept-Language": "en-US,en;q=0.9",
# Never gzip a webm: it cannot compress and it breaks Range resumption.
"Accept-Encoding": "identity",
"Referer": BOARDS_URL,
"Sec-Fetch-Dest": "video",
"Sec-Fetch-Mode": "no-cors",
"Sec-Fetch-Site": "cross-site",
},
)
MIN_REQUEST_INTERVAL_S: Final[float] = 1.0
# The 4chan API rules state at most one request per second.
BACKOFF_BASE_S: Final[float] = 2.0
BACKOFF_CAP_S: Final[float] = 120.0
MAX_ATTEMPTS: Final[int] = 3
# Download attempts before a file is written off as ``gone``.
CONNECT_TIMEOUT_S: Final[float] = 10.0
READ_TIMEOUT_S: Final[float] = 30.0
CHUNK_BYTES: Final[int] = 65536
# Refuse an API response larger than this; the real ones are ~300 KB.
MAX_JSON_BYTES: Final[int] = 32 * 1024 * 1024
# How far a download may exceed its advertised size before it is abandoned.
SIZE_SLACK_BYTES: Final[int] = 1024 * 1024
SCHEMA_VERSION: Final[int] = 3
DB_TIMEOUT_S: Final[float] = 30.0
DB_BUSY_TIMEOUT_MS: Final[int] = 5000
POLL_INTERVAL_MS: Final[int] = 150
# How often the Tk loop drains the downloader's event queue.
IDLE_POLL_S: Final[float] = 1.0
# How long the worker sleeps per tick once the board is exhausted.
IDLE_RESCAN_S: Final[float] = 60.0
# How long a finished sweep stays quiet before the board is walked again.
JOIN_TIMEOUT_S: Final[float] = 10.0
IPC_READY_TIMEOUT_S: Final[float] = 10.0
CONTROL_BAR_HEIGHT_PX: Final[int] = 96
WINDOW_WIDTH_PX: Final[int] = 1100
WINDOW_HEIGHT_PX: Final[int] = 800
# How many passed videos stay in trash/ (newest first) and so stay undoable.
# Everything older is deleted for good; 23k files / 77 GB had piled up before
# this bound existed.
TRASH_RETAIN: Final[int] = 200
KEEP_KEYS: Final[tuple[str, ...]] = ("<k>", "<Right>")
PASS_KEYS: Final[tuple[str, ...]] = ("<j>", "<space>", "<Left>")
UNDO_KEYS: Final[tuple[str, ...]] = ("<u>", "<BackSpace>")
QUIT_KEYS: Final[tuple[str, ...]] = ("<q>", "<Escape>")

164
python_pkg/wsg_grabber/db.py
View File

@ -1,164 +0,0 @@
"""Schema definition, connection setup and migration for the file index.
sqlite rather than the JSON state files used elsewhere in this repo: two threads
write concurrently (the downloader appending rows, the UI recording verdicts),
the seen-set grows without bound, and "have I seen this md5" must stay O(1).
Rewriting a whole JSON document per verdict would not survive either property.
``check_same_thread`` is deliberately left at its default so a connection that
escapes its owning thread raises loudly instead of corrupting the file.
"""
from __future__ import annotations
import sqlite3
from typing import TYPE_CHECKING, Final
from python_pkg.wsg_grabber.constants import (
DB_BUSY_TIMEOUT_MS,
DB_TIMEOUT_S,
SCHEMA_VERSION,
)
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
_SCHEMA: Final[tuple[str, ...]] = (
"""
CREATE TABLE IF NOT EXISTS files (
md5 TEXT PRIMARY KEY,
tim INTEGER NOT NULL,
ext TEXT NOT NULL,
orig_name TEXT NOT NULL,
fsize INTEGER NOT NULL,
width INTEGER NOT NULL,
height INTEGER NOT NULL,
thread_no INTEGER NOT NULL,
post_no INTEGER NOT NULL,
state TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
bytes_done INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
local_name TEXT,
first_seen TEXT NOT NULL,
downloaded_at TEXT,
reviewed_at TEXT,
reviewed_name TEXT
)
""",
"CREATE INDEX IF NOT EXISTS idx_files_state ON files(state)",
"CREATE INDEX IF NOT EXISTS idx_files_claim ON files(state, first_seen)",
"""
CREATE TABLE IF NOT EXISTS threads (
thread_no INTEGER PRIMARY KEY,
api_last_mod INTEGER NOT NULL,
http_last_mod TEXT,
last_checked TEXT NOT NULL,
status TEXT NOT NULL
)
""",
"CREATE INDEX IF NOT EXISTS idx_threads_status ON threads(status)",
"CREATE INDEX IF NOT EXISTS idx_files_reviewed ON files(reviewed_at)",
"""
CREATE TABLE IF NOT EXISTS http_cache (
url TEXT PRIMARY KEY,
last_modified TEXT NOT NULL
)
""",
)
class SchemaTooNewError(RuntimeError):
"""Raised when the index was written by a newer version of this tool."""
def connect(path: Path) -> sqlite3.Connection:
"""Open *path* with the pragmas this package relies on.
WAL is what lets the downloader thread write while the UI thread reads and
writes verdicts, without either blocking the other.
Args:
path: Location of the sqlite file; its parent must already exist.
Returns:
sqlite3.Connection: Autocommit connection with row access by name.
"""
conn = sqlite3.connect(path, timeout=DB_TIMEOUT_S, isolation_level=None)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode=WAL")
conn.execute("PRAGMA synchronous=NORMAL")
conn.execute(f"PRAGMA busy_timeout={DB_BUSY_TIMEOUT_MS}")
conn.execute("PRAGMA foreign_keys=ON")
return conn
def ensure_schema(conn: sqlite3.Connection) -> int:
"""Create or migrate the schema, returning the version now in force.
Args:
conn: Open connection.
Returns:
int: The schema version after this call.
Raises:
SchemaTooNewError: If the file was written by a newer release, in which
case silently downgrading it would lose data.
"""
version = int(conn.execute("PRAGMA user_version").fetchone()[0])
if version > SCHEMA_VERSION:
msg = (
f"index schema is version {version} but this build understands "
f"at most {SCHEMA_VERSION}; upgrade the tool instead of "
f"downgrading the index"
)
raise SchemaTooNewError(msg)
if version == SCHEMA_VERSION:
return version
for statement in _SCHEMA:
conn.execute(statement)
# A fresh database already has the current shape from _SCHEMA, so most
# steps have no migration. Each one that does is idempotent, which keeps
# this safe either way.
for step in range(version, SCHEMA_VERSION):
migrate = _MIGRATIONS.get(step)
if migrate is not None:
migrate(conn)
conn.execute(f"PRAGMA user_version={SCHEMA_VERSION}")
return SCHEMA_VERSION
def _to_v2(conn: sqlite3.Connection) -> None:
"""Add the column that makes a verdict reversible after a restart.
Undo needs to know the name the file was given in keep/ or trash/, which
may differ from its name in incoming/ when a collision forced a rename.
Args:
conn: Open connection.
"""
columns = {row["name"] for row in conn.execute("PRAGMA table_info(files)")}
if "reviewed_name" not in columns:
conn.execute("ALTER TABLE files ADD COLUMN reviewed_name TEXT")
# Keyed by the version being upgraded FROM.
_MIGRATIONS: Final[dict[int, Callable[[sqlite3.Connection], None]]] = {
1: _to_v2,
}
def open_index(path: Path) -> sqlite3.Connection:
"""Open *path* and bring its schema up to date.
Args:
path: Location of the sqlite file.
Returns:
sqlite3.Connection: Ready-to-use connection.
"""
conn = connect(path)
ensure_schema(conn)
return conn

238
python_pkg/wsg_grabber/downloader.py
View File

@ -1,238 +0,0 @@
"""The background worker: walk the board, fetch files, announce each one.
Structured so that :meth:`Worker.run_once` performs exactly one unit of work
and returns. The threaded loop is a thin wrapper around it, which is what lets
the tests drive the whole state machine synchronously without starting a thread
-- important because pytest treats warnings as errors and a leaked thread would
fail the suite.
The worker never touches Tk. It publishes events onto a queue that the UI
drains from its own event loop.
"""
from __future__ import annotations
from dataclasses import dataclass, field
import logging
import sqlite3
import threading
import time
from typing import TYPE_CHECKING
import requests
from python_pkg.wsg_grabber import net, scanner, store
from python_pkg.wsg_grabber._fetch import DownloadMixin
from python_pkg.wsg_grabber._scan import ScanMixin
from python_pkg.wsg_grabber.constants import (
IDLE_POLL_S,
IDLE_RESCAN_S,
)
from python_pkg.wsg_grabber.models import (
JsonResponse,
ScanFinished,
TaskKind,
ThreadRef,
WorkerFailed,
)
_LOGGER = logging.getLogger(__name__)
# Every failure family this worker can plausibly hit: the network, the disk,
# the index, and bad data coming back from the board. Named explicitly rather
# than catching Exception, which the linters reject and which would also
# swallow genuine programming errors that ought to be loud.
_RECOVERABLE = (
ArithmeticError,
OSError,
RuntimeError,
ValueError,
LookupError,
TypeError,
AttributeError,
sqlite3.Error,
requests.RequestException,
)
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
import queue
from python_pkg.wsg_grabber.models import DownloadEvent
@dataclass
class WorkerDeps:
"""Collaborators supplied to the worker, so tests can swap them out."""
conn: sqlite3.Connection
session: requests.Session
incoming: Path
events: queue.SimpleQueue[DownloadEvent]
stop: threading.Event = field(default_factory=threading.Event)
include_archive: bool = True
class Worker(ScanMixin, DownloadMixin):
"""Walks /wsg/ and downloads every file it has not seen before.
The board-walking and file-fetching halves live in _scan and _fetch to
keep this module under the 250-line cap; they are mixins because the
tests drive their methods as bound attributes of a Worker.
"""
def __init__(self, deps: WorkerDeps) -> None:
"""Store collaborators and initialise scan bookkeeping.
Args:
deps: Connection, session, destination and event queue.
"""
self._deps = deps
self._pending_threads: list[ThreadRef] = []
self._catalog_fresh = False
self._downloaded = 0
self._last_request = 0.0
self._finished_announced = False
self._sweep_finished_at = 0.0
@property
def downloaded(self) -> int:
"""Return how many files this worker has fetched.
Returns:
int: Completed downloads since construction.
"""
return self._downloaded
def run(self) -> None:
"""Loop until the stop flag is set, reporting any crash to the UI.
Failures on a background thread would otherwise vanish, leaving the
reviewer waiting for downloads that never arrive, so anything in
``_RECOVERABLE`` is logged and surfaced in the status bar. The session
and connection are closed on every exit path. The suite does not police
that for us: ``meta/pyproject.toml`` downgrades
``PytestUnraisableExceptionWarning``, so a leak here would pass CI
unnoticed.
"""
try:
while not self._deps.stop.is_set():
self.run_once()
except _RECOVERABLE as exc:
_LOGGER.exception("wsg download worker stopped")
self._publish(WorkerFailed(message=str(exc)))
finally:
self._deps.session.close()
self._deps.conn.close()
def run_once(self) -> TaskKind:
"""Perform one unit of work and return what it was.
Returns:
TaskKind: The action taken, so a caller can drive the worker step
by step.
"""
task = scanner.choose_task(
store.pending_downloads(self._deps.conn),
len(self._pending_threads),
catalog_fresh=self._catalog_fresh,
)
if task is TaskKind.DOWNLOAD:
self._download_one()
elif task is TaskKind.SCAN:
self._scan_step()
else:
self._idle()
return task
def _idle(self) -> None:
"""Report the board exhausted, then poll again after a quiet period.
The rescan is deliberately not immediate. Resetting the catalogue every
idle tick would walk the board once a second and re-announce completion
each time; waiting keeps a finished sweep quiet until there is a real
chance of new content.
"""
self._announce_finished()
self._deps.stop.wait(IDLE_POLL_S)
if time.monotonic() - self._sweep_finished_at >= IDLE_RESCAN_S:
self._catalog_fresh = False
self._finished_announced = False
def _announce_finished(self) -> None:
"""Emit ScanFinished once per completed sweep of the board."""
if self._finished_announced:
return
self._finished_announced = True
self._sweep_finished_at = time.monotonic()
self._publish(ScanFinished())
def _get_json(self, url: str, stamp: str | None = None) -> JsonResponse:
"""Fetch JSON while respecting the one-request-per-second rule.
Args:
url: Absolute API URL.
stamp: Stored ``Last-Modified`` to send conditionally.
Returns:
JsonResponse: The decoded response.
"""
self._wait_for_slot()
return net.get_json(self._deps.session, url, stamp)
def _wait_for_slot(self) -> None:
"""Sleep just long enough to stay within the API's rate limit."""
now = time.monotonic()
delay = scanner.next_delay(self._last_request, now)
if delay > 0:
self._deps.stop.wait(delay)
self._last_request = time.monotonic()
def _publish(self, event: DownloadEvent) -> None:
"""Hand an event to the UI thread.
Args:
event: What happened.
"""
self._deps.events.put(event)
def start(
build_deps: Callable[[], WorkerDeps],
on_failure: Callable[[str], None] = lambda _message: None,
) -> threading.Thread:
"""Run a worker on its own non-daemon thread.
A *factory* rather than ready-made dependencies, because sqlite forbids
using a connection from any thread but the one that opened it. Building
them inside the thread keeps ownership and disposal on the same side of the
boundary; passing a pre-opened connection in raises ProgrammingError the
moment the worker closes it.
Args:
build_deps: Called on the new thread to open the connection, session
and everything else the worker owns.
on_failure: Called with a message when *build_deps* itself fails, so
the UI learns there will be no downloads.
Returns:
threading.Thread: The started thread; set the stop flag, then join it.
"""
def _run() -> None:
try:
deps = build_deps()
except _RECOVERABLE as exc:
# Worker.run reports its own failures, but a crash while building
# the connection and session happens before there is a Worker at
# all. Without this the reviewer waits on downloads forever and the
# only trace is threading.excepthook.
_LOGGER.exception("wsg worker could not start")
on_failure(str(exc))
return
Worker(deps).run()
thread = threading.Thread(target=_run, name="wsg-downloader", daemon=False)
thread.start()
return thread

80
python_pkg/wsg_grabber/files.py
View File

@ -1,80 +0,0 @@
"""The only code in this package that moves bytes around on disk.
Deliberately tiny, because everything interesting about a verdict was already
decided in :mod:`python_pkg.wsg_grabber.verdict`. The one ``unlink`` lives
here too, so that "what touches the disk" stays a one-module question; only
:mod:`python_pkg.wsg_grabber._prune` and the keep migration call it.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import logs
if TYPE_CHECKING:
from pathlib import Path
from python_pkg.wsg_grabber.models import FileMove
def existing_names(directory: Path) -> set[str]:
"""Return the filenames already present in *directory*.
Args:
directory: Directory to inspect; need not exist.
Returns:
set[str]: Names currently in the directory.
"""
if not directory.is_dir():
return set()
return {entry.name for entry in directory.iterdir()}
def apply_move(move: FileMove) -> bool:
"""Carry out a planned relocation.
Refuses to overwrite an existing destination. ``Path.replace`` clobbers
silently, so without this check a caller that forgot to pick a
collision-free name would destroy a video the user had already decided to
keep -- exactly the thing this package promises never to do. Callers pick
the name with :func:`verdict.unique_destination`; this is the backstop that
makes the guarantee structural rather than a convention.
Args:
move: Where the file is and where it should go.
Returns:
bool: True when the file was moved. False when the source had already
vanished, or when the destination is occupied -- the caller treats
either as "this file is not where I thought", never as success.
"""
if not move.src.exists():
return False
if move.dst.exists():
logs.error(
"files.destination_occupied",
src=str(move.src),
dst=str(move.dst),
)
return False
move.dst.parent.mkdir(parents=True, exist_ok=True)
move.src.replace(move.dst)
return True
def remove(path: Path) -> bool:
"""Delete a file for good.
Args:
path: File to delete.
Returns:
bool: True when a file was removed; False when nothing was there.
"""
try:
path.unlink()
except FileNotFoundError:
return False
return True

66
python_pkg/wsg_grabber/housekeeping.py
View File

@ -1,66 +0,0 @@
"""What the session does around a verdict besides the verdict itself.
Kept out of :mod:`python_pkg.wsg_grabber.app` so that module stays about
composition and shutdown: migrating the old ``keep/`` directory, pruning
``trash/``, and working out where a kept file actually is when undo asks.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import paths
from python_pkg.wsg_grabber._migrate_keep import migrate_keep
from python_pkg.wsg_grabber._prune import prune_trash
from python_pkg.wsg_grabber.models import Verdict
if TYPE_CHECKING:
from pathlib import Path
import sqlite3
from python_pkg.wsg_grabber.models import ReviewedItem
def on_startup(conn: sqlite3.Connection) -> None:
"""Bring the on-disk layout in line with the index before reviewing.
Args:
conn: Open connection.
"""
migrate_keep(conn)
prune_trash(conn)
def after_verdict(conn: sqlite3.Connection, choice: Verdict) -> None:
"""Keep ``trash/`` at its cap once a pass has landed there.
Args:
conn: Open connection.
choice: The verdict just recorded.
"""
if choice is not Verdict.KEEP:
prune_trash(conn)
def locate_reviewed(action: ReviewedItem) -> Path:
"""Return where the file a verdict moved is sitting now.
A pass has exactly one home. A keep may be in this year's cloud folder,
last year's, ``~/Downloads``, wherever the media sync swept it, or the
legacy ``keep/`` -- the first candidate that holds the name wins, and when
none does the current keep dir is returned so the caller's move fails
visibly rather than silently.
Args:
action: The verdict being undone.
Returns:
Path: Expected location of the file.
"""
if action.choice is not Verdict.KEEP:
return paths.trash_dir() / action.reviewed_name
for directory in paths.kept_candidates():
candidate = directory / action.reviewed_name
if candidate.is_file():
return candidate
return paths.keep_dir() / action.reviewed_name

83
python_pkg/wsg_grabber/install.sh
View File

@ -1,83 +0,0 @@
#!/bin/bash
# ============================================================================
# Install the /wsg/ grabber: system dependencies plus a launcher on PATH.
# ============================================================================
set -euo pipefail
SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_NAME
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
readonly REPO_ROOT
readonly LAUNCHER="/usr/local/bin/wsg-grabber"
# mpv provides the player; tk provides the window. Both are runtime-only, and
# the package deliberately depends on neither via pip: python-mpv resolves
# libmpv at import time and would break any machine without it.
readonly PACKAGES=(mpv tk)
usage() {
echo "Usage: $SCRIPT_NAME [--uninstall]"
echo " Installs mpv + tk and a 'wsg-grabber' launcher."
exit 0
}
require_root() {
if [[ $EUID -ne 0 ]]; then
echo "Error: needs root to write $LAUNCHER and install packages." >&2
echo "Re-run with: sudo $0" >&2
exit 1
fi
}
install_packages() {
local missing=()
local package
for package in "${PACKAGES[@]}"; do
if ! pacman -Qq "$package" &>/dev/null; then
missing+=("$package")
fi
done
if [[ ${#missing[@]} -eq 0 ]]; then
echo "System packages already present: ${PACKAGES[*]}"
return
fi
echo "Installing: ${missing[*]}"
pacman -S --needed --noconfirm "${missing[@]}"
}
write_launcher() {
cat > "$LAUNCHER" <<EOF
#!/bin/bash
# Launcher for the /wsg/ grabber. Generated by $SCRIPT_NAME.
set -euo pipefail
export PYTHONPATH="$REPO_ROOT\${PYTHONPATH:+:\$PYTHONPATH}"
exec python3 -m python_pkg.wsg_grabber "\$@"
EOF
chmod 755 "$LAUNCHER"
echo "Installed $LAUNCHER"
}
uninstall() {
rm -f "$LAUNCHER"
echo "Removed $LAUNCHER"
echo "Downloaded videos in ~/.local/share/wsg_grabber were left alone."
}
main() {
require_root
install_packages
write_launcher
echo
echo "Done. Try:"
echo " wsg-grabber stats"
echo " wsg-grabber # opens the reviewer"
}
case "${1:-}" in
-h|--help) usage ;;
--uninstall) require_root; uninstall ;;
"") main ;;
*) echo "Unknown option: $1" >&2; exit 1 ;;
esac

192
python_pkg/wsg_grabber/logs.py
View File

@ -1,192 +0,0 @@
r"""Structured, machine-readable event logging.
Every interesting thing this package does emits one JSON object per line, so a
session can be reconstructed afterwards without a debugger and without having
reproduced the problem live. That matters here more than usual: the parts most
likely to misbehave -- what mpv actually did with a command, whether a file was
still on disk when it was played -- sit exactly on the boundaries the test
suite has to mock, so tests cannot see them.
Reading a log back:
jq -c 'select(.event|startswith("player."))' <logfile>
jq -r 'select(.level=="error") | "\\(.ts) \\(.event) \\(.detail)"' <file>
jq -r 'select(.event=="review.show") | .file' <file> # what was shown, in order
Field names are stable. Every record carries ``ts``, ``seq``, ``level``,
``event`` and ``thread``; anything else is event-specific and always appears
under the same key.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from datetime import UTC, datetime
import itertools
import json
import os
import sys
import threading
from typing import TYPE_CHECKING, Final, TextIO
from python_pkg.wsg_grabber import paths
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
LEVELS: Final[tuple[str, ...]] = ("debug", "info", "warning", "error")
_RANK: Final[dict[str, int]] = {name: index for index, name in enumerate(LEVELS)}
@dataclass
class _Session:
"""Where records go and which ones are kept."""
sink: TextIO | None = None
path: Path | None = None
threshold: int = _RANK["info"]
echo: bool = False
lock: threading.Lock = field(default_factory=threading.Lock)
counter: Iterator[int] = field(default_factory=lambda: itertools.count(1))
_SESSION = _Session()
def _free_path(directory: Path, stem: str) -> Path:
"""Return a log path that does not already exist.
The stem carries a whole-second timestamp and the pid, so restarting twice
inside one second would otherwise append a second session to the first
one's file and make the record numbering look corrupt.
Args:
directory: Where logs live.
stem: Preferred filename without its extension.
Returns:
Path: An unused path.
"""
candidate = directory / f"{stem}.jsonl"
suffix = 2
while candidate.exists():
candidate = directory / f"{stem}-{suffix}.jsonl"
suffix += 1
return candidate
def logs_dir() -> Path:
"""Return the directory holding session logs.
Returns:
Path: ``<data dir>/logs``.
"""
return paths.data_dir() / "logs"
def current_path() -> Path | None:
"""Return the log file this process is writing to.
Returns:
Path | None: The path, or None when logging has not been started.
"""
return _SESSION.path
def start(level: str = "info", *, echo: bool = False) -> Path:
"""Begin writing a session log and return its path.
Args:
level: Lowest level to record; one of :data:`LEVELS`.
echo: Also mirror records to stderr, for watching a run live.
Returns:
Path: The file now being written.
"""
stop()
directory = logs_dir()
directory.mkdir(parents=True, exist_ok=True)
stamp = datetime.now(tz=UTC).strftime("%Y%m%dT%H%M%S")
path = _free_path(directory, f"session-{stamp}-{os.getpid()}")
with _SESSION.lock:
_SESSION.sink = path.open("a", encoding="utf-8")
_SESSION.path = path
_SESSION.threshold = _RANK.get(level, _RANK["info"])
_SESSION.echo = echo
# Restart numbering: seq orders records within one file, so carrying it
# over from a previous session would make the first line read as if
# records had been lost.
_SESSION.counter = itertools.count(1)
event("session.start", level_wanted=level, log=str(path), pid=os.getpid())
return path
def stop() -> None:
"""Close the session log, if one is open."""
with _SESSION.lock:
if _SESSION.sink is not None:
_SESSION.sink.flush()
_SESSION.sink.close()
_SESSION.sink = None
_SESSION.path = None
def event(name: str, level: str = "info", **fields: object) -> None:
"""Record one event.
Args:
name: Dotted event name, e.g. ``player.command``.
level: One of :data:`LEVELS`.
**fields: Event-specific values. Anything not JSON-encodable is
stringified rather than dropped, so a log line is never lost to a
serialisation error.
"""
if _RANK.get(level, _RANK["info"]) < _SESSION.threshold:
return
record = {
"ts": datetime.now(tz=UTC).isoformat(),
"seq": next(_SESSION.counter),
"level": level,
"event": name,
"thread": threading.current_thread().name,
**fields,
}
line = json.dumps(record, default=str)
with _SESSION.lock:
if _SESSION.sink is not None:
_SESSION.sink.write(f"{line}\n")
_SESSION.sink.flush()
if _SESSION.echo:
sys.stderr.write(f"{line}\n")
def debug(name: str, **fields: object) -> None:
"""Record a debug-level event.
Args:
name: Dotted event name.
**fields: Event-specific values.
"""
event(name, level="debug", **fields)
def warning(name: str, **fields: object) -> None:
"""Record a warning-level event.
Args:
name: Dotted event name.
**fields: Event-specific values.
"""
event(name, level="warning", **fields)
def error(name: str, **fields: object) -> None:
"""Record an error-level event.
Args:
name: Dotted event name.
**fields: Event-specific values.
"""
event(name, level="error", **fields)

205
python_pkg/wsg_grabber/models.py
View File

@ -1,205 +0,0 @@
"""Plain-data records shared across the package.
Everything crossing a module boundary is a frozen dataclass or an enum, so the
pure planning code can be driven from hand-built literals and the GUI never has
to be constructed to test a decision.
"""
from __future__ import annotations
from dataclasses import dataclass
from enum import StrEnum
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from pathlib import Path
class Verdict(StrEnum):
"""What the user decided about a video.
The member is ``SKIP`` rather than ``PASS`` only because ruff's S105 reads a
name of ``PASS`` as a hardcoded password and this repo forbids ``noqa``. The
stored value stays ``"pass"``, which is the word the UI shows.
"""
KEEP = "keep"
SKIP = "pass"
class TaskKind(StrEnum):
"""What the background worker should do next."""
DOWNLOAD = "download"
SCAN = "scan"
IDLE = "idle"
class Emptiness(StrEnum):
"""Why the reviewer has nothing to show."""
NOT_EMPTY = "not_empty"
WAITING = "waiting"
EXHAUSTED = "exhausted"
@dataclass(frozen=True, slots=True)
class Size:
"""A pixel size."""
width: int
height: int
@dataclass(frozen=True, slots=True)
class ThreadRef:
"""A thread as advertised by ``threads.json``."""
thread_no: int
api_last_modified: int
@dataclass(frozen=True, slots=True)
class RemoteFile:
"""An attachment described by a post, before anything is downloaded.
``md5`` is the API's own ``base64(md5(file_bytes))`` and is the identity of
the file everywhere in this package.
"""
md5: str
tim: int
ext: str
orig_name: str
fsize: int
width: int
height: int
thread_no: int
post_no: int
@dataclass(frozen=True, slots=True)
class ReviewItem:
"""A downloaded video waiting for a verdict."""
md5: str
path: Path
orig_name: str
fsize: int
width: int
height: int
@dataclass(frozen=True, slots=True)
class FileMove:
"""A planned relocation. Applying it is the only filesystem write."""
md5: str
src: Path
dst: Path
@dataclass(frozen=True, slots=True)
class ReviewedItem:
"""A verdict recorded in the index, and therefore reversible.
``reviewed_name`` is the name the file was given in keep/ or trash/, which
differs from ``item.path.name`` whenever a collision forced a rename. Undo
needs it to find the file again, which is why it is persisted rather than
held in memory.
"""
item: ReviewItem
choice: Verdict
reviewed_name: str
@dataclass(frozen=True, slots=True)
class ResumePlan:
"""How to continue a partial download."""
offset: int
discard: bool
class Outcome(StrEnum):
"""How a download ended."""
COMPLETED = "completed"
NOT_FOUND = "not_found"
TRANSIENT = "transient"
CHECKSUM_MISMATCH = "checksum_mismatch"
ABORTED = "aborted"
@dataclass(frozen=True, slots=True)
class JsonResponse:
"""The result of a conditional GET against the API."""
payload: object | None
last_modified: str | None
not_modified: bool
not_found: bool
@dataclass(frozen=True, slots=True)
class DownloadResult:
"""The result of streaming one file to disk."""
outcome: Outcome
bytes_done: int
retry_after: float | None = None
message: str | None = None
@dataclass(frozen=True, slots=True)
class ReviewCommand:
"""Everything the GUI must render or do, already decided.
The window applies this verbatim; it never formats a string or picks a file.
"""
play: Path | None
stop: bool
status: str
title: str
filename: str
verdicts_enabled: bool
undo_enabled: bool
quit_app: bool
@dataclass(frozen=True, slots=True)
class FileReady:
"""A newly downloaded video is available for review."""
item: ReviewItem
@dataclass(frozen=True, slots=True)
class Indexed:
"""The catalogue scan learned about more files."""
known: int
@dataclass(frozen=True, slots=True)
class Downloaded:
"""Running count of files fetched this session."""
count: int
@dataclass(frozen=True, slots=True)
class ScanFinished:
"""The board has been walked and every known file downloaded."""
@dataclass(frozen=True, slots=True)
class WorkerFailed:
"""The background worker died; the message is shown in the status bar."""
message: str
DownloadEvent = FileReady | Indexed | Downloaded | ScanFinished | WorkerFailed

109
python_pkg/wsg_grabber/net.py
View File

@ -1,109 +0,0 @@
"""The only module that speaks HTTP.
Two hosts with different manners. ``a.4cdn.org`` is polite and supports
``If-Modified-Since``, so a steady-state rescan costs almost nothing.
``i.4cdn.org`` answered a bare request with 429 during development, and the
cookies it sets in reply (``__cf_bm``, ``_cfuvid``) live in the session jar --
which is why one long-lived :class:`requests.Session` is required rather than
merely convenient.
"""
from __future__ import annotations
import json
import requests
# Re-exported through __all__ so net.download / net.Transfer / net.digest_of /
# net.size_of keep working for downloader.py and test_net.py; the transfer
# machinery lives in _download to keep this module under the 250-line cap.
from python_pkg.wsg_grabber._download import (
Transfer,
digest_of,
download,
size_of,
)
from python_pkg.wsg_grabber.constants import (
API_HEADERS,
CONNECT_TIMEOUT_S,
MAX_JSON_BYTES,
READ_TIMEOUT_S,
)
from python_pkg.wsg_grabber.models import JsonResponse
__all__ = [
"Transfer",
"build_session",
"digest_of",
"download",
"get_json",
"size_of",
]
_TIMEOUT = (CONNECT_TIMEOUT_S, READ_TIMEOUT_S)
_OK = 200
_PARTIAL = 206
_NOT_MODIFIED = 304
_NOT_FOUND = 404
_GONE = 410
_RANGE_NOT_SATISFIABLE = 416
_TOO_MANY = 429
_SERVER_ERROR = 500
def build_session() -> requests.Session:
"""Return a session configured for the API host.
Returns:
requests.Session: Caller owns it and must close it.
"""
session = requests.Session()
session.headers.update(API_HEADERS)
return session
def get_json(
session: requests.Session,
url: str,
last_modified: str | None = None,
) -> JsonResponse:
"""Fetch and decode a JSON endpoint, conditionally when possible.
Args:
session: Shared session.
url: Absolute API URL.
last_modified: Previously seen ``Last-Modified`` value, echoed back as
``If-Modified-Since`` so an unchanged thread costs a 304.
Returns:
JsonResponse: Decoded payload, or a flag explaining why there is none.
"""
headers = {"If-Modified-Since": last_modified} if last_modified else {}
response = session.get(url, headers=headers, timeout=_TIMEOUT)
status = response.status_code
if status == _NOT_MODIFIED:
return JsonResponse(
payload=None,
last_modified=last_modified,
not_modified=True,
not_found=False,
)
if status in (_NOT_FOUND, _GONE):
return JsonResponse(
payload=None,
last_modified=None,
not_modified=False,
not_found=True,
)
response.raise_for_status()
body = response.content[: MAX_JSON_BYTES + 1]
if len(body) > MAX_JSON_BYTES:
msg = f"API response exceeded {MAX_JSON_BYTES} bytes"
raise ValueError(msg)
return JsonResponse(
payload=json.loads(body),
last_modified=response.headers.get("Last-Modified"),
not_modified=False,
not_found=False,
)

160
python_pkg/wsg_grabber/paths.py
View File

@ -1,160 +0,0 @@
"""Storage locations, resolved lazily.
Every path is a *function*. A module-level ``Path.home() / ...`` constant is
evaluated at import time, which makes it impossible for a test to redirect --
``brother_printer.consumables`` learned that the hard way. Resolving on each
call keeps the whole package redirectable from one fixture.
Kept videos leave the data dir entirely: they go to the dufs cloud folder
(``~/data/cloud/Media/<year>/wsg``) so they show up in the gallery and on the
phone, or to ``~/Downloads`` on a machine without the cloud folder. Both are
resolved per call, so the answer follows the filesystem rather than the process
start time.
Nothing in this module deletes anything; ``trash_dir`` is a destination.
:mod:`python_pkg.wsg_grabber._prune` is what keeps it bounded.
"""
from __future__ import annotations
from datetime import UTC, datetime
import os
from pathlib import Path
from python_pkg.wsg_grabber.constants import BOARD
_APP_DIRNAME = "wsg_grabber"
def data_dir() -> Path:
"""Return the root directory holding the index and every downloaded file.
Honours ``XDG_DATA_HOME`` when set, else ``~/.local/share``.
Returns:
Path: ``<data home>/wsg_grabber`` (not guaranteed to exist).
"""
override = os.environ.get("XDG_DATA_HOME")
base = Path(override) if override else Path.home() / ".local" / "share"
return base / _APP_DIRNAME
def db_path() -> Path:
"""Return the sqlite index file.
Returns:
Path: ``<data dir>/index.db``.
"""
return data_dir() / "index.db"
def incoming_dir() -> Path:
"""Return the directory holding downloaded, not-yet-reviewed videos.
Returns:
Path: ``<data dir>/incoming``.
"""
return data_dir() / "incoming"
def cloud_root() -> Path:
"""Return the folder dufs serves as the personal cloud.
Returns:
Path: ``~/data/cloud`` (may not exist on this machine).
"""
return Path.home() / "data" / "cloud"
def downloads_dir() -> Path:
"""Return the fallback destination for kept videos.
Returns:
Path: ``~/Downloads``.
"""
return Path.home() / "Downloads"
def cloud_keep_dir(year: int) -> Path:
"""Return the cloud folder kept videos go to for *year*.
Args:
year: Calendar year of the keep verdict.
Returns:
Path: ``<cloud>/Media/<year>/wsg``.
"""
return cloud_root() / "Media" / str(year) / "wsg"
def keep_dir() -> Path:
"""Return the directory videos are moved to when kept.
The cloud folder wins whenever it exists; otherwise ``~/Downloads``, which
is at least somewhere a person looks.
Returns:
Path: ``<cloud>/Media/<this year>/wsg`` or ``~/Downloads``.
"""
if cloud_root().is_dir():
return cloud_keep_dir(datetime.now(tz=UTC).astimezone().year)
return downloads_dir()
def legacy_keep_dir() -> Path:
"""Return where kept videos used to go before they moved to the cloud.
Only :mod:`python_pkg.wsg_grabber._migrate_keep` should care.
Returns:
Path: ``<data dir>/keep``.
"""
return data_dir() / "keep"
def kept_candidates() -> list[Path]:
"""Return every directory a kept video may be sitting in, most likely first.
Undo has to find a file that ``keep_dir()`` would not name today: a keep
from last year, one made while the cloud folder was absent, one the
media-cloud-sync timer has since swept out of ``~/Downloads`` into
``Media/<year>/<month>``, or one still in the pre-cloud ``keep/``.
Returns:
list[Path]: Candidate directories; entries need not exist.
"""
media = cloud_root() / "Media"
return [
keep_dir(),
*sorted(media.glob("*/wsg")),
legacy_keep_dir(),
downloads_dir(),
*sorted(p for p in media.glob("*/*") if p.is_dir() and p.name != "wsg"),
]
def trash_dir() -> Path:
"""Return the directory videos are moved to when passed.
Bounded to the newest ``TRASH_RETAIN`` passes by
:mod:`python_pkg.wsg_grabber._prune`; older ones are deleted for good.
Returns:
Path: ``<data dir>/trash``.
"""
return data_dir() / "trash"
def ipc_socket_path() -> Path:
"""Return the unix socket used to drive the embedded mpv process.
Returns:
Path: ``<data dir>/mpv-<board>.sock``.
"""
return data_dir() / f"mpv-{BOARD}.sock"
def ensure_dirs() -> None:
"""Create the data, incoming, keep and trash directories if absent."""
for directory in (data_dir(), incoming_dir(), keep_dir(), trash_dir()):
directory.mkdir(parents=True, exist_ok=True)

235
python_pkg/wsg_grabber/player.py
View File

@ -1,235 +0,0 @@
"""Drives mpv as a subprocess embedded in a Tk frame, over its JSON IPC socket.
Not ``python-mpv``: that binding calls ``ctypes.util.find_library("mpv")`` at
import time and raises ``OSError`` wherever libmpv is absent, which includes
this repo's CI runner. It would pass locally and fail on push, and the usual
lazy-import escape trips ``PLC0415`` with no suppression available. A
subprocess needs only ``subprocess``, ``socket`` and ``json``.
``--wid`` is an mpv feature, so embedding behaves identically either way: mpv
reparents its own window onto the Tk frame and, per ``man mpv``, "will always be
resized to cover the parent window fully". That last part is why there is no
resize method here: setting ``geometry`` on an embedded mpv fights the
automatic sizing and leaves the video drawn at the wrong size.
``build_argv`` and ``encode`` are pure and tested directly. The rest is process
and socket handling: connecting, draining, and shutting down without leaving an
orphan. No reviewing decisions are made here.
"""
from __future__ import annotations
import socket
import subprocess
import threading
import time
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import logs
# Re-exported through __all__ so player.Player / player.build_argv /
# player.encode keep working for ui.py and test_player.py; the wire format and
# the protocol live in their own modules to keep this one under the 250-line cap.
from python_pkg.wsg_grabber._mpv_ipc import (
build_argv,
encode,
note_errors,
probe_properties,
)
from python_pkg.wsg_grabber._player_api import Player
from python_pkg.wsg_grabber.constants import IPC_READY_TIMEOUT_S
__all__ = ["MpvPlayer", "Player", "build_argv", "encode"]
if TYPE_CHECKING:
from collections.abc import Sequence
from pathlib import Path
_SOCKET_POLL_S = 0.05
_READ_CHUNK = 65536
_READ_TIMEOUT_S = 0.2
_READER_JOIN_S = 5.0
class MpvPlayer:
"""An mpv process drawing into a Tk frame."""
def __init__(self, wid: int, ipc_path: Path) -> None:
"""Start mpv and connect to its control socket.
Args:
wid: X11 window id of the frame to draw into.
ipc_path: Where mpv should create its control socket.
Raises:
TimeoutError: If mpv does not create the socket in time, which
means it failed to start and there is nothing to talk to.
OSError: If the socket exists but refuses the connection, e.g. a
stale file left by a previous run.
"""
ipc_path.unlink(missing_ok=True)
ipc_path.parent.mkdir(parents=True, exist_ok=True)
self._ipc_path = ipc_path
argv = build_argv(wid, ipc_path)
logs.event("player.start", wid=wid, socket=str(ipc_path), argv=argv)
self._process = subprocess.Popen(argv)
try:
self._socket = self._connect()
except BaseException:
# Reaping matters as much as killing: an unwaited child stays a
# zombie, and the caller is about to abandon this object entirely.
self._process.kill()
self._process.wait(timeout=IPC_READY_TIMEOUT_S)
raise
self._stopping = threading.Event()
self._reader = threading.Thread(
target=self._drain,
name="mpv-ipc-reader",
daemon=False,
)
self._reader.start()
logs.event("player.ready", pid=self._process.pid)
def _drain(self) -> None:
"""Continuously read and discard what mpv sends us.
This is not optional bookkeeping, it is what keeps the player working.
mpv pushes an asynchronous event stream plus a reply per command down
the same socket. A client that only ever writes lets that backlog fill
the kernel buffer; mpv's writer for this connection then blocks, and it
stops reading our commands. The symptom is nasty because nothing
errors: ``sendall`` still succeeds, mpv stays alive and keeps playing
whatever it had, so the reviewer advances through filenames while the
picture never changes. Measured before this existed: commands were
silently ignored from roughly the thirty-fifth video onwards, while the
very same command sent on a fresh connection worked instantly.
Replies carrying an error are logged, which is also how a file mpv
cannot decode becomes visible instead of just looking stuck.
"""
while not self._stopping.is_set():
try:
chunk = self._socket.recv(_READ_CHUNK)
except TimeoutError:
continue
except OSError:
return
if not chunk:
return
self._note_errors(chunk)
def _note_errors(self, chunk: bytes) -> None:
"""Log any failure mpv reported back.
Delegates to :func:`._mpv_ipc.note_errors`; kept as a bound method
because the reader thread and the tests both reach it that way.
Args:
chunk: Raw bytes just read from the socket.
"""
note_errors(chunk)
def _connect(self) -> socket.socket:
"""Wait for mpv's socket to appear and connect to it.
Returns:
socket.socket: Connected unix socket.
Raises:
TimeoutError: If the socket never appears.
OSError: If connecting to it fails.
"""
deadline = time.monotonic() + IPC_READY_TIMEOUT_S
while not self._ipc_path.exists():
if time.monotonic() >= deadline:
# __init__ owns killing and reaping the process on every
# failure path; doing it here too would just double up.
msg = f"mpv did not create {self._ipc_path} within the timeout"
raise TimeoutError(msg)
time.sleep(_SOCKET_POLL_S)
control = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
try:
control.connect(str(self._ipc_path))
except BaseException:
control.close()
raise
# so the reader thread can notice the stop flag rather than
# blocking in recv() forever at shutdown
control.settimeout(_READ_TIMEOUT_S)
return control
def _send(self, command: Sequence[object]) -> None:
"""Write a command to mpv.
A failure here is logged rather than raised -- mpv may simply have
exited, and is_alive() reports that to the caller -- but it is never
silently discarded, because a dead socket makes a working reviewer and
a frozen one look identical from the outside.
Args:
command: Command name followed by its arguments.
"""
try:
self._socket.sendall(encode(command))
except OSError as exc:
logs.error(
"player.send_failed",
command=list(command),
detail=str(exc),
alive=self.is_alive(),
)
return
logs.debug("player.command", command=list(command))
def play(self, path: Path) -> None:
"""Show *path*, replacing whatever is on screen.
Args:
path: Video file to play.
"""
logs.event("player.load", file=path.name, path=str(path), exists=path.exists())
self._send(["loadfile", str(path), "replace"])
def stop(self) -> None:
"""Blank the video area."""
self._send(["stop"])
def probe(self) -> dict[str, object]:
"""Ask mpv what it is actually doing.
This exists for after-the-fact diagnosis. If the reviewer ever appears
to freeze on one video while the filenames advance, the recorded answer
settles in one line whether mpv was handed the new file and was
decoding it, or whether the problem is upstream of mpv.
Returns:
dict[str, object]: Property name to value; a property that could
not be read maps to a string starting with ``error:``.
"""
return probe_properties(self._ipc_path)
def is_alive(self) -> bool:
"""Report whether mpv is still running.
Returns:
bool: True while the process lives.
"""
return self._process.poll() is None
def close(self) -> None:
"""Ask mpv to quit, then make sure it has.
Leaving an orphaned mpv behind would keep a window on screen after the
reviewer exits, so a process that ignores ``quit`` is killed.
"""
logs.event("player.close", alive=self.is_alive())
self._stopping.set()
self._send(["quit"])
try:
self._process.wait(timeout=IPC_READY_TIMEOUT_S)
except subprocess.TimeoutExpired:
self._process.kill()
self._process.wait(timeout=IPC_READY_TIMEOUT_S)
self._reader.join(timeout=_READER_JOIN_S)
self._socket.close()
self._ipc_path.unlink(missing_ok=True)

196
python_pkg/wsg_grabber/review.py
View File

@ -1,196 +0,0 @@
"""The reviewer's behaviour, with no toolkit anywhere in sight.
This module is the whole application as far as behaviour is concerned. It
decides what plays next, what a verdict does, when the queue has run dry and
exactly what the status bar reads. :mod:`python_pkg.wsg_grabber.ui` only
applies the :class:`ReviewCommand` this produces, so every decision is testable
from plain values and none of them need a display.
"""
from __future__ import annotations
from dataclasses import dataclass, replace
from typing import TYPE_CHECKING
# Re-exported through __all__ so review.render / review.status_line and friends
# keep working for ui.py and test_review.py; the implementations live in
# _render to keep this module under the 250-line cap. _render only needs
# SessionState for typing, so this import is not circular at runtime.
from python_pkg.wsg_grabber._render import (
emptiness,
filename_line,
render,
status_line,
title_line,
)
from python_pkg.wsg_grabber._undo import can_undo, forget_last, on_undo
from python_pkg.wsg_grabber.models import (
Downloaded,
FileReady,
Indexed,
ScanFinished,
Verdict,
)
if TYPE_CHECKING:
from collections.abc import Sequence
from python_pkg.wsg_grabber.models import DownloadEvent, ReviewItem
__all__ = [
"SessionState",
"can_undo",
"emptiness",
"filename_line",
"forget_last",
"initial_state",
"on_event",
"on_missing_locally",
"on_new_files",
"on_quit",
"on_undo",
"on_verdict",
"render",
"status_line",
"title_line",
]
@dataclass(frozen=True, slots=True)
class SessionState:
"""Everything the reviewer knows, as one immutable value."""
pending: tuple[ReviewItem, ...] = ()
current: ReviewItem | None = None
kept: int = 0
passed: int = 0
indexed: int = 0
downloaded: int = 0
scan_complete: bool = False
quitting: bool = False
error: str | None = None
undoable: int = 0
"""Verdicts recorded in the index that could still be taken back."""
def initial_state(
indexed: int,
ready: Sequence[ReviewItem],
undoable: int = 0,
) -> SessionState:
"""Build the state a session starts from.
Args:
indexed: How many files the index already knows about.
ready: Videos downloaded on a previous run and still unreviewed.
undoable: Verdicts already in the index that can still be reversed, so
undo carries across restarts rather than starting empty.
Returns:
SessionState: Starting state, already showing the first video if any.
"""
state = SessionState(indexed=indexed, undoable=undoable)
return on_new_files(state, ready)
def on_new_files(state: SessionState, items: Sequence[ReviewItem]) -> SessionState:
"""Add freshly downloaded videos to the queue.
The first arrival is promoted straight to the screen, which is what lets
watching start while the rest of the board is still downloading.
Args:
state: Current state.
items: Newly available videos.
Returns:
SessionState: Updated state.
"""
if not items:
return state
queue = [*state.pending, *items]
current = state.current
if current is None:
current = queue.pop(0)
return replace(state, pending=tuple(queue), current=current)
def on_verdict(state: SessionState, choice: Verdict) -> SessionState:
"""Record a decision about the current video and advance.
Args:
state: Current state.
choice: What the user pressed.
Returns:
SessionState: Updated state showing the next video.
"""
if state.current is None:
return state
counted = (
replace(state, kept=state.kept + 1)
if choice is Verdict.KEEP
else replace(state, passed=state.passed + 1)
)
return _advance(replace(counted, undoable=state.undoable + 1))
def on_missing_locally(state: SessionState) -> SessionState:
"""Drop the current video because its file has disappeared.
Args:
state: Current state.
Returns:
SessionState: Updated state showing the next video.
"""
if state.current is None:
return state
return _advance(state)
def on_event(state: SessionState, event: DownloadEvent) -> SessionState:
"""Fold one background-worker event into the state.
Args:
state: Current state.
event: What the worker reported.
Returns:
SessionState: Updated state.
"""
if isinstance(event, FileReady):
return on_new_files(state, [event.item])
if isinstance(event, Indexed):
return replace(state, indexed=event.known)
if isinstance(event, Downloaded):
return replace(state, downloaded=event.count)
if isinstance(event, ScanFinished):
return replace(state, scan_complete=True)
return replace(state, error=event.message)
def on_quit(state: SessionState) -> SessionState:
"""Mark the session as finishing.
Args:
state: Current state.
Returns:
SessionState: Updated state.
"""
return replace(state, quitting=True)
def _advance(state: SessionState) -> SessionState:
"""Move to the next queued video.
Args:
state: State whose current item has been dealt with.
Returns:
SessionState: Updated state.
"""
queue = list(state.pending)
following = queue.pop(0) if queue else None
return replace(state, pending=tuple(queue), current=following)

147
python_pkg/wsg_grabber/scanner.py
View File

@ -1,147 +0,0 @@
"""Pure planning helpers for the background worker.
Nothing here performs IO, so every scheduling and retry decision is testable
from plain values. The worker in :mod:`python_pkg.wsg_grabber.downloader` is a
thin shell that calls these and then acts.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.constants import (
BACKOFF_BASE_S,
BACKOFF_CAP_S,
MIN_REQUEST_INTERVAL_S,
)
from python_pkg.wsg_grabber.models import ResumePlan, TaskKind
if TYPE_CHECKING:
from collections.abc import Mapping, Sequence
from python_pkg.wsg_grabber.models import ThreadRef
def choose_task(
pending_downloads: int,
stale_thread_count: int,
*,
catalog_fresh: bool,
) -> TaskKind:
"""Decide what the worker should do next.
Downloads win over scanning so the reviewer gets its first playable video
as soon as possible, which is the whole point of loading in the background.
Args:
pending_downloads: Files known but not yet fetched.
stale_thread_count: Threads whose contents may have changed.
catalog_fresh: Whether the thread list has been read at least once this
cycle.
Returns:
TaskKind: The next action.
"""
if pending_downloads > 0:
return TaskKind.DOWNLOAD
if stale_thread_count > 0 or not catalog_fresh:
return TaskKind.SCAN
return TaskKind.IDLE
def stale_threads(
known: Mapping[int, int],
live: Sequence[ThreadRef],
) -> list[ThreadRef]:
"""Return threads worth fetching.
A thread is worth fetching when it has never been seen, or when the board
reports a newer ``last_modified`` than the one already stored. On a steady
board this reduces a cycle to a single request.
Args:
known: Thread number mapped to the stored timestamp.
live: Threads currently advertised by the board.
Returns:
list[ThreadRef]: Threads to fetch, in the order given.
"""
return [
ref
for ref in live
if ref.thread_no not in known or ref.api_last_modified > known[ref.thread_no]
]
def next_delay(last_request: float, now: float) -> float:
"""Return how long to wait before the next API request.
The 4chan API rules ask for at most one request per second.
Args:
last_request: Monotonic timestamp of the previous request, or 0.
now: Current monotonic timestamp.
Returns:
float: Seconds to sleep, never negative.
"""
elapsed = now - last_request
remaining = MIN_REQUEST_INTERVAL_S - elapsed
return max(0.0, remaining)
def backoff_seconds(attempt: int, retry_after: float | None = None) -> float:
"""Return how long to pause after a rejected request.
An explicit ``Retry-After`` always wins; otherwise the delay doubles per
attempt up to a cap so a sustained block does not spin.
Args:
attempt: 1 for the first failure.
retry_after: Server-supplied delay in seconds, when present.
Returns:
float: Seconds to wait, capped.
"""
if retry_after is not None and retry_after > 0:
return min(retry_after, BACKOFF_CAP_S)
exponent = max(0, attempt - 1)
return min(BACKOFF_BASE_S * (2.0**exponent), BACKOFF_CAP_S)
def parse_retry_after(header: str | None) -> float | None:
"""Read a ``Retry-After`` header expressed in seconds.
The HTTP-date form is deliberately not handled; falling back to exponential
backoff is safe and avoids a clock-skew dependency.
Args:
header: Raw header value, if any.
Returns:
float | None: Seconds to wait, or None when absent or not numeric.
"""
if header is None:
return None
try:
value = float(header.strip())
except ValueError:
return None
return value if value >= 0 else None
def resume_plan(part_size: int, expected_size: int) -> ResumePlan:
"""Decide how to continue a partially downloaded file.
Args:
part_size: Bytes already on disk in the ``.part`` file.
expected_size: Size the API says the file should be.
Returns:
ResumePlan: Byte offset to request, and whether to bin what is there.
"""
if part_size <= 0:
return ResumePlan(offset=0, discard=False)
if 0 < expected_size <= part_size:
return ResumePlan(offset=0, discard=True)
return ResumePlan(offset=part_size, discard=False)

148
python_pkg/wsg_grabber/states.py
View File

@ -1,148 +0,0 @@
"""The lifecycle of a single file, as a pure transition function.
``GONE`` is the load-bearing state. A full first pass over /wsg/ takes hours, so
threads fall off page 10 and files 404 between being catalogued and being
fetched. Without a terminal sink those files would be retried on every future
run forever.
"""
from __future__ import annotations
from enum import StrEnum
from typing import Final
from python_pkg.wsg_grabber.constants import MAX_ATTEMPTS
# ``str, Enum`` not ``StrEnum``: mypy targets 3.10, where ``StrEnum`` is absent
# and would collapse to ``Any``, disabling checking on every member below.
class FileState(StrEnum):
"""Where a file is in its lifecycle."""
NEW = "new"
DOWNLOADING = "downloading"
READY = "ready"
FAILED = "failed"
CORRUPT = "corrupt"
KEPT = "kept"
PASSED = "passed"
GONE = "gone"
# A pass whose file has since been pruned from trash/: still known, so it
# is never downloaded again, but no longer undoable.
PURGED = "purged"
class FileEvent(StrEnum):
"""Something that happened to a file."""
CLAIMED = "claimed"
COMPLETED = "completed"
NOT_FOUND = "not_found"
DELETED_UPSTREAM = "deleted_upstream"
TRANSIENT_ERROR = "transient_error"
CHECKSUM_MISMATCH = "checksum_mismatch"
KEPT = "kept"
PASSED = "passed"
MISSING_LOCALLY = "missing_locally"
TERMINAL: Final[frozenset[FileState]] = frozenset(
{FileState.KEPT, FileState.PASSED, FileState.GONE, FileState.PURGED},
)
CLAIMABLE: Final[frozenset[FileState]] = frozenset(
{FileState.NEW, FileState.FAILED, FileState.CORRUPT},
)
def is_terminal(state: FileState) -> bool:
"""Report whether *state* can never change again.
Args:
state: State to classify.
Returns:
bool: True when no further transition is possible.
"""
return state in TERMINAL
def next_state(current: FileState, event: FileEvent, attempts: int) -> FileState:
"""Return the state *current* moves to on *event*.
Args:
current: The file's present state.
event: What happened.
attempts: Download attempts made so far, including the one that just
failed. Retryable failures give up once this reaches
``MAX_ATTEMPTS``.
Returns:
FileState: The new state.
Raises:
ValueError: If the transition is not legal, which means a caller has a
bug rather than the board behaving oddly.
"""
if current in CLAIMABLE and event is FileEvent.CLAIMED:
return FileState.DOWNLOADING
if current in CLAIMABLE and event is FileEvent.DELETED_UPSTREAM:
return FileState.GONE
if current is FileState.DOWNLOADING:
return _from_downloading(event, attempts)
if current is FileState.READY:
return _from_ready(event)
msg = f"illegal transition: {current.value} + {event.value}"
raise ValueError(msg)
def _from_downloading(event: FileEvent, attempts: int) -> FileState:
"""Resolve a transition out of ``DOWNLOADING``.
Args:
event: What happened.
attempts: Download attempts made so far.
Returns:
FileState: The new state.
Raises:
ValueError: If *event* cannot follow ``DOWNLOADING``.
"""
if event is FileEvent.COMPLETED:
return FileState.READY
if event is FileEvent.NOT_FOUND:
return FileState.GONE
exhausted = attempts >= MAX_ATTEMPTS
if event is FileEvent.TRANSIENT_ERROR:
return FileState.GONE if exhausted else FileState.FAILED
if event is FileEvent.CHECKSUM_MISMATCH:
return FileState.GONE if exhausted else FileState.CORRUPT
msg = f"illegal transition: downloading + {event.value}"
raise ValueError(msg)
def _from_ready(event: FileEvent) -> FileState:
"""Resolve a transition out of ``READY``.
Upstream deletion is deliberately not handled here: once the bytes are on
disk it no longer matters that the post was removed.
Args:
event: What happened.
Returns:
FileState: The new state.
Raises:
ValueError: If *event* cannot follow ``READY``.
"""
if event is FileEvent.KEPT:
return FileState.KEPT
if event is FileEvent.PASSED:
return FileState.PASSED
if event is FileEvent.MISSING_LOCALLY:
return FileState.GONE
msg = f"illegal transition: ready + {event.value}"
raise ValueError(msg)

202
python_pkg/wsg_grabber/store.py
View File

@ -1,202 +0,0 @@
"""The download lifecycle of a file, as typed reads and writes.
One of three query modules, split to stay under this repo's 500-line-per-file
cap: this one owns how a file got downloaded, :mod:`store_threads` owns which
threads were visited, and :mod:`store_verdicts` owns what the user decided.
Between them they hold all the SQL outside :mod:`db`, so the rest of the
package deals in dataclasses. Every function takes an already-open connection
owned by the calling thread.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import TYPE_CHECKING
# scalar() is shared: record_files counts rows before and after. The rest are
# re-exported through __all__ so store.claim_next / store.ready_items / etc.
# keep working for downloader.py, cli.py and test_store.py unchanged.
from python_pkg.wsg_grabber._store_queries import (
attempts_for,
claim_next,
claimable_values,
counts,
known_md5s,
pending_downloads,
ready_item,
ready_items,
scalar,
state_of,
)
from python_pkg.wsg_grabber.states import FileState
__all__ = [
"attempts_for",
"claim_next",
"claimable_values",
"counts",
"known_md5s",
"local_name",
"mark_downloaded",
"pending_downloads",
"ready_item",
"ready_items",
"record_files",
"record_progress",
"reset_in_flight",
"set_state",
"state_of",
]
if TYPE_CHECKING:
from collections.abc import Iterable
import sqlite3
from python_pkg.wsg_grabber.models import (
RemoteFile,
)
def _now() -> str:
"""Return the current UTC time as an ISO-8601 string.
Returns:
str: Timestamp suitable for the index's text date columns.
"""
return datetime.now(tz=UTC).isoformat()
def local_name(md5_b64: str, tim: int, ext: str) -> str:
"""Return the on-disk filename for a file.
The poster-supplied name is untrusted text and is never used to build a
path; it is carried alongside purely for display.
Args:
md5_b64: The API's base64 MD5, used only to disambiguate.
tim: The CDN filename stem.
ext: Extension including the leading dot.
Returns:
str: A filesystem-safe name unique per file.
"""
suffix = md5_b64.rstrip("=")[-6:].replace("/", "_").replace("+", "-")
return f"{tim}-{suffix}{ext}"
def record_files(conn: sqlite3.Connection, files: Iterable[RemoteFile]) -> int:
"""Insert any files not already known.
Dedupe is structural: ``md5`` is the primary key, so a repost of identical
bytes in another thread collapses onto the existing row and never costs a
download.
Args:
conn: Open connection.
files: Candidate attachments parsed from a thread.
Returns:
int: How many rows were genuinely new.
"""
rows = [
(
item.md5,
item.tim,
item.ext,
item.orig_name,
item.fsize,
item.width,
item.height,
item.thread_no,
item.post_no,
FileState.NEW.value,
_now(),
)
for item in files
]
if not rows:
return 0
before = scalar(conn, "SELECT COUNT(*) FROM files")
conn.executemany(
"""
INSERT INTO files (md5, tim, ext, orig_name, fsize, width, height,
thread_no, post_no, state, first_seen)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(md5) DO NOTHING
""",
rows,
)
return scalar(conn, "SELECT COUNT(*) FROM files") - before
def set_state(
conn: sqlite3.Connection,
md5_b64: str,
state: FileState,
*,
error: str | None = None,
) -> None:
"""Move a file to *state*.
Args:
conn: Open connection.
md5_b64: File identity.
state: New state.
error: Optional diagnostic kept for the next run's benefit.
"""
conn.execute(
"UPDATE files SET state = ?, last_error = ? WHERE md5 = ?",
(state.value, error, md5_b64),
)
def mark_downloaded(conn: sqlite3.Connection, md5_b64: str, name: str) -> None:
"""Record that a file finished downloading and is ready to review.
Args:
conn: Open connection.
md5_b64: File identity.
name: Filename inside the incoming directory.
"""
conn.execute(
"""
UPDATE files
SET state = ?, local_name = ?, downloaded_at = ?, bytes_done = 0,
last_error = NULL
WHERE md5 = ?
""",
(FileState.READY.value, name, _now(), md5_b64),
)
def record_progress(conn: sqlite3.Connection, md5_b64: str, done: int) -> None:
"""Persist partial download progress so a restart can resume.
Args:
conn: Open connection.
md5_b64: File identity.
done: Bytes written to the ``.part`` file so far.
"""
conn.execute(
"UPDATE files SET bytes_done = ? WHERE md5 = ?",
(done, md5_b64),
)
def reset_in_flight(conn: sqlite3.Connection) -> int:
"""Return files interrupted mid-download to the pending pool.
A row left in ``downloading`` means the process died; the partial file on
disk is still valid and will be resumed.
Args:
conn: Open connection.
Returns:
int: How many rows were reset.
"""
cursor = conn.execute(
"UPDATE files SET state = ? WHERE state = ?",
(FileState.FAILED.value, FileState.DOWNLOADING.value),
)
return int(cursor.rowcount)

157
python_pkg/wsg_grabber/store_threads.py
View File

@ -1,157 +0,0 @@
"""Bookkeeping for the ``threads`` table.
Split out of :mod:`python_pkg.wsg_grabber.store` to stay under this repo's
500-line-per-file cap. The seam is the natural one: this module owns which
threads have been visited and what their last ``Last-Modified`` was, which is
what lets an unchanged thread cost a 304 rather than a re-parse.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import sqlite3
from python_pkg.wsg_grabber.models import ThreadRef
def _now() -> str:
"""Return the current UTC time as an ISO-8601 string.
Returns:
str: Timestamp suitable for the index's text date columns.
"""
return datetime.now(tz=UTC).isoformat()
def known_threads(conn: sqlite3.Connection) -> dict[int, int]:
"""Return each known thread's last-modified stamp.
Args:
conn: Open connection.
Returns:
dict[int, int]: Thread number mapped to its stored API timestamp.
"""
return {
int(row["thread_no"]): int(row["api_last_mod"])
for row in conn.execute("SELECT thread_no, api_last_mod FROM threads")
}
def http_last_modified(conn: sqlite3.Connection, thread_no: int) -> str | None:
"""Return the stored ``Last-Modified`` header for a thread.
Args:
conn: Open connection.
thread_no: Thread identifier.
Returns:
str | None: Header value to send back as ``If-Modified-Since``.
"""
row = conn.execute(
"SELECT http_last_mod FROM threads WHERE thread_no = ?",
(thread_no,),
).fetchone()
if row is None or row["http_last_mod"] is None:
return None
return str(row["http_last_mod"])
def record_thread(
conn: sqlite3.Connection,
ref: ThreadRef,
header: str | None,
status: str = "live",
) -> None:
"""Insert or update a thread's bookkeeping row.
Args:
conn: Open connection.
ref: Thread number and its API timestamp.
header: ``Last-Modified`` header from the thread fetch, if any.
status: ``live`` or ``gone``.
"""
conn.execute(
"""
INSERT INTO threads (thread_no, api_last_mod, http_last_mod,
last_checked, status)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(thread_no) DO UPDATE SET
api_last_mod = excluded.api_last_mod,
http_last_mod = COALESCE(excluded.http_last_mod, threads.http_last_mod),
last_checked = excluded.last_checked,
status = excluded.status
""",
(ref.thread_no, ref.api_last_modified, header, _now(), status),
)
def mark_thread_gone(conn: sqlite3.Connection, thread_no: int) -> None:
"""Flag a thread as no longer fetchable.
This inserts rather than merely updating: a thread can 404 the very first
time it is seen, and a plain UPDATE would touch no rows, leaving nothing to
record that it is dead. It would then be re-fetched on every future run.
Args:
conn: Open connection.
thread_no: Thread identifier.
"""
conn.execute(
"""
INSERT INTO threads (thread_no, api_last_mod, http_last_mod,
last_checked, status)
VALUES (?, 0, NULL, ?, 'gone')
ON CONFLICT(thread_no) DO UPDATE SET
status = 'gone',
last_checked = excluded.last_checked
""",
(thread_no, _now()),
)
def cached_last_modified(conn: sqlite3.Connection, url: str) -> str | None:
"""Return the stored ``Last-Modified`` for a board-level endpoint.
``threads.json`` and ``archive.json`` are not tied to any one thread, so
their stamps live here rather than in the threads table. Without this the
reviewer re-fetches both unconditionally on every idle cycle, forever.
Args:
conn: Open connection.
url: Absolute API URL.
Returns:
str | None: Header value to send back as ``If-Modified-Since``.
"""
row = conn.execute(
"SELECT last_modified FROM http_cache WHERE url = ?",
(url,),
).fetchone()
return None if row is None else str(row["last_modified"])
def record_last_modified(
conn: sqlite3.Connection,
url: str,
stamp: str | None,
) -> None:
"""Store the ``Last-Modified`` a board-level endpoint returned.
Args:
conn: Open connection.
url: Absolute API URL.
stamp: Header value; ignored when the server did not send one.
"""
if stamp is None:
return
conn.execute(
"""
INSERT INTO http_cache (url, last_modified) VALUES (?, ?)
ON CONFLICT(url) DO UPDATE SET last_modified = excluded.last_modified
""",
(url, stamp),
)

236
python_pkg/wsg_grabber/store_verdicts.py
View File

@ -1,236 +0,0 @@
"""The review trail: verdicts, and everything undo needs to reverse them.
Split out of :mod:`python_pkg.wsg_grabber.store` to stay under this repo's
500-line-per-file cap, along the seam that matters: this module owns what the
user decided, the other owns how a file got downloaded.
The trail lives in the index rather than in memory, which is what makes undo
unbounded and lets it survive quitting the reviewer.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import ReviewedItem, ReviewItem, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from pathlib import Path
import sqlite3
def _now() -> str:
"""Return the current UTC time as an ISO-8601 string.
Returns:
str: Timestamp suitable for the index's text date columns.
"""
return datetime.now(tz=UTC).isoformat()
def _scalar(conn: sqlite3.Connection, sql: str, params: list[str]) -> int:
"""Run a single-value query and return it as an int.
Args:
conn: Open connection.
sql: Query yielding exactly one column.
params: Bound parameters.
Returns:
int: The first column of the first row.
"""
return int(conn.execute(sql, params).fetchone()[0])
def record_verdict(
conn: sqlite3.Connection,
md5_b64: str,
state: FileState,
reviewed_name: str | None = None,
) -> None:
"""Store the user's decision about a file.
Args:
conn: Open connection.
md5_b64: File identity.
state: Either ``KEPT`` or ``PASSED``.
reviewed_name: Name the file now has in keep/ or trash/. Persisting it
is what lets the verdict be undone in a later session.
"""
conn.execute(
"""
UPDATE files
SET state = ?, reviewed_at = ?, reviewed_name = ?
WHERE md5 = ?
""",
(state.value, _now(), reviewed_name, md5_b64),
)
def reviewed_count(conn: sqlite3.Connection) -> int:
"""Return how many verdicts could still be taken back.
Args:
conn: Open connection.
Returns:
int: Count of reviewed files whose destination name is known.
"""
return _scalar(
conn,
"""
SELECT COUNT(*) FROM files
WHERE state IN (?, ?) AND reviewed_name IS NOT NULL
""",
[FileState.KEPT.value, FileState.PASSED.value],
)
def newest_verdict(
conn: sqlite3.Connection,
directory: Path,
) -> ReviewedItem | None:
"""Return the most recent verdict that can be undone.
Reading this from the index rather than from memory is what makes undo
survive quitting, and unbounded: every verdict ever recorded is a candidate,
newest first.
Args:
conn: Open connection.
directory: Where an undone file should be put back.
Returns:
ReviewedItem | None: The verdict to reverse, or None when there is none.
"""
row = conn.execute(
"""
SELECT md5, local_name, reviewed_name, orig_name, ext, fsize,
width, height, state
FROM files
WHERE state IN (?, ?) AND reviewed_name IS NOT NULL
ORDER BY reviewed_at DESC, md5 DESC
LIMIT 1
""",
(FileState.KEPT.value, FileState.PASSED.value),
).fetchone()
if row is None:
return None
return ReviewedItem(
item=ReviewItem(
md5=row["md5"],
path=directory / row["local_name"],
orig_name=f"{row['orig_name']}{row['ext']}",
fsize=row["fsize"],
width=row["width"],
height=row["height"],
),
choice=Verdict.KEEP if row["state"] == FileState.KEPT.value else Verdict.SKIP,
reviewed_name=row["reviewed_name"],
)
def forget_verdict(conn: sqlite3.Connection, md5_b64: str) -> None:
"""Make a verdict un-undoable without changing it.
Used when the file is no longer where the verdict left it, so the same
entry does not keep surfacing as the next thing to undo.
Args:
conn: Open connection.
md5_b64: File identity.
"""
conn.execute(
"UPDATE files SET reviewed_name = NULL WHERE md5 = ?",
(md5_b64,),
)
def restore_for_review(conn: sqlite3.Connection, md5_b64: str, name: str) -> None:
"""Undo a verdict, returning the file to the review queue.
Args:
conn: Open connection.
md5_b64: File identity.
name: Filename it has again inside the incoming directory.
"""
conn.execute(
"""
UPDATE files
SET state = ?, local_name = ?, reviewed_at = NULL,
reviewed_name = NULL
WHERE md5 = ?
""",
(FileState.READY.value, name, md5_b64),
)
def recent_passed_names(conn: sqlite3.Connection, limit: int) -> set[str]:
"""Return the trash names of the *limit* most recent passes.
These are the files the trash prune keeps; everything else in ``trash/``
goes.
Args:
conn: Open connection.
limit: How many passes to keep.
Returns:
set[str]: Filenames inside ``trash/``.
"""
rows = conn.execute(
"""
SELECT reviewed_name FROM files
WHERE state = ? AND reviewed_name IS NOT NULL
ORDER BY reviewed_at DESC, md5 DESC
LIMIT ?
""",
(FileState.PASSED.value, limit),
).fetchall()
return {str(row["reviewed_name"]) for row in rows}
def purge_older_passes(conn: sqlite3.Connection, limit: int) -> int:
"""Mark every pass beyond the *limit* newest as purged.
A purged row keeps its md5, so the catalog never re-downloads the file,
but drops out of the undo trail because its bytes are gone.
Args:
conn: Open connection.
limit: How many passes stay undoable.
Returns:
int: Rows changed.
"""
cursor = conn.execute(
"""
UPDATE files
SET state = ?, reviewed_name = NULL
WHERE state = ?
AND md5 NOT IN (
SELECT md5 FROM files
WHERE state = ? AND reviewed_name IS NOT NULL
ORDER BY reviewed_at DESC, md5 DESC
LIMIT ?
)
""",
(FileState.PURGED.value, FileState.PASSED.value, FileState.PASSED.value, limit),
)
return int(cursor.rowcount)
def rename_kept(conn: sqlite3.Connection, old: str, new: str) -> None:
"""Record that a kept file changed name while being migrated.
Args:
conn: Open connection.
old: Name the verdict recorded.
new: Name the file has now.
"""
conn.execute(
"UPDATE files SET reviewed_name = ? WHERE state = ? AND reviewed_name = ?",
(new, FileState.KEPT.value, old),
)

0
python_pkg/wsg_grabber/tests/__init__.py
View File

85
python_pkg/wsg_grabber/tests/conftest.py
View File

@ -1,85 +0,0 @@
"""Test isolation for the /wsg/ grabber.
This package writes an index and tens of gigabytes of video into
``~/.local/share/wsg_grabber``. A test that escapes into the real directory
would corrupt live user data, so redirection is autouse and belt-and-braces:
1. ``Path.home`` is patched on the *class*, so it holds no matter whether a
module wrote ``from pathlib import Path`` or ``import pathlib``.
2. ``HOME`` and ``XDG_DATA_HOME`` are redirected, catching anything that goes
through ``os.environ`` or ``expanduser`` instead.
3. Every module reaches storage only through :mod:`python_pkg.wsg_grabber.paths`
functions, which resolve on each call rather than at import.
``tests/test_paths.py`` adds a static check that no module captures a home
directory at import time, so layer 3 stays true as the package grows.
Also holds the fixture builders the reviewer test modules share. They live
here rather than in a ``_helpers.py`` because the ``name-tests-test`` hook
requires every other file under ``tests/`` to be named ``test_*.py``.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from python_pkg.wsg_grabber.models import ReviewedItem, ReviewItem, Verdict
@pytest.fixture(autouse=True)
def isolate_paths(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Point every storage location at a throwaway directory.
No teardown is needed: ``monkeypatch`` unwinds both the environment and the
``Path.home`` patch when the test ends.
Args:
tmp_path: Per-test temporary directory supplied by pytest.
monkeypatch: Patching helper, unwound automatically after the test.
Returns:
Path: The fake home directory backing this test.
"""
home = tmp_path / "home"
data_home = home / ".local" / "share"
data_home.mkdir(parents=True)
monkeypatch.setenv("HOME", str(home))
monkeypatch.setenv("XDG_DATA_HOME", str(data_home))
monkeypatch.setattr(Path, "home", classmethod(lambda _cls: home))
return home
def item(name: str = "a") -> ReviewItem:
"""Build a ReviewItem.
Args:
name: Distinguishes items within a test.
Returns:
ReviewItem: Fixture value.
"""
return ReviewItem(
md5=f"{name}-md5",
path=Path(f"/incoming/{name}.webm"),
orig_name=f"{name}.webm",
fsize=2 * 1024 * 1024,
width=480,
height=360,
)
def reviewed(item: ReviewItem, choice: Verdict = Verdict.SKIP) -> ReviewedItem:
"""Build the record a verdict would have persisted.
Args:
item: The reviewed video.
choice: What was decided.
Returns:
ReviewedItem: Reversible verdict.
"""
return ReviewedItem(item=item, choice=choice, reviewed_name=item.path.name)

211
python_pkg/wsg_grabber/tests/test_app.py
View File

@ -1,211 +0,0 @@
"""Tests for the composition root and verdict application."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import (
app,
db,
downloader,
paths,
review,
store,
)
from python_pkg.wsg_grabber.models import RemoteFile, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Callable, Iterator
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
@pytest.fixture
def session() -> Iterator[app.Session]:
"""Build a session over a sandboxed index, with no worker running.
Yields:
app.Session: Session whose connection is closed afterwards.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
try:
yield built
finally:
conn.close()
def _downloaded(session: app.Session, name: str = "clip") -> None:
"""Put one downloaded, unreviewed file into the index and on disk.
Args:
session: The session to populate.
name: Local filename stem.
"""
store.record_files(
session.conn,
[
RemoteFile(
md5=_MD5,
tim=1,
ext=".webm",
orig_name=name,
fsize=5,
width=4,
height=3,
thread_no=7,
post_no=1,
),
],
)
local = f"{name}.webm"
(paths.incoming_dir() / local).write_bytes(b"video")
store.mark_downloaded(session.conn, _MD5, local)
def test_keep_moves_the_file_and_records_it(session: app.Session) -> None:
_downloaded(session)
state = app.initial_state(session)
result = session.commit(state, Verdict.KEEP)
assert (paths.keep_dir() / "clip.webm").exists()
assert not (paths.incoming_dir() / "clip.webm").exists()
assert store.state_of(session.conn, _MD5) is FileState.KEPT
assert result.kept == 1
def test_pass_moves_to_trash_and_never_deletes(session: app.Session) -> None:
"""The whole point of trash/: a pass is reversible until the user says so."""
_downloaded(session)
state = app.initial_state(session)
result = session.commit(state, Verdict.SKIP)
assert (paths.trash_dir() / "clip.webm").exists()
assert (paths.trash_dir() / "clip.webm").read_bytes() == b"video"
assert store.state_of(session.conn, _MD5) is FileState.PASSED
assert result.passed == 1
def test_a_name_collision_in_the_destination_is_resolved(
session: app.Session,
) -> None:
_downloaded(session)
(paths.keep_dir() / "clip.webm").write_bytes(b"older")
session.commit(app.initial_state(session), Verdict.KEEP)
assert (paths.keep_dir() / "clip.webm").read_bytes() == b"older"
assert (paths.keep_dir() / "clip-2.webm").read_bytes() == b"video"
def test_a_verdict_on_a_vanished_file_writes_it_off(session: app.Session) -> None:
_downloaded(session)
state = app.initial_state(session)
(paths.incoming_dir() / "clip.webm").unlink()
result = session.commit(state, Verdict.KEEP)
assert store.state_of(session.conn, _MD5) is FileState.GONE
assert result.kept == 0
assert result.current is None
def test_a_verdict_with_nothing_showing_is_ignored(session: app.Session) -> None:
state = review.initial_state(0, [])
assert session.commit(state, Verdict.KEEP) is state
assert session.missing(state) is state
def test_initial_state_resumes_a_previous_run(session: app.Session) -> None:
_downloaded(session)
state = app.initial_state(session)
assert state.current is not None
assert state.current.md5 == _MD5
assert state.indexed == 1
def test_shutdown_stops_the_worker_and_closes_the_index() -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
worker = MagicMock()
worker.is_alive.return_value = False
built = app.Session(
conn=conn,
events=queue.SimpleQueue(),
stop=threading.Event(),
worker=worker,
)
built.shutdown()
assert built.stop.is_set()
worker.join.assert_called_once()
assert built.worker is None
def test_shutdown_without_a_worker() -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
built.shutdown()
assert built.stop.is_set()
def test_open_session_starts_a_worker_owning_its_own_connection() -> None:
"""sqlite forbids sharing a connection across threads, so the factory must
build one on the worker's side of the boundary."""
captured: list[downloader.WorkerDeps] = []
def fake_start(
build: Callable[[], downloader.WorkerDeps],
_on_failure: Callable[[str], None] = lambda _m: None,
) -> MagicMock:
deps = build()
captured.append(deps)
deps.conn.close()
deps.session.close()
return MagicMock()
with patch.object(downloader, "start", side_effect=fake_start):
built = app.open_session(include_archive=False)
try:
assert captured
assert captured[0].incoming == paths.incoming_dir()
assert not captured[0].include_archive
assert paths.incoming_dir().is_dir()
assert paths.trash_dir().is_dir()
finally:
built.shutdown()
def test_open_session_recovers_interrupted_downloads() -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
store.record_files(
conn,
[
RemoteFile(
md5=_MD5,
tim=1,
ext=".webm",
orig_name="x",
fsize=1,
width=1,
height=1,
thread_no=1,
post_no=1,
),
],
)
store.claim_next(conn)
conn.close()
with patch.object(downloader, "start", return_value=MagicMock()):
built = app.open_session()
try:
assert store.pending_downloads(built.conn) == 1
finally:
built.shutdown()

111
python_pkg/wsg_grabber/tests/test_app_undo.py
View File

@ -1,111 +0,0 @@
"""Tests for undoing verdicts in a review session."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import (
app,
db,
paths,
store,
)
from python_pkg.wsg_grabber.models import RemoteFile, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
@pytest.fixture
def session() -> Iterator[app.Session]:
"""Build a session over a sandboxed index, with no worker running.
Yields:
app.Session: Session whose connection is closed afterwards.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
try:
yield built
finally:
conn.close()
def _downloaded(session: app.Session, name: str = "clip") -> None:
"""Put one downloaded, unreviewed file into the index and on disk.
Args:
session: The session to populate.
name: Local filename stem.
"""
store.record_files(
session.conn,
[
RemoteFile(
md5=_MD5,
tim=1,
ext=".webm",
orig_name=name,
fsize=5,
width=4,
height=3,
thread_no=7,
post_no=1,
),
],
)
local = f"{name}.webm"
(paths.incoming_dir() / local).write_bytes(b"video")
store.mark_downloaded(session.conn, _MD5, local)
def test_undo_puts_a_kept_file_back_in_the_queue(session: app.Session) -> None:
"""The misclick this exists for: kept something you meant to pass."""
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.KEEP)
assert (paths.keep_dir() / "clip.webm").exists()
state = session.undo(state)
assert not (paths.keep_dir() / "clip.webm").exists()
assert (paths.incoming_dir() / "clip.webm").exists()
assert store.state_of(session.conn, _MD5) is FileState.READY
assert state.kept == 0
assert state.current is not None
assert state.current.md5 == _MD5
def test_undo_puts_a_passed_file_back(session: app.Session) -> None:
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.SKIP)
assert (paths.trash_dir() / "clip.webm").exists()
state = session.undo(state)
assert (paths.incoming_dir() / "clip.webm").exists()
assert not (paths.trash_dir() / "clip.webm").exists()
assert state.passed == 0
def test_undo_survives_a_collision_rename(session: app.Session) -> None:
"""A verdict may have renamed the file; undo must follow the real name.
The stray ``clip.webm`` has no index row, so the prune that follows the
pass deletes it -- which is exactly why the rename must never be undone
by guessing the original name.
"""
_downloaded(session)
(paths.trash_dir() / "clip.webm").write_bytes(b"older")
state = session.commit(app.initial_state(session), Verdict.SKIP)
assert (paths.trash_dir() / "clip-2.webm").exists()
assert not (paths.trash_dir() / "clip.webm").exists()
state = session.undo(state)
assert (paths.incoming_dir() / "clip.webm").read_bytes() == b"video"
assert not (paths.trash_dir() / "clip-2.webm").exists()

185
python_pkg/wsg_grabber/tests/test_app_undo_history.py
View File

@ -1,185 +0,0 @@
"""Tests for stepping back through several verdicts."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import (
app,
db,
paths,
review,
store,
store_verdicts,
)
from python_pkg.wsg_grabber.models import RemoteFile, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
@pytest.fixture
def session() -> Iterator[app.Session]:
"""Build a session over a sandboxed index, with no worker running.
Yields:
app.Session: Session whose connection is closed afterwards.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
try:
yield built
finally:
conn.close()
def _downloaded(session: app.Session, name: str = "clip") -> None:
"""Put one downloaded, unreviewed file into the index and on disk.
Args:
session: The session to populate.
name: Local filename stem.
"""
store.record_files(
session.conn,
[
RemoteFile(
md5=_MD5,
tim=1,
ext=".webm",
orig_name=name,
fsize=5,
width=4,
height=3,
thread_no=7,
post_no=1,
),
],
)
local = f"{name}.webm"
(paths.incoming_dir() / local).write_bytes(b"video")
store.mark_downloaded(session.conn, _MD5, local)
def test_undo_steps_back_through_several_verdicts(session: app.Session) -> None:
for index in range(3):
store.record_files(
session.conn,
[
RemoteFile(
md5=f"M{index}",
tim=index,
ext=".webm",
orig_name=f"v{index}",
fsize=5,
width=1,
height=1,
thread_no=1,
post_no=index,
),
],
)
(paths.incoming_dir() / f"v{index}.webm").write_bytes(b"video")
store.mark_downloaded(session.conn, f"M{index}", f"v{index}.webm")
state = app.initial_state(session)
for _ in range(3):
state = session.commit(state, Verdict.SKIP)
assert state.passed == 3
for expected in ("M2", "M1", "M0"):
state = session.undo(state)
assert state.current is not None
assert state.current.md5 == expected
assert state.passed == 0
assert not review.can_undo(state)
def test_undo_with_nothing_to_undo_is_a_no_op(session: app.Session) -> None:
state = app.initial_state(session)
assert session.undo(state) is state
def test_undo_gives_up_gracefully_when_the_file_was_removed(
session: app.Session,
) -> None:
"""You may have emptied trash/ by hand; undo must not wedge on it."""
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.SKIP)
(paths.trash_dir() / "clip.webm").unlink()
state = session.undo(state)
assert not review.can_undo(state)
assert store.state_of(session.conn, _MD5) is FileState.PASSED
def test_undo_survives_quitting_and_reopening(session: app.Session) -> None:
"""The whole point of persisting the trail: undo is not lost on exit."""
_downloaded(session)
session.commit(app.initial_state(session), Verdict.SKIP)
assert (paths.trash_dir() / "clip.webm").exists()
session.conn.close()
# a brand new session, as if the reviewer had been restarted
reopened = app.Session(
conn=db.open_index(paths.db_path()),
events=queue.SimpleQueue(),
stop=threading.Event(),
)
try:
fresh = app.initial_state(reopened)
assert fresh.undoable == 1
restored = reopened.undo(fresh)
assert (paths.incoming_dir() / "clip.webm").exists()
assert store.state_of(reopened.conn, _MD5) is FileState.READY
assert restored.current is not None
assert restored.current.md5 == _MD5
assert restored.undoable == 0
finally:
reopened.conn.close()
def test_the_undo_trail_is_unbounded(session: app.Session) -> None:
"""Any number of last verdicts, not a fixed window."""
total = 25
for index in range(total):
store.record_files(
session.conn,
[
RemoteFile(
md5=f"U{index:02d}",
tim=index,
ext=".webm",
orig_name=f"u{index:02d}",
fsize=5,
width=1,
height=1,
thread_no=1,
post_no=index,
),
],
)
(paths.incoming_dir() / f"u{index:02d}.webm").write_bytes(b"video")
store.mark_downloaded(session.conn, f"U{index:02d}", f"u{index:02d}.webm")
state = app.initial_state(session)
for _ in range(total):
state = session.commit(state, Verdict.SKIP)
assert state.passed == total
assert store_verdicts.reviewed_count(session.conn) == total
for index in reversed(range(total)):
state = session.undo(state)
assert state.current is not None
assert state.current.md5 == f"U{index:02d}"
assert state.passed == 0
assert store_verdicts.reviewed_count(session.conn) == 0
assert len(list(paths.trash_dir().iterdir())) == 0

132
python_pkg/wsg_grabber/tests/test_app_undo_safety.py
View File

@ -1,132 +0,0 @@
"""Tests that undo never clobbers an existing file."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import (
app,
db,
downloader,
paths,
store,
store_verdicts,
)
from python_pkg.wsg_grabber.models import RemoteFile, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
@pytest.fixture
def session() -> Iterator[app.Session]:
"""Build a session over a sandboxed index, with no worker running.
Yields:
app.Session: Session whose connection is closed afterwards.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
try:
yield built
finally:
conn.close()
def _downloaded(session: app.Session, name: str = "clip") -> None:
"""Put one downloaded, unreviewed file into the index and on disk.
Args:
session: The session to populate.
name: Local filename stem.
"""
store.record_files(
session.conn,
[
RemoteFile(
md5=_MD5,
tim=1,
ext=".webm",
orig_name=name,
fsize=5,
width=4,
height=3,
thread_no=7,
post_no=1,
),
],
)
local = f"{name}.webm"
(paths.incoming_dir() / local).write_bytes(b"video")
store.mark_downloaded(session.conn, _MD5, local)
def test_undo_never_overwrites_a_file_sitting_in_incoming(
session: app.Session,
) -> None:
"""A verdict must never be reversed on top of another video.
Path.replace clobbers silently, so undo picks a collision-free name the
same way commit does.
"""
_downloaded(session)
state = session.commit(app.initial_state(session), Verdict.KEEP)
# something else now occupies the name the file used to have
(paths.incoming_dir() / "clip.webm").write_bytes(b"DO NOT DESTROY ME")
session.undo(state)
assert (paths.incoming_dir() / "clip.webm").read_bytes() == b"DO NOT DESTROY ME"
assert (paths.incoming_dir() / "clip-2.webm").read_bytes() == b"video"
row = session.conn.execute(
"SELECT local_name FROM files WHERE md5 = ?",
(_MD5,),
).fetchone()
assert row["local_name"] == "clip-2.webm"
def test_a_missing_file_never_clobbers_an_existing_verdict(
session: app.Session,
) -> None:
"""Two reviewers on one index must not corrupt each other's undo trail."""
_downloaded(session)
state = app.initial_state(session)
session.commit(state, Verdict.KEEP)
assert store_verdicts.reviewed_count(session.conn) == 1
# the same item, still queued in a second reviewer, whose file is now gone
session.missing(state)
assert store.state_of(session.conn, _MD5) is FileState.KEPT
assert store_verdicts.reviewed_count(session.conn) == 1
def test_open_session_closes_the_index_if_the_worker_cannot_start() -> None:
with (
patch.object(downloader, "start", side_effect=RuntimeError("no thread")),
pytest.raises(RuntimeError),
):
app.open_session()
# a second open must succeed, which it cannot if the first leaked a handle
with patch.object(downloader, "start", return_value=MagicMock()):
again = app.open_session()
again.shutdown()
def test_a_worker_that_outlives_the_join_is_not_silently_forgotten(
session: app.Session,
) -> None:
worker = MagicMock()
worker.is_alive.return_value = True
session.worker = worker
session.shutdown()
assert session.worker is worker # handle kept so the failure is visible

171
python_pkg/wsg_grabber/tests/test_catalog.py
View File

@ -1,171 +0,0 @@
"""Tests for parsing the /wsg/ JSON payloads.
Fixtures mirror the shape of real responses captured from the live API,
including the fields that are simply absent on text-only posts.
"""
from __future__ import annotations
from python_pkg.wsg_grabber import catalog
_WEBM = "Nz9OEKdMuMZEdYE6eLTKmA=="
_MP4 = "b9EO3Kaq7PHBgc6kWmHgew=="
def _post(**overrides: object) -> dict[str, object]:
"""Build a post carrying a webm attachment.
Args:
**overrides: Fields to replace or, with a None value, remove.
Returns:
dict[str, object]: Post payload.
"""
post: dict[str, object] = {
"no": 6195878,
"resto": 0,
"tim": 1784059525831380,
"ext": ".webm",
"filename": "kino moment",
"fsize": 1008689,
"md5": _WEBM,
"w": 480,
"h": 360,
}
post.update(overrides)
return {key: value for key, value in post.items() if value is not None}
def test_media_url_uses_tim_not_the_poster_filename() -> None:
assert (
catalog.media_url(1784059525831380, ".webm")
== "https://i.4cdn.org/wsg/1784059525831380.webm"
)
def test_parse_thread_list_flattens_every_page() -> None:
payload = [
{"page": 1, "threads": [{"no": 1, "last_modified": 100, "replies": 2}]},
{"page": 2, "threads": [{"no": 2, "last_modified": 200, "replies": 5}]},
]
refs = catalog.parse_thread_list(payload)
assert [(ref.thread_no, ref.api_last_modified) for ref in refs] == [
(1, 100),
(2, 200),
]
def test_parse_thread_list_survives_malformed_payloads() -> None:
assert catalog.parse_thread_list("not a list") == []
assert catalog.parse_thread_list([{"threads": "nope"}]) == []
assert catalog.parse_thread_list([{"threads": [{"no": None}]}]) == []
assert catalog.parse_thread_list([{"threads": [{"no": True}]}]) == []
refs = catalog.parse_thread_list([{"threads": [{"no": 5}]}])
assert refs[0].api_last_modified == 0
def test_parse_archive_reads_bare_thread_numbers() -> None:
refs = catalog.parse_archive([11, 22, "junk", None])
assert [ref.thread_no for ref in refs] == [11, 22]
assert refs[0].api_last_modified == 0
def test_parse_thread_keeps_all_three_video_extensions() -> None:
payload = {
"posts": [
_post(ext=".webm", md5=_WEBM, no=1),
_post(ext=".mp4", md5=_MP4, no=2),
_post(ext=".gif", md5="c" * 22 + "==", no=3),
],
}
files = catalog.parse_thread(99, payload)
assert [item.ext for item in files] == [".webm", ".mp4", ".gif"]
assert all(item.thread_no == 99 for item in files)
def test_parse_thread_skips_images_and_text_posts() -> None:
payload = {
"posts": [
{"no": 1, "com": "text only"},
_post(ext=".jpg", no=2),
_post(ext=".png", no=3),
_post(no=4),
],
}
files = catalog.parse_thread(99, payload)
assert len(files) == 1
assert files[0].post_no == 4
def test_parse_thread_skips_deleted_attachments() -> None:
payload = {"posts": [_post(filedeleted=1)]}
assert catalog.parse_thread(99, payload) == []
def test_parse_thread_rejects_malformed_file_fields() -> None:
cases = [
_post(tim=None),
_post(no=None),
_post(md5=None),
_post(md5="too-short"),
_post(ext=12345),
]
for post in cases:
assert catalog.parse_thread(99, {"posts": [post]}) == []
def test_parse_thread_defaults_missing_optional_fields() -> None:
payload = {"posts": [_post(filename=None, fsize=None, w=None, h=None)]}
item = catalog.parse_thread(99, payload)[0]
assert item.orig_name == str(item.tim)
assert (item.fsize, item.width, item.height) == (0, 0, 0)
def test_parse_thread_survives_a_payload_that_is_not_a_dict() -> None:
assert catalog.parse_thread(1, "nonsense") == []
assert catalog.parse_thread(1, {"posts": "nonsense"}) == []
assert catalog.parse_thread(1, {"posts": ["nonsense"]}) == []
def test_deleted_md5s_collects_removed_attachments() -> None:
payload = {
"posts": [
_post(no=1, filedeleted=1),
_post(no=2, md5=_MP4),
_post(no=3, filedeleted=1, md5=None),
],
}
assert catalog.deleted_md5s(payload) == {_WEBM}
def test_new_files_drops_anything_already_indexed() -> None:
parsed = catalog.parse_thread(
1,
{"posts": [_post(no=1), _post(no=2, md5=_MP4)]},
)
fresh = catalog.new_files(parsed, {_WEBM})
assert [item.md5 for item in fresh] == [_MP4]
def test_new_files_dedupes_within_one_batch() -> None:
"""A file quoted twice in the same thread must only be queued once."""
parsed = catalog.parse_thread(
1,
{"posts": [_post(no=1), _post(no=2, tim=999)]},
)
assert len(parsed) == 2
assert len(catalog.new_files(parsed, set())) == 1
def test_a_hostile_md5_is_rejected_before_it_can_reach_a_filename() -> None:
"""The digest tail ends up in a filename; length alone is not enough."""
for hostile in ("../.." + "x" * 18, "a" * 23 + "\x00", "/" * 24, "a" * 23 + "\n"):
assert catalog.parse_thread(1, {"posts": [_post(md5=hostile)]}) == []
assert catalog.parse_thread(1, {"posts": [_post(md5=_WEBM)]}) != []
def test_an_out_of_range_integer_is_rejected() -> None:
"""Wider than sqlite's signed 64-bit range kills the worker on insert."""
assert catalog.parse_thread(1, {"posts": [_post(tim=2**63)]}) == []
assert catalog.parse_thread(1, {"posts": [_post(no=-(2**63) - 1)]}) == []
assert catalog.parse_thread(1, {"posts": [_post(tim=2**63 - 1)]}) != []

242
python_pkg/wsg_grabber/tests/test_cli.py
View File

@ -1,242 +0,0 @@
"""Tests for the command-line interface."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import app, cli, db, logs, paths, store
from python_pkg.wsg_grabber.models import RemoteFile
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
@pytest.fixture
def fake_session() -> Iterator[app.Session]:
"""Build a session over a sandboxed index with no worker.
Yields:
app.Session: Session closed afterwards.
"""
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
built = app.Session(conn=conn, events=queue.SimpleQueue(), stop=threading.Event())
try:
yield built
finally:
conn.close()
def test_parser_defaults_to_the_reviewer() -> None:
assert cli.build_parser().parse_args([]).command is None
assert cli.build_parser().parse_args(["review"]).command == "review"
def test_parser_exposes_the_subcommands() -> None:
assert cli.build_parser().parse_args(["stats"]).command == "stats"
parsed = cli.build_parser().parse_args(["scrape", "--seconds", "5"])
assert parsed.command == "scrape"
assert parsed.seconds == pytest.approx(5.0)
def test_help_says_where_keeps_and_passes_go() -> None:
text = cli.build_parser().format_help()
assert "cloud folder" in text
assert "newest passes are kept" in text
assert "trash" in text
def test_stats_reports_every_state(capsys: pytest.CaptureFixture[str]) -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
store.record_files(
conn,
[
RemoteFile(
md5="A" * 24,
tim=1,
ext=".webm",
orig_name="x",
fsize=1,
width=1,
height=1,
thread_no=1,
post_no=1,
),
],
)
conn.close()
assert cli.main(["stats"]) == 0
out = capsys.readouterr().out
assert "known files 1" in out
assert "new 1" in out
assert "newest 200 passes kept" in out
def test_scrape_runs_and_shuts_down(
fake_session: app.Session,
capsys: pytest.CaptureFixture[str],
) -> None:
ticks = {"n": 0}
def fake_sleep(_seconds: float) -> None:
ticks["n"] += 1
with (
patch.object(app, "open_session", return_value=fake_session),
patch.object(fake_session, "shutdown") as stopped,
patch(
"python_pkg.wsg_grabber.cli.time.monotonic",
side_effect=[0.0, 1.0, 99.0, 99.0],
),
):
assert cli.scrape(5.0, sleeper=fake_sleep) == 0
assert ticks["n"] >= 1
stopped.assert_called_once()
assert "scraping /wsg/" in capsys.readouterr().out
def test_scrape_reports_progress_periodically(
fake_session: app.Session,
capsys: pytest.CaptureFixture[str],
) -> None:
with (
patch.object(app, "open_session", return_value=fake_session),
patch.object(fake_session, "shutdown"),
patch(
"python_pkg.wsg_grabber.cli.time.monotonic",
side_effect=[0.0, 999.0, 999.0, 999.0],
),
):
cli.scrape(1.0, sleeper=lambda _s: None)
assert "known 0" in capsys.readouterr().out
def test_scrape_handles_an_interrupt(
fake_session: app.Session,
capsys: pytest.CaptureFixture[str],
) -> None:
def interrupt(_seconds: float) -> None:
raise KeyboardInterrupt
with (
patch.object(app, "open_session", return_value=fake_session),
patch.object(fake_session, "shutdown") as stopped,
):
assert cli.scrape(0.0, sleeper=interrupt) == 0
stopped.assert_called_once()
assert "stopping" in capsys.readouterr().out
def test_main_dispatches_to_the_reviewer() -> None:
with patch.object(cli, "open_reviewer", return_value=0) as opened:
assert cli.main([]) == 0
opened.assert_called_once()
def test_main_dispatches_to_scrape() -> None:
with patch.object(cli, "scrape", return_value=0) as scraped:
assert cli.main(["scrape", "--seconds", "2"]) == 0
assert scraped.call_args.args[0] == pytest.approx(2.0)
def test_open_reviewer_wires_the_window_to_the_player(
fake_session: app.Session,
) -> None:
window = MagicMock()
window.video_wid.return_value = 4321
with (
patch.object(app, "open_session", return_value=fake_session),
patch(
"python_pkg.wsg_grabber.cli.ui.ReviewWindow",
return_value=window,
) as built,
patch("python_pkg.wsg_grabber.cli.player.MpvPlayer") as made_player,
):
assert cli.open_reviewer() == 0
made_player.assert_called_once_with(4321, paths.ipc_socket_path())
window.attach_player.assert_called_once()
window.run.assert_called_once()
callbacks = built.call_args.args[2]
assert callbacks.commit == fake_session.commit
assert callbacks.shutdown == fake_session.shutdown
def test_log_level_flags_are_exposed() -> None:
parsed = cli.build_parser().parse_args(["--log-level", "debug", "--echo-log"])
assert parsed.log_level == "debug"
assert parsed.echo_log
assert cli.build_parser().parse_args([]).log_level == "info"
def test_running_a_command_writes_a_session_log() -> None:
assert cli.main(["stats"]) == 0
written = sorted(logs.logs_dir().glob("session-*.jsonl"))
assert written
assert "session.start" in written[-1].read_text(encoding="utf-8")
def test_logs_command_reports_where_logs_live(
capsys: pytest.CaptureFixture[str],
) -> None:
cli.main(["stats"])
capsys.readouterr()
assert cli.main(["logs"]) == 0
out = capsys.readouterr().out
assert str(logs.logs_dir()) in out
assert "newest session" in out
assert "jq" in out
def test_logs_command_before_anything_has_run(
capsys: pytest.CaptureFixture[str],
) -> None:
logs.logs_dir().mkdir(parents=True, exist_ok=True)
for stale in logs.logs_dir().glob("*.jsonl"):
stale.unlink()
assert cli.show_logs() == 0
assert "no session logs yet" in capsys.readouterr().out
def test_summarise_counts_events_and_skips_junk(tmp_path: Path) -> None:
sample = tmp_path / "s.jsonl"
sample.write_text(
'{"event":"a","level":"info"}\n'
"not json at all\n"
'{"event":"a","level":"error"}\n'
'{"event":"b","level":"info"}\n',
encoding="utf-8",
)
text = cli.summarise(sample)
assert "error=1" in text
assert "info=2" in text
assert "a" in text
def test_a_failed_player_start_does_not_strand_the_worker(
fake_session: app.Session,
) -> None:
"""The worker is non-daemon; without cleanup the process never exits."""
window = MagicMock()
window.video_wid.return_value = 1
with (
patch.object(app, "open_session", return_value=fake_session),
patch("python_pkg.wsg_grabber.cli.ui.ReviewWindow", return_value=window),
patch(
"python_pkg.wsg_grabber.cli.player.MpvPlayer",
side_effect=TimeoutError("mpv never came up"),
),
patch.object(fake_session, "shutdown") as stopped,
pytest.raises(TimeoutError),
):
cli.open_reviewer()
stopped.assert_called_once()

130
python_pkg/wsg_grabber/tests/test_db.py
View File

@ -1,130 +0,0 @@
"""Tests for connection setup and schema migration."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import db, paths
from python_pkg.wsg_grabber.constants import SCHEMA_VERSION
if TYPE_CHECKING:
from pathlib import Path
import sqlite3
def _fresh(tmp_path: Path) -> sqlite3.Connection:
"""Open a brand new index inside the sandbox.
Args:
tmp_path: Per-test temporary directory.
Returns:
sqlite3.Connection: Migrated connection.
"""
return db.open_index(tmp_path / "index.db")
def test_connect_enables_wal_and_row_access(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
mode = conn.execute("PRAGMA journal_mode").fetchone()[0]
assert str(mode).lower() == "wal"
conn.execute("CREATE TABLE t (a INTEGER)")
conn.execute("INSERT INTO t VALUES (1)")
assert conn.execute("SELECT a FROM t").fetchone()["a"] == 1
finally:
conn.close()
def test_ensure_schema_creates_tables_and_stamps_the_version(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
assert db.ensure_schema(conn) == SCHEMA_VERSION
names = {
row[0]
for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
assert {"files", "threads"} <= names
assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == SCHEMA_VERSION
finally:
conn.close()
def test_ensure_schema_is_idempotent(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
db.ensure_schema(conn)
assert db.ensure_schema(conn) == SCHEMA_VERSION
finally:
conn.close()
def test_a_newer_schema_is_refused_rather_than_downgraded(tmp_path: Path) -> None:
conn = db.connect(tmp_path / "index.db")
try:
conn.execute(f"PRAGMA user_version={SCHEMA_VERSION + 1}")
with pytest.raises(db.SchemaTooNewError, match="upgrade the tool"):
db.ensure_schema(conn)
finally:
conn.close()
def test_open_index_returns_a_migrated_connection(tmp_path: Path) -> None:
conn = _fresh(tmp_path)
try:
assert conn.execute("SELECT COUNT(*) FROM files").fetchone()[0] == 0
finally:
conn.close()
def test_index_lands_inside_the_sandbox(tmp_path: Path) -> None:
paths.ensure_dirs()
conn = db.open_index(paths.db_path())
try:
assert str(paths.db_path()).startswith(str(tmp_path))
assert paths.db_path().exists()
finally:
conn.close()
def test_a_version_1_index_is_migrated_in_place(tmp_path: Path) -> None:
"""An index written before undo existed must keep its rows and gain the column."""
path = tmp_path / "old.db"
conn = db.connect(path)
try:
conn.execute(
"""
CREATE TABLE files (
md5 TEXT PRIMARY KEY, tim INTEGER NOT NULL, ext TEXT NOT NULL,
orig_name TEXT NOT NULL, fsize INTEGER NOT NULL,
width INTEGER NOT NULL, height INTEGER NOT NULL,
thread_no INTEGER NOT NULL, post_no INTEGER NOT NULL,
state TEXT NOT NULL, attempts INTEGER NOT NULL DEFAULT 0,
bytes_done INTEGER NOT NULL DEFAULT 0, last_error TEXT,
local_name TEXT, first_seen TEXT NOT NULL,
downloaded_at TEXT, reviewed_at TEXT
)
""",
)
conn.execute(
"INSERT INTO files (md5,tim,ext,orig_name,fsize,width,height,"
"thread_no,post_no,state,first_seen) "
"VALUES ('OLD',1,'.webm','x',1,1,1,1,1,'kept','then')",
)
conn.execute("PRAGMA user_version=1")
finally:
conn.close()
conn = db.open_index(path)
try:
assert int(conn.execute("PRAGMA user_version").fetchone()[0]) == SCHEMA_VERSION
columns = {row["name"] for row in conn.execute("PRAGMA table_info(files)")}
assert "reviewed_name" in columns
# the existing row survives, simply without an undo trail
row = conn.execute("SELECT state, reviewed_name FROM files").fetchone()
assert row["state"] == "kept"
assert row["reviewed_name"] is None
finally:
conn.close()

223
python_pkg/wsg_grabber/tests/test_downloader.py
View File

@ -1,223 +0,0 @@
"""Tests for the background worker.
Every case drives ``run_once`` directly. No real thread runs and no socket is
opened, so the suite stays deterministic and cannot leak either.
"""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net, store, store_threads
from python_pkg.wsg_grabber.models import (
DownloadEvent,
Indexed,
JsonResponse,
RemoteFile,
TaskKind,
ThreadRef,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_first_step_reads_the_thread_list(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
payload = [{"threads": [{"no": 7, "last_modified": 100}]}]
with patch.object(net, "get_json", return_value=_json(payload)):
assert worker.run_once() is TaskKind.SCAN
def test_scan_then_fetch_records_files(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
thread = {
"posts": [
{
"no": 1,
"tim": 1,
"ext": ".webm",
"md5": _MD5,
"fsize": 11,
"w": 4,
"h": 3,
"filename": "clip",
},
],
}
with patch.object(
net,
"get_json",
side_effect=[_json(listing), _json(thread, last_modified="stamp")],
):
worker.run_once()
worker.run_once()
assert store.known_md5s(deps.conn) == {_MD5}
assert store_threads.known_threads(deps.conn) == {7: 100}
assert store_threads.http_last_modified(deps.conn, 7) == "stamp"
assert any(isinstance(event, Indexed) for event in _drain(deps.events))
def test_an_unchanged_thread_costs_only_a_304(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store_threads.record_thread(deps.conn, ThreadRef(7, 50), "old-stamp")
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
with patch.object(
net,
"get_json",
side_effect=[_json(listing), _json(None, not_modified=True)],
) as fake:
worker.run_once()
worker.run_once()
assert fake.call_args.args[2] == "old-stamp"
assert store_threads.known_threads(deps.conn) == {7: 100}
assert store.known_md5s(deps.conn) == set()
def test_a_vanished_thread_is_marked_gone(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
with patch.object(
net,
"get_json",
side_effect=[_json(listing), _json(None, not_found=True)],
):
worker.run_once()
worker.run_once()
row = deps.conn.execute("SELECT status FROM threads WHERE thread_no=7").fetchone()
assert row["status"] == "gone"
def test_archive_threads_are_included_when_enabled(
deps: downloader.WorkerDeps,
) -> None:
deps.include_archive = True
worker = downloader.Worker(deps)
with patch.object(
net,
"get_json",
side_effect=[
_json([{"threads": [{"no": 7, "last_modified": 1}]}]),
_json([88]),
],
):
worker.run_once()
with patch.object(net, "get_json", side_effect=[_json({"posts": []})]) as fake:
worker.run_once()
assert "/thread/" in fake.call_args.args[1]
def test_a_board_without_an_archive_is_tolerated(
deps: downloader.WorkerDeps,
) -> None:
deps.include_archive = True
worker = downloader.Worker(deps)
with patch.object(
net,
"get_json",
side_effect=[_json([]), _json(None, not_found=True)],
):
assert worker.run_once() is TaskKind.SCAN

168
python_pkg/wsg_grabber/tests/test_downloader_conditional.py
View File

@ -1,168 +0,0 @@
"""Tests for conditional board fetches and stored stamps."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net, store_threads
from python_pkg.wsg_grabber.constants import THREADS_URL
from python_pkg.wsg_grabber.models import (
DownloadEvent,
JsonResponse,
RemoteFile,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_the_board_endpoints_are_fetched_conditionally(
deps: downloader.WorkerDeps,
) -> None:
"""Without this an idle reviewer re-fetches the board unconditionally forever."""
deps.include_archive = True
worker = downloader.Worker(deps)
stamp = "Mon, 27 Jul 2026 10:00:00 GMT"
with patch.object(
net,
"get_json",
return_value=_json([], last_modified=stamp),
) as fake:
worker.run_once()
sent_first = [call.args[2] for call in fake.call_args_list]
worker._catalog_fresh = False
fake.reset_mock()
worker.run_once()
sent_again = [call.args[2] for call in fake.call_args_list]
assert sent_first == [None, None] # nothing stored yet
assert sent_again == [stamp, stamp] # the stamp goes back
def test_a_304_on_the_board_keeps_the_stored_stamp(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
stamp = "Mon, 27 Jul 2026 10:00:00 GMT"
with patch.object(net, "get_json", return_value=_json([], last_modified=stamp)):
worker.run_once()
with patch.object(net, "get_json", return_value=_json(None, not_modified=True)):
worker._catalog_fresh = False
worker.run_once()
assert store_threads.cached_last_modified(deps.conn, THREADS_URL) == stamp
def test_a_worker_that_cannot_build_its_resources_reports_it(
tmp_path: Path,
) -> None:
"""Otherwise the reviewer waits on downloads that will never come."""
reported: list[str] = []
thread = downloader.start(
lambda: (_ for _ in ()).throw(OSError("no database")),
reported.append,
)
thread.join(timeout=5)
assert not thread.is_alive()
assert reported
assert "no database" in reported[0]

219
python_pkg/wsg_grabber/tests/test_downloader_lifecycle.py
View File

@ -1,219 +0,0 @@
"""Tests for worker idle announcements and shutdown."""
from __future__ import annotations
import queue
import threading
import time
from typing import TYPE_CHECKING, cast
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net
from python_pkg.wsg_grabber.constants import IDLE_RESCAN_S
from python_pkg.wsg_grabber.models import (
DownloadEvent,
JsonResponse,
RemoteFile,
ScanFinished,
TaskKind,
WorkerFailed,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_idle_announces_completion_exactly_once(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
with patch.object(net, "get_json", return_value=_json([])):
worker.run_once()
with patch.object(deps.stop, "wait"):
assert worker.run_once() is TaskKind.IDLE
assert worker.run_once() is TaskKind.IDLE
finished = [e for e in _drain(deps.events) if isinstance(e, ScanFinished)]
assert len(finished) == 1
def test_the_board_is_rescanned_after_the_quiet_period(
deps: downloader.WorkerDeps,
) -> None:
"""A finished sweep goes quiet, then polls again for new content."""
worker = downloader.Worker(deps)
with patch.object(net, "get_json", return_value=_json([])):
worker.run_once()
with patch.object(deps.stop, "wait"):
assert worker.run_once() is TaskKind.IDLE
with patch(
"python_pkg.wsg_grabber.downloader.time.monotonic",
return_value=time.monotonic() + IDLE_RESCAN_S + 1,
):
assert worker.run_once() is TaskKind.IDLE
# the quiet period has elapsed, so the next step walks the board again
assert worker.run_once() is TaskKind.SCAN
finished = [e for e in _drain(deps.events) if isinstance(e, ScanFinished)]
assert len(finished) == 1
def test_rate_limiting_waits_between_requests(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
listing = [{"threads": [{"no": 7, "last_modified": 100}]}]
with (
patch.object(
net,
"get_json",
side_effect=[_json(listing), _json({"posts": []})],
),
patch.object(deps.stop, "wait") as waited,
):
worker.run_once() # the first request is free
worker.run_once() # the second must be spaced out
delays = [call.args[0] for call in waited.call_args_list if call.args]
assert delays
assert max(delays) <= 1.0
def test_run_reports_a_crash_and_closes_resources(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
with patch.object(worker, "run_once", side_effect=RuntimeError("kaput")):
worker.run()
events = _drain(deps.events)
assert any(
isinstance(event, WorkerFailed) and event.message == "kaput" for event in events
)
cast("MagicMock", deps.session).close.assert_called_once()
def test_run_exits_cleanly_when_stopped(deps: downloader.WorkerDeps) -> None:
deps.stop.set()
worker = downloader.Worker(deps)
worker.run()
assert _drain(deps.events) == []
cast("MagicMock", deps.session).close.assert_called_once()
@pytest.mark.timeout(10)
def test_start_runs_and_joins_a_real_thread(tmp_path: Path) -> None:
"""The one test that starts a thread; it must stop on demand."""
incoming = tmp_path / "incoming"
incoming.mkdir()
stop = threading.Event()
stop.set()
def build() -> downloader.WorkerDeps:
"""Open the connection on the worker thread, which must own it."""
return downloader.WorkerDeps(
conn=db.open_index(tmp_path / "index.db"),
session=MagicMock(),
incoming=incoming,
events=queue.SimpleQueue(),
stop=stop,
include_archive=False,
)
thread = downloader.start(build)
thread.join(timeout=5)
assert not thread.is_alive()

161
python_pkg/wsg_grabber/tests/test_downloader_outcomes.py
View File

@ -1,161 +0,0 @@
"""Tests for how completed and failed downloads are recorded."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net, store
from python_pkg.wsg_grabber.models import (
Downloaded,
DownloadEvent,
DownloadResult,
FileReady,
JsonResponse,
Outcome,
RemoteFile,
TaskKind,
)
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_a_completed_download_is_renamed_and_announced(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
def fake_download(_session: object, transfer: net.Transfer) -> DownloadResult:
transfer.part_path.write_bytes(_BODY)
return DownloadResult(outcome=Outcome.COMPLETED, bytes_done=len(_BODY))
with patch.object(net, "download", side_effect=fake_download):
assert worker.run_once() is TaskKind.DOWNLOAD
events = _drain(deps.events)
ready = [event for event in events if isinstance(event, FileReady)]
assert len(ready) == 1
assert ready[0].item.md5 == _MD5
assert ready[0].item.path.exists()
assert not ready[0].item.path.name.endswith(".part")
assert ready[0].item.orig_name == "clip.webm"
assert any(isinstance(event, Downloaded) for event in events)
assert worker.downloaded == 1
assert store.counts(deps.conn)[FileState.READY.value] == 1
def test_a_404_is_terminal_and_never_retried(deps: downloader.WorkerDeps) -> None:
"""This is what stops a fallen-off thread being re-fetched every run."""
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.NOT_FOUND, bytes_done=0),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.GONE.value] == 1
assert store.pending_downloads(deps.conn) == 0

179
python_pkg/wsg_grabber/tests/test_downloader_part2.py
View File

@ -1,179 +0,0 @@
"""More tests for the background worker.
Split from test_downloader.py to stay under this repo's 500-line-per-file cap;
the fixtures are duplicated because each test module must stand alone.
"""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net, store
from python_pkg.wsg_grabber.models import (
DownloadEvent,
DownloadResult,
JsonResponse,
Outcome,
RemoteFile,
ThreadRef,
)
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_download_progress_is_persisted(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
def fake_download(_session: object, transfer: net.Transfer) -> DownloadResult:
assert transfer.on_progress is not None
transfer.on_progress(7)
return DownloadResult(outcome=Outcome.ABORTED, bytes_done=7)
with patch.object(net, "download", side_effect=fake_download):
worker.run_once()
assert deps.conn.execute("SELECT bytes_done FROM files").fetchone()[0] == 7
def test_claim_losing_a_race_is_a_no_op(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with (
patch.object(store, "claim_next", return_value=None),
patch.object(
net,
"download",
) as never,
):
worker._download_one()
never.assert_not_called()
def test_deleted_upstream_files_are_written_off(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
thread = {"posts": [{"no": 1, "md5": _MD5, "filedeleted": 1}]}
# Driven directly: with a file queued, run_once would always pick DOWNLOAD
# over SCAN, so the thread fetch under test would never happen.
with patch.object(net, "get_json", return_value=_json(thread)):
worker._fetch_thread(ThreadRef(7, 100))
assert store.counts(deps.conn)[FileState.GONE.value] == 1
assert store.pending_downloads(deps.conn) == 0
def test_an_already_downloaded_file_survives_upstream_deletion(
deps: downloader.WorkerDeps,
) -> None:
"""Once the bytes are ours, the post being deleted is irrelevant."""
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
store.mark_downloaded(deps.conn, _MD5, "1-x.webm")
thread = {"posts": [{"no": 1, "md5": _MD5, "filedeleted": 1}]}
with patch.object(net, "get_json", return_value=_json(thread)):
worker._fetch_thread(ThreadRef(7, 100))
assert store.state_of(deps.conn, _MD5) is FileState.READY

193
python_pkg/wsg_grabber/tests/test_downloader_retry.py
View File

@ -1,193 +0,0 @@
"""Tests for retrying and resuming interrupted downloads."""
from __future__ import annotations
import queue
import threading
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import db, downloader, net, store
from python_pkg.wsg_grabber.models import (
DownloadEvent,
DownloadResult,
JsonResponse,
Outcome,
RemoteFile,
)
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
_MD5 = "Nz9OEKdMuMZEdYE6eLTKmA=="
_BODY = b"video bytes"
@pytest.fixture
def deps(tmp_path: Path) -> Iterator[downloader.WorkerDeps]:
"""Build worker dependencies backed by a sandboxed index.
Args:
tmp_path: Per-test temporary directory.
Yields:
downloader.WorkerDeps: Ready dependencies; resources closed after.
"""
incoming = tmp_path / "incoming"
incoming.mkdir()
conn = db.open_index(tmp_path / "index.db")
session = MagicMock()
built = downloader.WorkerDeps(
conn=conn,
session=session,
incoming=incoming,
events=queue.SimpleQueue(),
stop=threading.Event(),
include_archive=False,
)
try:
yield built
finally:
conn.close()
def _json(
payload: object,
*,
last_modified: str | None = None,
not_modified: bool = False,
not_found: bool = False,
) -> JsonResponse:
"""Build a JsonResponse with sensible defaults.
Args:
payload: Decoded body.
last_modified: Header value the response carried.
not_modified: Whether the server answered 304.
not_found: Whether the thread has gone.
Returns:
JsonResponse: Response double.
"""
return JsonResponse(
payload=payload,
last_modified=last_modified,
not_modified=not_modified,
not_found=not_found,
)
def _remote(md5: str = _MD5, tim: int = 1) -> RemoteFile:
"""Build a RemoteFile.
Args:
md5: Identity.
tim: CDN stem.
Returns:
RemoteFile: Fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=len(_BODY),
width=480,
height=360,
thread_no=7,
post_no=tim,
)
def _drain(events: queue.SimpleQueue[DownloadEvent]) -> list[DownloadEvent]:
"""Pull everything currently queued.
Args:
events: Queue to empty.
Returns:
list[DownloadEvent]: Events in order.
"""
out: list[DownloadEvent] = []
while not events.empty():
out.append(events.get_nowait())
return out
def test_a_transient_failure_is_retried(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.TRANSIENT, bytes_done=0),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.FAILED.value] == 1
assert store.pending_downloads(deps.conn) == 1
def test_a_transient_failure_gives_up_after_the_attempt_budget(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.TRANSIENT, bytes_done=0),
):
for _ in range(3):
worker.run_once()
assert store.counts(deps.conn)[FileState.GONE.value] == 1
def test_a_retry_after_pauses_the_worker(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with (
patch.object(
net,
"download",
return_value=DownloadResult(
outcome=Outcome.TRANSIENT,
bytes_done=0,
retry_after=5.0,
),
),
patch.object(deps.stop, "wait") as waited,
):
worker.run_once()
assert any(call.args and call.args[0] == 5.0 for call in waited.call_args_list)
def test_a_checksum_mismatch_marks_the_file_corrupt(
deps: downloader.WorkerDeps,
) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.CHECKSUM_MISMATCH, bytes_done=0),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.CORRUPT.value] == 1
def test_an_aborted_download_stays_resumable(deps: downloader.WorkerDeps) -> None:
worker = downloader.Worker(deps)
store.record_files(deps.conn, [_remote()])
with patch.object(
net,
"download",
return_value=DownloadResult(outcome=Outcome.ABORTED, bytes_done=4),
):
worker.run_once()
assert store.counts(deps.conn)[FileState.FAILED.value] == 1
assert store.pending_downloads(deps.conn) == 1

92
python_pkg/wsg_grabber/tests/test_housekeeping.py
View File

@ -1,92 +0,0 @@
"""Tests for the around-a-verdict chores and for undo finding moved keeps."""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import housekeeping, paths
from python_pkg.wsg_grabber.models import Verdict
from python_pkg.wsg_grabber.tests.conftest import item, reviewed
if TYPE_CHECKING:
from collections.abc import Callable
@pytest.fixture
def prune_calls(monkeypatch: pytest.MonkeyPatch) -> list[object]:
"""Replace the prune with a recorder.
Args:
monkeypatch: Patching helper.
Returns:
list[object]: One entry per prune call.
"""
calls: list[object] = []
monkeypatch.setattr(
housekeeping,
"prune_trash",
calls.append,
)
return calls
def test_startup_migrates_then_prunes(
monkeypatch: pytest.MonkeyPatch,
prune_calls: list[object],
) -> None:
order: list[str] = []
monkeypatch.setattr(
housekeeping,
"migrate_keep",
lambda _conn: order.append("migrate"),
)
housekeeping.on_startup("conn")
assert order == ["migrate"]
assert prune_calls == ["conn"]
@pytest.mark.parametrize(
("choice", "expected"),
[(Verdict.SKIP, ["conn"]), (Verdict.KEEP, [])],
)
def test_only_a_pass_triggers_a_prune(
prune_calls: list[object],
choice: Verdict,
expected: list[object],
) -> None:
housekeeping.after_verdict("conn", choice)
assert prune_calls == expected
def test_a_pass_is_located_in_trash() -> None:
action = reviewed(item(), Verdict.SKIP)
assert housekeeping.locate_reviewed(action) == paths.trash_dir() / "a.webm"
@pytest.mark.parametrize(
"place",
[
paths.keep_dir,
lambda: paths.cloud_keep_dir(2001),
paths.legacy_keep_dir,
paths.downloads_dir,
lambda: paths.cloud_root() / "Media" / "2001" / "03",
],
)
def test_a_keep_is_found_wherever_it_ended_up(place: Callable[[], Path]) -> None:
paths.cloud_root().mkdir(parents=True)
directory = place()
directory.mkdir(parents=True, exist_ok=True)
(directory / "a.webm").write_bytes(b"v")
action = reviewed(item(), Verdict.KEEP)
assert housekeeping.locate_reviewed(action) == directory / "a.webm"
def test_a_keep_that_is_nowhere_points_at_the_current_keep_dir() -> None:
action = reviewed(item(), Verdict.KEEP)
assert housekeeping.locate_reviewed(action) == paths.keep_dir() / "a.webm"

163
python_pkg/wsg_grabber/tests/test_logs.py
View File

@ -1,163 +0,0 @@
"""Tests for the structured event log."""
from __future__ import annotations
import json
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import logs
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
@pytest.fixture(autouse=True)
def _closed() -> Iterator[None]:
"""Make sure no test leaves the log file open.
Yields:
None: Control returns to the test.
"""
try:
yield
finally:
logs.stop()
def _records(path: Path) -> list[dict[str, object]]:
"""Parse a session log.
Args:
path: The JSONL file.
Returns:
list[dict[str, object]]: One dict per line.
"""
return [
json.loads(line)
for line in path.read_text(encoding="utf-8").splitlines()
if line
]
def test_starting_creates_a_log_and_announces_itself(tmp_path: Path) -> None:
path = logs.start()
assert path.parent == logs.logs_dir()
assert str(path).startswith(str(tmp_path))
assert logs.current_path() == path
first = _records(path)[0]
assert first["event"] == "session.start"
assert first["seq"] == 1
assert first["level"] == "info"
def test_every_record_carries_the_standard_fields() -> None:
path = logs.start()
logs.event("thing.happened", detail="x", count=3)
record = _records(path)[-1]
assert set(record) >= {"ts", "seq", "level", "event", "thread"}
assert record["event"] == "thing.happened"
assert record["detail"] == "x"
assert record["count"] == 3
def test_sequence_numbers_are_monotonic() -> None:
path = logs.start()
for index in range(5):
logs.event("tick", index=index)
seqs = [int(str(record["seq"])) for record in _records(path)]
assert seqs == sorted(seqs)
assert len(set(seqs)) == len(seqs)
def test_level_helpers_set_the_level() -> None:
path = logs.start("debug")
logs.debug("a")
logs.warning("b")
logs.error("c")
levels = [record["level"] for record in _records(path)[1:]]
assert levels == ["debug", "warning", "error"]
def test_records_below_the_threshold_are_dropped() -> None:
path = logs.start("warning")
logs.debug("dropped")
logs.event("also-dropped")
logs.warning("kept")
events = [record["event"] for record in _records(path)]
assert "dropped" not in events
assert "also-dropped" not in events
assert "kept" in events
def test_an_unknown_level_name_falls_back_to_info() -> None:
path = logs.start("nonsense")
logs.event("recorded", level="nonsense")
assert any(record["event"] == "recorded" for record in _records(path))
def test_values_that_cannot_be_serialised_are_stringified() -> None:
path = logs.start()
logs.event("odd", value=object(), where=tmp_marker())
record = _records(path)[-1]
assert isinstance(record["value"], str)
def tmp_marker() -> object:
"""Return a non-JSON-encodable value.
Returns:
object: Something json cannot encode directly.
"""
return {1, 2, 3}
def test_events_before_start_go_nowhere_and_do_not_raise() -> None:
logs.stop()
logs.event("ignored")
assert logs.current_path() is None
def test_echo_mirrors_records_to_stderr(
capsys: pytest.CaptureFixture[str],
) -> None:
logs.start(echo=True)
logs.event("mirrored", detail="yes")
captured = capsys.readouterr().err
assert "mirrored" in captured
assert json.loads(captured.strip().splitlines()[-1])["detail"] == "yes"
def test_starting_twice_rotates_to_a_new_file() -> None:
first = logs.start()
logs.event("one")
second = logs.start()
logs.event("two")
assert logs.current_path() == second
assert "one" in first.read_text(encoding="utf-8")
def test_stop_is_idempotent() -> None:
logs.start()
logs.stop()
logs.stop()
assert logs.current_path() is None
def test_the_log_lands_inside_the_sandbox(tmp_path: Path) -> None:
"""The suite must never write a log into the real data directory."""
assert str(logs.logs_dir()).startswith(str(tmp_path))
def test_each_session_numbers_its_own_records_from_one() -> None:
"""seq orders records within one file, so it must restart per session."""
first = logs.start()
logs.event("a")
logs.event("b")
second = logs.start()
assert _records(second)[0]["seq"] == 1
assert len(_records(first)) == 3

30
python_pkg/wsg_grabber/tests/test_main_entry.py
View File

@ -1,30 +0,0 @@
"""Tests for the ``python -m python_pkg.wsg_grabber`` entry point."""
from __future__ import annotations
import runpy
import sys
from unittest.mock import patch
import pytest
_MODULE = "python_pkg.wsg_grabber.__main__"
def test_running_as_a_module_exits_with_the_cli_status() -> None:
# pytest-randomly shuffles test order, so a previous import must not linger
sys.modules.pop(_MODULE, None)
with (
patch("python_pkg.wsg_grabber.cli.main", return_value=3) as entry,
pytest.raises(SystemExit) as exit_info,
):
runpy.run_module(_MODULE, run_name="__main__")
assert exit_info.value.code == 3
entry.assert_called_once_with()
def test_importing_the_module_has_no_side_effect() -> None:
sys.modules.pop(_MODULE, None)
with patch("python_pkg.wsg_grabber.cli.main") as entry:
runpy.run_module(_MODULE, run_name="not_main")
entry.assert_not_called()

130
python_pkg/wsg_grabber/tests/test_migrate_keep.py
View File

@ -1,130 +0,0 @@
"""Tests for moving kept videos out of the data dir."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import db, paths, store, store_verdicts
from python_pkg.wsg_grabber._migrate_keep import migrate_keep
from python_pkg.wsg_grabber.models import RemoteFile
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
import sqlite3
@pytest.fixture
def conn() -> Iterator[sqlite3.Connection]:
"""Open a sandboxed index.
Yields:
sqlite3.Connection: Closed afterwards.
"""
paths.ensure_dirs()
opened = db.open_index(paths.db_path())
try:
yield opened
finally:
opened.close()
def _kept(conn: sqlite3.Connection, name: str, payload: bytes) -> None:
"""Record a keep whose file sits in the legacy keep/.
Args:
conn: Open connection.
name: Filename.
payload: File contents.
"""
store.record_files(
conn,
[
RemoteFile(
md5=f"md5-{name}",
tim=1,
ext=".webm",
orig_name=name,
fsize=len(payload),
width=1,
height=1,
thread_no=1,
post_no=1,
),
],
)
store.mark_downloaded(conn, f"md5-{name}", name)
store_verdicts.record_verdict(conn, f"md5-{name}", FileState.KEPT, name)
paths.legacy_keep_dir().mkdir(parents=True, exist_ok=True)
(paths.legacy_keep_dir() / name).write_bytes(payload)
def test_moves_everything_and_removes_the_empty_legacy_dir(
conn: sqlite3.Connection,
) -> None:
_kept(conn, "a.webm", b"aa")
_kept(conn, "b.webm", b"bb")
assert migrate_keep(conn) == 2
assert (paths.keep_dir() / "a.webm").read_bytes() == b"aa"
assert (paths.keep_dir() / "b.webm").read_bytes() == b"bb"
assert not paths.legacy_keep_dir().exists()
def test_no_legacy_dir_is_a_no_op(conn: sqlite3.Connection) -> None:
assert not paths.legacy_keep_dir().exists()
assert migrate_keep(conn) == 0
def test_identical_copy_at_destination_drops_the_legacy_file(
conn: sqlite3.Connection,
) -> None:
_kept(conn, "a.webm", b"aa")
paths.keep_dir().mkdir(parents=True, exist_ok=True)
(paths.keep_dir() / "a.webm").write_bytes(b"zz")
assert migrate_keep(conn) == 0
assert (paths.keep_dir() / "a.webm").read_bytes() == b"zz"
assert not paths.legacy_keep_dir().exists()
def test_name_clash_with_different_size_is_suffixed_and_reindexed(
conn: sqlite3.Connection,
) -> None:
_kept(conn, "a.webm", b"aa")
paths.keep_dir().mkdir(parents=True, exist_ok=True)
(paths.keep_dir() / "a.webm").write_bytes(b"other")
assert migrate_keep(conn) == 1
assert (paths.keep_dir() / "a.webm").read_bytes() == b"other"
assert (paths.keep_dir() / "a-2.webm").read_bytes() == b"aa"
row = conn.execute(
"SELECT reviewed_name FROM files WHERE md5 = ?",
("md5-a.webm",),
).fetchone()
assert row["reviewed_name"] == "a-2.webm"
def test_subdirectories_are_left_alone(conn: sqlite3.Connection) -> None:
paths.legacy_keep_dir().mkdir(parents=True)
(paths.legacy_keep_dir() / "nested").mkdir()
assert migrate_keep(conn) == 0
assert (paths.legacy_keep_dir() / "nested").is_dir()
def test_a_file_that_vanishes_mid_move_is_skipped(
conn: sqlite3.Connection,
monkeypatch: pytest.MonkeyPatch,
) -> None:
_kept(conn, "a.webm", b"aa")
monkeypatch.setattr(
"python_pkg.wsg_grabber._migrate_keep.files.apply_move",
lambda _move: False,
)
assert migrate_keep(conn) == 0

250
python_pkg/wsg_grabber/tests/test_net.py
View File

@ -1,250 +0,0 @@
"""Tests for the HTTP layer.
No socket is opened: ``requests.Session`` is replaced with a mock whose
``iter_content`` yields real bytes, so the md5 verification path computes a
genuine digest rather than comparing two mocks.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
import pytest
import requests
from python_pkg.wsg_grabber import net
from python_pkg.wsg_grabber.constants import API_HEADERS, MEDIA_HEADERS
from python_pkg.wsg_grabber.models import Outcome
if TYPE_CHECKING:
from pathlib import Path
_BODY = b"fake webm bytes, but real bytes" * 40
def _response(
status: int = 200,
*,
body: bytes = b"",
headers: dict[str, str] | None = None,
text: str = "",
) -> MagicMock:
"""Build a stand-in for a requests Response.
Args:
status: HTTP status code.
body: Payload streamed by ``iter_content``.
headers: Response headers.
text: Body for JSON decoding.
Returns:
MagicMock: Response double.
"""
response = MagicMock()
response.status_code = status
response.headers = headers or {}
response.text = text
response.content = text.encode("utf-8")
response.iter_content.return_value = [body] if body else []
return response
def _transfer(tmp_path: Path, *, md5: str | None = None, size: int = 0) -> net.Transfer:
"""Build a Transfer pointing at a sandboxed part file.
Args:
tmp_path: Per-test temporary directory.
md5: Expected digest; defaults to the real digest of ``_BODY``.
size: Expected file size.
Returns:
net.Transfer: Transfer description.
"""
return net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5=md5 if md5 is not None else net.digest_of(_BODY),
expected_size=size,
should_stop=lambda: False,
)
def test_digest_matches_the_field_the_api_publishes() -> None:
"""Verified against a real /wsg/ file during design."""
assert net.digest_of(b"") == "1B2M2Y8AsgTpgAmY7PhCfg=="
assert len(net.digest_of(_BODY)) == 24
def test_build_session_carries_the_api_headers() -> None:
session = net.build_session()
try:
assert session.headers["User-Agent"] == API_HEADERS["User-Agent"]
assert session.headers["Referer"] == API_HEADERS["Referer"]
finally:
session.close()
def test_get_json_decodes_a_fresh_payload() -> None:
session = MagicMock()
session.get.return_value = _response(
200,
text='{"posts": [1]}',
headers={"Last-Modified": "Mon, 27 Jul 2026 08:00:00 GMT"},
)
result = net.get_json(session, "https://a.4cdn.org/wsg/threads.json")
assert result.payload == {"posts": [1]}
assert result.last_modified == "Mon, 27 Jul 2026 08:00:00 GMT"
assert not result.not_modified
assert session.get.call_args.kwargs["headers"] == {}
def test_get_json_sends_if_modified_since_and_honours_304() -> None:
"""An unchanged thread must cost a 304, not a re-parse."""
session = MagicMock()
session.get.return_value = _response(304)
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
result = net.get_json(session, "https://a.4cdn.org/wsg/thread/1.json", stamp)
assert result.not_modified
assert result.payload is None
assert result.last_modified == stamp
assert session.get.call_args.kwargs["headers"] == {"If-Modified-Since": stamp}
@pytest.mark.parametrize("status", [404, 410])
def test_get_json_reports_a_vanished_thread(status: int) -> None:
session = MagicMock()
session.get.return_value = _response(status)
result = net.get_json(session, "https://a.4cdn.org/wsg/thread/1.json")
assert result.not_found
assert result.payload is None
def test_get_json_raises_on_other_failures() -> None:
session = MagicMock()
response = _response(500)
response.raise_for_status.side_effect = requests.HTTPError("boom")
session.get.return_value = response
with pytest.raises(requests.HTTPError):
net.get_json(session, "https://a.4cdn.org/wsg/threads.json")
def test_download_writes_and_verifies_the_file(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = _transfer(tmp_path)
result = net.download(session, transfer)
assert result.outcome is Outcome.COMPLETED
assert result.bytes_done == len(_BODY)
assert transfer.part_path.read_bytes() == _BODY
assert (
session.get.call_args.kwargs["headers"]["Referer"] == MEDIA_HEADERS["Referer"]
)
assert "Range" not in session.get.call_args.kwargs["headers"]
def test_download_reports_progress(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
seen: list[int] = []
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5=net.digest_of(_BODY),
expected_size=0,
should_stop=lambda: False,
on_progress=seen.append,
)
net.download(session, transfer)
assert seen == [len(_BODY)]
def test_download_rejects_a_body_whose_digest_disagrees(tmp_path: Path) -> None:
"""A mangled transfer must never reach the reviewer."""
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = _transfer(tmp_path, md5="A" * 24)
result = net.download(session, transfer)
assert result.outcome is Outcome.CHECKSUM_MISMATCH
assert not transfer.part_path.exists()
@pytest.mark.parametrize("status", [404, 410])
def test_download_marks_a_missing_file_not_found(tmp_path: Path, status: int) -> None:
session = MagicMock()
session.get.return_value = _response(status)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.NOT_FOUND
def test_download_reads_retry_after_on_429(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(429, headers={"Retry-After": "45"})
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.retry_after == pytest.approx(45.0)
def test_download_treats_server_errors_as_transient(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(503)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.retry_after is None
def test_download_treats_an_unexpected_status_as_transient(tmp_path: Path) -> None:
session = MagicMock()
session.get.return_value = _response(302)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.message is not None
assert "302" in result.message
def test_download_turns_a_network_error_into_a_transient_result(
tmp_path: Path,
) -> None:
session = MagicMock()
session.get.side_effect = requests.ConnectionError("no route")
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.message == "no route"
def test_download_resumes_a_partial_file(tmp_path: Path) -> None:
part = tmp_path / "1.webm.part"
part.write_bytes(_BODY[:100])
session = MagicMock()
session.get.return_value = _response(206, body=_BODY[100:])
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert session.get.call_args.kwargs["headers"]["Range"] == "bytes=100-"
assert result.outcome is Outcome.COMPLETED
assert part.read_bytes() == _BODY
def test_a_server_ignoring_range_restarts_rather_than_appending(
tmp_path: Path,
) -> None:
"""Replying 200 to a Range request means the whole file is coming again."""
part = tmp_path / "1.webm.part"
part.write_bytes(_BODY[:100])
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.COMPLETED
assert part.read_bytes() == _BODY

184
python_pkg/wsg_grabber/tests/test_net_resume.py
View File

@ -1,184 +0,0 @@
"""Tests for resuming and discarding partial downloads."""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
import pytest
from python_pkg.wsg_grabber import net
from python_pkg.wsg_grabber.constants import API_HEADERS
from python_pkg.wsg_grabber.models import Outcome
if TYPE_CHECKING:
from pathlib import Path
_BODY = b"fake webm bytes, but real bytes" * 40
def _response(
status: int = 200,
*,
body: bytes = b"",
headers: dict[str, str] | None = None,
text: str = "",
) -> MagicMock:
"""Build a stand-in for a requests Response.
Args:
status: HTTP status code.
body: Payload streamed by ``iter_content``.
headers: Response headers.
text: Body for JSON decoding.
Returns:
MagicMock: Response double.
"""
response = MagicMock()
response.status_code = status
response.headers = headers or {}
response.text = text
response.content = text.encode("utf-8")
response.iter_content.return_value = [body] if body else []
return response
def _transfer(tmp_path: Path, *, md5: str | None = None, size: int = 0) -> net.Transfer:
"""Build a Transfer pointing at a sandboxed part file.
Args:
tmp_path: Per-test temporary directory.
md5: Expected digest; defaults to the real digest of ``_BODY``.
size: Expected file size.
Returns:
net.Transfer: Transfer description.
"""
return net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5=md5 if md5 is not None else net.digest_of(_BODY),
expected_size=size,
should_stop=lambda: False,
)
def test_an_oversized_leftover_is_discarded_before_requesting(
tmp_path: Path,
) -> None:
part = tmp_path / "1.webm.part"
part.write_bytes(b"x" * 9999)
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert "Range" not in session.get.call_args.kwargs["headers"]
assert result.outcome is Outcome.COMPLETED
def test_a_rejected_range_restarts_from_zero(tmp_path: Path) -> None:
part = tmp_path / "1.webm.part"
part.write_bytes(_BODY[:100])
session = MagicMock()
session.get.return_value = _response(416)
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=part,
expected_md5=net.digest_of(_BODY),
expected_size=len(_BODY),
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.TRANSIENT
assert not part.exists()
def test_shutdown_aborts_mid_stream_and_keeps_the_partial(tmp_path: Path) -> None:
"""Stopping must be immediate and must leave a resumable .part behind."""
session = MagicMock()
response = _response(200)
response.iter_content.return_value = [b"a" * 10, b"b" * 10, b"c" * 10]
session.get.return_value = response
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5="whatever" + "=" * 16,
expected_size=30,
should_stop=lambda: True,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.ABORTED
def test_empty_chunks_are_skipped(tmp_path: Path) -> None:
"""requests yields keep-alive chunks; they must not corrupt the digest."""
session = MagicMock()
response = _response(200)
response.iter_content.return_value = [b"", _BODY, b""]
session.get.return_value = response
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.COMPLETED
def test_size_of_missing_file_is_zero(tmp_path: Path) -> None:
assert net.size_of(tmp_path / "nope") == 0
def test_a_compressed_body_is_refused(tmp_path: Path) -> None:
"""We ask for identity; requests would silently expand a gzip bomb."""
session = MagicMock()
session.get.return_value = _response(
200,
body=_BODY,
headers={"Content-Encoding": "gzip"},
)
result = net.download(session, _transfer(tmp_path))
assert result.outcome is Outcome.TRANSIENT
assert result.message is not None
assert "Content-Encoding" in result.message
def test_a_body_larger_than_advertised_is_abandoned(tmp_path: Path) -> None:
"""A server that lies about its size must not fill the disk."""
session = MagicMock()
response = _response(200)
response.iter_content.return_value = [b"x" * (1024 * 1024)] * 4
session.get.return_value = response
transfer = net.Transfer(
url="https://i.4cdn.org/wsg/1.webm",
part_path=tmp_path / "1.webm.part",
expected_md5=net.digest_of(_BODY),
expected_size=16,
should_stop=lambda: False,
)
result = net.download(session, transfer)
assert result.outcome is Outcome.TRANSIENT
assert not transfer.part_path.exists()
def test_an_unknown_size_is_not_treated_as_over_budget(tmp_path: Path) -> None:
"""fsize 0 means the API did not say; that must not block the download."""
session = MagicMock()
session.get.return_value = _response(200, body=_BODY)
result = net.download(session, _transfer(tmp_path, size=0))
assert result.outcome is Outcome.COMPLETED
def test_an_oversized_api_response_is_refused() -> None:
session = MagicMock()
response = _response(200)
response.content = b"[" + b"1," * 20_000_000 + b"1]"
session.get.return_value = response
with pytest.raises(ValueError, match="exceeded"):
net.get_json(session, "https://a.4cdn.org/wsg/threads.json")
def test_the_api_session_refuses_compression() -> None:
assert API_HEADERS["Accept-Encoding"] == "identity"

151
python_pkg/wsg_grabber/tests/test_paths.py
View File

@ -1,151 +0,0 @@
"""Tests for storage locations and for the isolation guarantee itself."""
from __future__ import annotations
import ast
import os
from pathlib import Path
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import paths
if TYPE_CHECKING:
import pytest
_HOME_CAPTURING_CALLS = frozenset({"home", "expanduser", "getenv", "expandvars"})
def test_data_dir_follows_xdg_data_home(tmp_path: Path) -> None:
assert paths.data_dir() == Path(os.environ["XDG_DATA_HOME"]) / "wsg_grabber"
assert str(paths.data_dir()).startswith(str(tmp_path))
def test_data_dir_falls_back_to_dot_local_share(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
) -> None:
monkeypatch.delenv("XDG_DATA_HOME", raising=False)
expected = Path.home() / ".local" / "share" / "wsg_grabber"
assert paths.data_dir() == expected
assert str(paths.data_dir()).startswith(str(tmp_path))
def test_derived_paths_live_under_the_data_dir() -> None:
root = paths.data_dir()
for candidate in (
paths.db_path(),
paths.incoming_dir(),
paths.legacy_keep_dir(),
paths.trash_dir(),
paths.ipc_socket_path(),
):
assert candidate.parent == root
def test_keep_dir_falls_back_to_downloads_without_the_cloud_folder() -> None:
assert not paths.cloud_root().exists()
assert paths.keep_dir() == Path.home() / "Downloads"
def test_keep_dir_uses_the_cloud_media_folder_for_this_year() -> None:
paths.cloud_root().mkdir(parents=True)
kept = paths.keep_dir()
assert kept.parent.parent == paths.cloud_root() / "Media"
assert kept.name == "wsg"
assert kept.parent.name.isdigit()
assert kept == paths.cloud_keep_dir(int(kept.parent.name))
def test_kept_candidates_cover_every_place_a_keep_can_end_up() -> None:
media = paths.cloud_root() / "Media"
for sub in ("2025/wsg", "2026/wsg", "2026/09", "2026/notes.txt"):
(media / sub).parent.mkdir(parents=True, exist_ok=True)
(media / "2025/wsg").mkdir(parents=True)
(media / "2026/wsg").mkdir(parents=True)
(media / "2026/09").mkdir(parents=True)
(media / "2026/notes.txt").write_text("")
candidates = paths.kept_candidates()
assert candidates[0] == paths.keep_dir()
assert candidates[1:3] == [media / "2025/wsg", media / "2026/wsg"]
assert candidates[3:5] == [paths.legacy_keep_dir(), paths.downloads_dir()]
assert candidates[5:] == [media / "2026/09"]
def test_socket_path_names_the_board() -> None:
assert paths.ipc_socket_path().name == "mpv-wsg.sock"
def test_ensure_dirs_creates_every_directory() -> None:
paths.ensure_dirs()
for directory in (
paths.data_dir(),
paths.incoming_dir(),
paths.keep_dir(),
paths.trash_dir(),
):
assert directory.is_dir()
def test_ensure_dirs_is_idempotent() -> None:
paths.ensure_dirs()
paths.ensure_dirs()
assert paths.incoming_dir().is_dir()
def _module_level_home_calls(source: str) -> list[str]:
"""Return names of home-resolving calls made outside any function body.
Args:
source: Python source text of one module.
Returns:
list[str]: Offending attribute/function names, empty when the module is
clean.
"""
tree = ast.parse(source)
offenders: list[str] = []
for node in tree.body:
for inner in ast.walk(node):
if isinstance(inner, (ast.FunctionDef, ast.AsyncFunctionDef)):
break
else:
offenders.extend(_home_calls_in(node))
return offenders
def _home_calls_in(node: ast.AST) -> list[str]:
"""Collect home-resolving call names anywhere beneath *node*.
Args:
node: AST node to scan.
Returns:
list[str]: Names such as ``home`` or ``expanduser``.
"""
found: list[str] = []
for inner in ast.walk(node):
if not isinstance(inner, ast.Call):
continue
func = inner.func
name = func.attr if isinstance(func, ast.Attribute) else None
if name is None and isinstance(func, ast.Name):
name = func.id
if name in _HOME_CAPTURING_CALLS:
found.append(name)
return found
def test_no_module_captures_a_home_directory_at_import_time() -> None:
"""Guard the whole package, not just paths.py.
A constant like ``STATE = Path.home() / "x"`` is evaluated on import and
can never be redirected afterwards, which is exactly how a previous package
in this repo leaked test writes into real user data.
"""
package_root = Path(paths.__file__).parent
offenders: dict[str, list[str]] = {}
for source_file in sorted(package_root.glob("*.py")):
calls = _module_level_home_calls(source_file.read_text(encoding="utf-8"))
if calls:
offenders[source_file.name] = calls
assert offenders == {}

229
python_pkg/wsg_grabber/tests/test_player.py
View File

@ -1,229 +0,0 @@
"""Tests for the mpv subprocess driver.
No mpv binary is ever started: ``subprocess.Popen`` and ``socket.socket`` are
patched in the module's own namespace.
"""
from __future__ import annotations
import json
from pathlib import Path
import subprocess
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import player
if TYPE_CHECKING:
from collections.abc import Iterator
def _socket_mock(**kwargs: object) -> MagicMock:
"""Build a socket stand-in whose reader thread terminates immediately.
An unconfigured MagicMock returns a MagicMock from recv(), which is truthy
and instant, so the drain loop spins as fast as the CPU allows -- and
MagicMock records every call, so memory grows without bound. That once
reached 21 GB and took the machine down, hence this helper: every socket
mock yields EOF unless a test deliberately says otherwise.
Args:
**kwargs: Attributes to set on the mock.
Returns:
MagicMock: Socket double.
"""
sock = MagicMock()
sock.recv.return_value = b""
for name, value in kwargs.items():
setattr(sock, name, value)
return sock
@pytest.fixture(autouse=True)
def _safe_sockets() -> Iterator[None]:
"""Stop a stray socket mock from turning the drain loop into a busy loop.
``MpvPlayer`` runs a reader thread that calls ``recv`` until EOF. An
unconfigured MagicMock returns a MagicMock -- truthy, instant, and it
records every call -- so the loop spins and memory grows without bound.
That once reached 21 GB and OOM-killed the machine. Defaulting the module's
socket factory to something that reports EOF means a test has to opt into
the hazard deliberately.
Yields:
None: Control returns to the test.
"""
with patch(
"python_pkg.wsg_grabber.player.socket.socket",
side_effect=lambda *_a, **_k: _socket_mock(),
):
yield
@pytest.fixture
def started(tmp_path: Path) -> Iterator[tuple[player.MpvPlayer, MagicMock, MagicMock]]:
"""Construct an MpvPlayer with its process and socket mocked out.
Args:
tmp_path: Per-test temporary directory.
Yields:
tuple: The player, its process mock and its socket mock.
"""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
def create_socket(_seconds: float) -> None:
"""Stand in for mpv finally creating its control socket."""
ipc.write_bytes(b"")
# __init__ unlinks any stale socket first, so it has to appear *during*
# the wait rather than being pre-created.
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch("python_pkg.wsg_grabber.player.time.sleep", side_effect=create_socket),
):
yield player.MpvPlayer(1234, ipc), process, sock
def test_argv_embeds_into_the_given_window(tmp_path: Path) -> None:
argv = player.build_argv(999, tmp_path / "s.sock")
assert argv[0] == "mpv"
assert "--wid=999" in argv
assert f"--input-ipc-server={tmp_path / 's.sock'}" in argv
def test_argv_declines_the_keyboard_so_tk_can_have_it(tmp_path: Path) -> None:
"""With --wid mpv's child window would otherwise swallow every keypress."""
argv = player.build_argv(1, tmp_path / "s.sock")
assert "--input-vo-keyboard=no" in argv
assert "--input-default-bindings=no" in argv
def test_argv_loops_and_stays_open(tmp_path: Path) -> None:
argv = player.build_argv(1, tmp_path / "s.sock")
assert "--loop-file=inf" in argv
assert "--idle=yes" in argv
assert "--keep-open=yes" in argv
def test_encode_produces_one_json_line() -> None:
payload = player.encode(["loadfile", "a.webm", "replace"])
assert payload.endswith(b"\n")
assert json.loads(payload) == {
"command": ["loadfile", "a.webm", "replace"],
}
def test_play_sends_loadfile(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
mpv, _process, sock = started
mpv.play(tmp_path / "clip.webm")
sock.sendall.assert_called_once_with(
player.encode(["loadfile", str(tmp_path / "clip.webm"), "replace"]),
)
def test_stop_blanks_the_video(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, _process, sock = started
mpv.stop()
assert sock.sendall.call_args.args[0] == player.encode(["stop"])
def test_a_dead_socket_does_not_raise(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
"""mpv can exit underneath us; is_alive reports that, sending must not blow up."""
mpv, _process, sock = started
sock.sendall.side_effect = BrokenPipeError("gone")
mpv.play(tmp_path / "clip.webm")
def test_is_alive_follows_the_process(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, process, _sock = started
assert mpv.is_alive()
process.poll.return_value = 0
assert not mpv.is_alive()
def test_close_quits_cleanly_and_removes_the_socket(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
mpv, process, sock = started
mpv.close()
assert sock.sendall.call_args.args[0] == player.encode(["quit"])
process.wait.assert_called_once()
process.kill.assert_not_called()
sock.close.assert_called_once()
assert not (tmp_path / "mpv.sock").exists()
def test_close_kills_an_mpv_that_ignores_quit(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""An orphaned mpv would leave a window on screen after the reviewer exits."""
mpv, process, _sock = started
process.wait.side_effect = [subprocess.TimeoutExpired("mpv", 1), None]
mpv.close()
process.kill.assert_called_once()
def test_a_socket_that_never_appears_is_a_timeout(tmp_path: Path) -> None:
ipc = tmp_path / "never.sock"
process = MagicMock()
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.time.sleep"),
patch(
"python_pkg.wsg_grabber.player.time.monotonic",
side_effect=[0.0, 0.0, 9999.0],
),
pytest.raises(TimeoutError, match="did not create"),
):
player.MpvPlayer(1, ipc)
process.kill.assert_called_once()
def test_the_socket_is_awaited_rather_than_assumed(tmp_path: Path) -> None:
"""mpv takes a moment to create the socket; connecting too early fails."""
ipc = tmp_path / "late.sock"
process = MagicMock()
sock = _socket_mock()
waits = {"n": 0}
def create_on_third_try(_seconds: float) -> None:
waits["n"] += 1
if waits["n"] >= 3:
ipc.write_bytes(b"")
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=create_on_third_try,
),
):
player.MpvPlayer(1, ipc)
assert waits["n"] == 3
def test_the_player_never_sets_geometry() -> None:
"""--wid makes mpv fill its parent; setting geometry shrinks the video."""
argv = player.build_argv(1, Path("sock"))
assert not any(arg.startswith("--geometry") for arg in argv)
assert not hasattr(player.MpvPlayer, "resize")

219
python_pkg/wsg_grabber/tests/test_player_errors.py
View File

@ -1,219 +0,0 @@
"""Tests for how the player reports IPC failures."""
from __future__ import annotations
import json
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import logs, player
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
def _socket_mock(**kwargs: object) -> MagicMock:
"""Build a socket stand-in whose reader thread terminates immediately.
An unconfigured MagicMock returns a MagicMock from recv(), which is truthy
and instant, so the drain loop spins as fast as the CPU allows -- and
MagicMock records every call, so memory grows without bound. That once
reached 21 GB and took the machine down, hence this helper: every socket
mock yields EOF unless a test deliberately says otherwise.
Args:
**kwargs: Attributes to set on the mock.
Returns:
MagicMock: Socket double.
"""
sock = MagicMock()
sock.recv.return_value = b""
for name, value in kwargs.items():
setattr(sock, name, value)
return sock
@pytest.fixture(autouse=True)
def _safe_sockets() -> Iterator[None]:
"""Stop a stray socket mock from turning the drain loop into a busy loop.
``MpvPlayer`` runs a reader thread that calls ``recv`` until EOF. An
unconfigured MagicMock returns a MagicMock -- truthy, instant, and it
records every call -- so the loop spins and memory grows without bound.
That once reached 21 GB and OOM-killed the machine. Defaulting the module's
socket factory to something that reports EOF means a test has to opt into
the hazard deliberately.
Yields:
None: Control returns to the test.
"""
with patch(
"python_pkg.wsg_grabber.player.socket.socket",
side_effect=lambda *_a, **_k: _socket_mock(),
):
yield
@pytest.fixture
def started(tmp_path: Path) -> Iterator[tuple[player.MpvPlayer, MagicMock, MagicMock]]:
"""Construct an MpvPlayer with its process and socket mocked out.
Args:
tmp_path: Per-test temporary directory.
Yields:
tuple: The player, its process mock and its socket mock.
"""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
def create_socket(_seconds: float) -> None:
"""Stand in for mpv finally creating its control socket."""
ipc.write_bytes(b"")
# __init__ unlinks any stale socket first, so it has to appear *during*
# the wait rather than being pre-created.
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch("python_pkg.wsg_grabber.player.time.sleep", side_effect=create_socket),
):
yield player.MpvPlayer(1234, ipc), process, sock
def test_a_failed_send_is_logged_not_swallowed(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
"""A dead socket must not make a frozen reviewer look like a working one."""
mpv, _process, sock = started
sock.sendall.side_effect = BrokenPipeError("gone")
log = logs.start("debug")
try:
mpv.play(tmp_path / "clip.webm")
finally:
logs.stop()
recorded = [
json.loads(line)
for line in log.read_text(encoding="utf-8").splitlines()
if line
]
assert any(r["event"] == "player.send_failed" for r in recorded)
assert any(r["level"] == "error" for r in recorded)
def test_the_socket_is_drained_so_mpv_keeps_accepting_commands(
tmp_path: Path,
) -> None:
"""The regression that made the reviewer freeze on one video.
mpv pushes events down the same socket; a client that never reads lets the
buffer fill, at which point mpv stops reading commands while still looking
perfectly alive.
"""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
seen: list[int] = []
def recv(size: int) -> bytes:
seen.append(size)
return b'{"event":"property-change"}\n' if len(seen) < 3 else b""
sock.recv.side_effect = recv
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
):
mpv = player.MpvPlayer(1, ipc)
mpv._reader.join(timeout=5)
mpv.close()
assert seen, "the reader never read from the socket"
assert not mpv._reader.is_alive()
def test_the_reader_stops_on_a_socket_error(tmp_path: Path) -> None:
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
sock.recv.side_effect = ConnectionResetError("gone")
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
):
mpv = player.MpvPlayer(1, ipc)
mpv._reader.join(timeout=5)
assert not mpv._reader.is_alive()
mpv.close()
def test_the_reader_keeps_going_through_read_timeouts(tmp_path: Path) -> None:
"""A quiet socket must not be mistaken for a closed one."""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
calls = {"n": 0}
def recv(_size: int) -> bytes:
calls["n"] += 1
if calls["n"] < 3:
raise TimeoutError
return b""
sock.recv.side_effect = recv
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
):
mpv = player.MpvPlayer(1, ipc)
mpv._reader.join(timeout=5)
mpv.close()
assert calls["n"] >= 3
def test_mpv_errors_are_logged_rather_than_discarded(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""A file mpv cannot decode should be visible, not just look stuck."""
mpv, _process, _sock = started
log = logs.start("debug")
try:
mpv._note_errors(
b'{"error":"success"}\n'
b"\n"
b"garbage\n"
b'{"request_id":1,"error":"loadfile failed"}\n',
)
finally:
logs.stop()
recorded = [
json.loads(line)
for line in log.read_text(encoding="utf-8").splitlines()
if line
]
failures = [r for r in recorded if r["event"] == "player.reply_error"]
assert len(failures) == 1
assert failures[0]["detail"] == "loadfile failed"

154
python_pkg/wsg_grabber/tests/test_player_probe.py
View File

@ -1,154 +0,0 @@
"""Tests for probing what mpv is doing over its IPC socket."""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import player
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
def _socket_mock(**kwargs: object) -> MagicMock:
"""Build a socket stand-in whose reader thread terminates immediately.
An unconfigured MagicMock returns a MagicMock from recv(), which is truthy
and instant, so the drain loop spins as fast as the CPU allows -- and
MagicMock records every call, so memory grows without bound. That once
reached 21 GB and took the machine down, hence this helper: every socket
mock yields EOF unless a test deliberately says otherwise.
Args:
**kwargs: Attributes to set on the mock.
Returns:
MagicMock: Socket double.
"""
sock = MagicMock()
sock.recv.return_value = b""
for name, value in kwargs.items():
setattr(sock, name, value)
return sock
@pytest.fixture(autouse=True)
def _safe_sockets() -> Iterator[None]:
"""Stop a stray socket mock from turning the drain loop into a busy loop.
``MpvPlayer`` runs a reader thread that calls ``recv`` until EOF. An
unconfigured MagicMock returns a MagicMock -- truthy, instant, and it
records every call -- so the loop spins and memory grows without bound.
That once reached 21 GB and OOM-killed the machine. Defaulting the module's
socket factory to something that reports EOF means a test has to opt into
the hazard deliberately.
Yields:
None: Control returns to the test.
"""
with patch(
"python_pkg.wsg_grabber.player.socket.socket",
side_effect=lambda *_a, **_k: _socket_mock(),
):
yield
@pytest.fixture
def started(tmp_path: Path) -> Iterator[tuple[player.MpvPlayer, MagicMock, MagicMock]]:
"""Construct an MpvPlayer with its process and socket mocked out.
Args:
tmp_path: Per-test temporary directory.
Yields:
tuple: The player, its process mock and its socket mock.
"""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
def create_socket(_seconds: float) -> None:
"""Stand in for mpv finally creating its control socket."""
ipc.write_bytes(b"")
# __init__ unlinks any stale socket first, so it has to appear *during*
# the wait rather than being pre-created.
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch("python_pkg.wsg_grabber.player.time.sleep", side_effect=create_socket),
):
yield player.MpvPlayer(1234, ipc), process, sock
def test_probe_reports_what_mpv_is_doing(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
tmp_path: Path,
) -> None:
"""The answer that settles a 'it froze on one video' report."""
mpv, _process, _sock = started
reply = _socket_mock()
reply.recv.return_value = (
b'{"request_id": 9001, "data": "value", "error": "success"}\n'
)
reply.__enter__ = lambda self: self
reply.__exit__ = lambda *_a: False
with patch("python_pkg.wsg_grabber.player.socket.socket", return_value=reply):
result = mpv.probe()
assert set(result) == {
"path",
"pause",
"core-idle",
"eof-reached",
"playback-time",
"width",
"height",
}
assert result["path"] == "value"
def test_probe_records_a_failure_rather_than_raising(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, _process, _sock = started
with patch(
"python_pkg.wsg_grabber.player.socket.socket",
side_effect=ConnectionRefusedError("gone"),
):
result = mpv.probe()
assert all(str(value).startswith("error:") for value in result.values())
def test_probe_survives_a_socket_that_closes_early(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
mpv, _process, _sock = started
reply = _socket_mock()
reply.recv.return_value = b""
reply.__enter__ = lambda self: self
reply.__exit__ = lambda *_a: False
with patch("python_pkg.wsg_grabber.player.socket.socket", return_value=reply):
result = mpv.probe()
assert result["path"] == "error: socket closed"
def test_probe_ignores_unrelated_traffic_on_the_socket(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""mpv pushes async events down the socket; they must not be mistaken
for the reply, and a partial line must not crash the reader."""
mpv, _process, _sock = started
reply = _socket_mock()
reply.recv.side_effect = [
b'{"event": "property-change"}\nnot json\n',
b'{"request_id": 9001, "data": 42}\n',
] * 10
reply.__enter__ = lambda self: self
reply.__exit__ = lambda *_a: False
with patch("python_pkg.wsg_grabber.player.socket.socket", return_value=reply):
assert mpv.probe()["path"] == 42

130
python_pkg/wsg_grabber/tests/test_player_socket.py
View File

@ -1,130 +0,0 @@
"""Tests for the socket stand-ins and connection teardown."""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import player
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
def _socket_mock(**kwargs: object) -> MagicMock:
"""Build a socket stand-in whose reader thread terminates immediately.
An unconfigured MagicMock returns a MagicMock from recv(), which is truthy
and instant, so the drain loop spins as fast as the CPU allows -- and
MagicMock records every call, so memory grows without bound. That once
reached 21 GB and took the machine down, hence this helper: every socket
mock yields EOF unless a test deliberately says otherwise.
Args:
**kwargs: Attributes to set on the mock.
Returns:
MagicMock: Socket double.
"""
sock = MagicMock()
sock.recv.return_value = b""
for name, value in kwargs.items():
setattr(sock, name, value)
return sock
@pytest.fixture(autouse=True)
def _safe_sockets() -> Iterator[None]:
"""Stop a stray socket mock from turning the drain loop into a busy loop.
``MpvPlayer`` runs a reader thread that calls ``recv`` until EOF. An
unconfigured MagicMock returns a MagicMock -- truthy, instant, and it
records every call -- so the loop spins and memory grows without bound.
That once reached 21 GB and OOM-killed the machine. Defaulting the module's
socket factory to something that reports EOF means a test has to opt into
the hazard deliberately.
Yields:
None: Control returns to the test.
"""
with patch(
"python_pkg.wsg_grabber.player.socket.socket",
side_effect=lambda *_a, **_k: _socket_mock(),
):
yield
@pytest.fixture
def started(tmp_path: Path) -> Iterator[tuple[player.MpvPlayer, MagicMock, MagicMock]]:
"""Construct an MpvPlayer with its process and socket mocked out.
Args:
tmp_path: Per-test temporary directory.
Yields:
tuple: The player, its process mock and its socket mock.
"""
ipc = tmp_path / "mpv.sock"
process = MagicMock()
process.poll.return_value = None
sock = _socket_mock()
def create_socket(_seconds: float) -> None:
"""Stand in for mpv finally creating its control socket."""
ipc.write_bytes(b"")
# __init__ unlinks any stale socket first, so it has to appear *during*
# the wait rather than being pre-created.
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch("python_pkg.wsg_grabber.player.time.sleep", side_effect=create_socket),
):
yield player.MpvPlayer(1234, ipc), process, sock
def test_socket_stand_ins_always_report_eof() -> None:
"""The contract that keeps the reader thread from becoming a busy loop.
A mock whose recv() returns a MagicMock is truthy and instant, so the drain
loop spins and the mock's recorded-call list grows without bound -- 21 GB
and an OOM kill, in the incident this guards against.
"""
assert _socket_mock().recv(65536) == b""
def test_the_reader_returns_promptly_once_asked_to_stop(
started: tuple[player.MpvPlayer, MagicMock, MagicMock],
) -> None:
"""Shutdown must not wait on another read."""
mpv, _process, sock = started
sock.recv.reset_mock()
mpv._stopping.set()
mpv._drain()
sock.recv.assert_not_called()
def test_a_refused_connection_does_not_leak_the_socket(tmp_path: Path) -> None:
"""A stale socket file makes connect() fail; nothing may be left open."""
ipc = tmp_path / "stale.sock"
process = MagicMock()
sock = _socket_mock()
sock.connect.side_effect = ConnectionRefusedError("stale")
with (
patch("python_pkg.wsg_grabber.player.subprocess.Popen", return_value=process),
patch("python_pkg.wsg_grabber.player.socket.socket", return_value=sock),
patch(
"python_pkg.wsg_grabber.player.time.sleep",
side_effect=lambda _s: ipc.write_bytes(b""),
),
pytest.raises(ConnectionRefusedError),
):
player.MpvPlayer(1, ipc)
sock.close.assert_called_once()
process.kill.assert_called_once()
process.wait.assert_called_once() # reaped, not left a zombie

127
python_pkg/wsg_grabber/tests/test_prune.py
View File

@ -1,127 +0,0 @@
"""Tests for the trash cap."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from python_pkg.wsg_grabber import db, files, paths, store, store_verdicts
from python_pkg.wsg_grabber._prune import prune_trash
from python_pkg.wsg_grabber.models import RemoteFile
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Iterator
import sqlite3
@pytest.fixture
def conn() -> Iterator[sqlite3.Connection]:
"""Open a sandboxed index with the on-disk layout in place.
Yields:
sqlite3.Connection: Closed afterwards.
"""
paths.ensure_dirs()
opened = db.open_index(paths.db_path())
try:
yield opened
finally:
opened.close()
def _passed(conn: sqlite3.Connection, index: int, reviewed_at: str) -> str:
"""Record one pass whose file sits in trash/.
Args:
conn: Open connection.
index: Distinguishes files within a test.
reviewed_at: Verdict timestamp, so ordering is explicit.
Returns:
str: Name of the trashed file.
"""
md5 = f"md5-{index:04d}"
name = f"{index:04d}.webm"
store.record_files(
conn,
[
RemoteFile(
md5=md5,
tim=index,
ext=".webm",
orig_name=name,
fsize=1,
width=1,
height=1,
thread_no=1,
post_no=index,
),
],
)
store.mark_downloaded(conn, md5, name)
store_verdicts.record_verdict(conn, md5, FileState.PASSED, name)
conn.execute("UPDATE files SET reviewed_at = ? WHERE md5 = ?", (reviewed_at, md5))
(paths.trash_dir() / name).write_bytes(b"v")
return name
def test_keeps_only_the_newest_passes(conn: sqlite3.Connection) -> None:
names = [_passed(conn, i, f"2026-09-{i + 1:02d}") for i in range(5)]
assert prune_trash(conn, retain=2) == 3
assert sorted(p.name for p in paths.trash_dir().iterdir()) == names[3:]
assert store.state_of(conn, "md5-0000") is FileState.PURGED
assert store.state_of(conn, "md5-0004") is FileState.PASSED
assert store_verdicts.reviewed_count(conn) == 2
def test_orphans_without_an_index_row_are_deleted_too(
conn: sqlite3.Connection,
) -> None:
_passed(conn, 1, "2026-09-01")
(paths.trash_dir() / "orphan.webm").write_bytes(b"?")
(paths.trash_dir() / "subdir").mkdir()
assert prune_trash(conn, retain=5) == 1
assert not (paths.trash_dir() / "orphan.webm").exists()
assert (paths.trash_dir() / "0001.webm").exists()
assert (paths.trash_dir() / "subdir").is_dir()
def test_under_the_cap_is_a_no_op(conn: sqlite3.Connection) -> None:
_passed(conn, 1, "2026-09-01")
assert prune_trash(conn, retain=5) == 0
assert store.state_of(conn, "md5-0001") is FileState.PASSED
def test_missing_trash_dir_is_tolerated(conn: sqlite3.Connection) -> None:
paths.trash_dir().rmdir()
assert prune_trash(conn) == 0
def test_a_pass_whose_file_was_forgotten_is_purged(conn: sqlite3.Connection) -> None:
"""A row with no reviewed_name cannot be undone, so it is written off."""
_passed(conn, 1, "2026-09-01")
store_verdicts.forget_verdict(conn, "md5-0001")
prune_trash(conn, retain=5)
assert store.state_of(conn, "md5-0001") is FileState.PURGED
def test_purged_rows_are_still_known_to_the_catalog(conn: sqlite3.Connection) -> None:
_passed(conn, 1, "2026-09-01")
prune_trash(conn, retain=0)
assert "md5-0001" in store.known_md5s(conn)
def test_remove_reports_whether_anything_was_there() -> None:
target = paths.data_dir() / "x"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_bytes(b"")
assert files.remove(target) is True
assert files.remove(target) is False

112
python_pkg/wsg_grabber/tests/test_review.py
View File

@ -1,112 +0,0 @@
"""Tests for the reviewer's behaviour.
Because every decision lives here rather than in the Tk module, the whole UI is
specified by these tests and none of them need a display.
"""
from __future__ import annotations
from python_pkg.wsg_grabber import review
from python_pkg.wsg_grabber.models import (
Downloaded,
Emptiness,
FileReady,
Indexed,
ScanFinished,
Verdict,
WorkerFailed,
)
from python_pkg.wsg_grabber.tests.conftest import item as _item
def test_the_first_download_goes_straight_on_screen() -> None:
"""Watching starts as soon as one file lands, not when all of them do."""
state = review.initial_state(indexed=9000, ready=[])
assert state.current is None
state = review.on_new_files(state, [_item("a")])
assert state.current is not None
assert state.current.md5 == "a-md5"
assert state.pending == ()
def test_later_downloads_queue_behind_the_current_one() -> None:
state = review.initial_state(9000, [_item("a")])
state = review.on_new_files(state, [_item("b"), _item("c")])
assert state.current is not None
assert state.current.md5 == "a-md5"
assert [item.md5 for item in state.pending] == ["b-md5", "c-md5"]
def test_new_files_with_nothing_to_add_is_a_no_op() -> None:
state = review.initial_state(1, [_item("a")])
assert review.on_new_files(state, []) is state
def test_a_previous_run_resumes_with_its_unreviewed_videos() -> None:
state = review.initial_state(50, [_item("a"), _item("b")])
assert state.current is not None
assert state.current.md5 == "a-md5"
assert len(state.pending) == 1
def test_keep_counts_and_advances() -> None:
state = review.initial_state(2, [_item("a"), _item("b")])
state = review.on_verdict(state, Verdict.KEEP)
assert state.kept == 1
assert state.passed == 0
assert state.current is not None
assert state.current.md5 == "b-md5"
def test_pass_counts_and_advances() -> None:
state = review.initial_state(2, [_item("a"), _item("b")])
state = review.on_verdict(state, Verdict.SKIP)
assert state.passed == 1
assert state.kept == 0
assert state.current is not None
assert state.current.md5 == "b-md5"
def test_the_last_verdict_empties_the_screen() -> None:
state = review.initial_state(1, [_item("a")])
state = review.on_verdict(state, Verdict.KEEP)
assert state.current is None
assert review.emptiness(state) is Emptiness.WAITING
def test_a_verdict_with_nothing_showing_is_ignored() -> None:
state = review.initial_state(0, [])
assert review.on_verdict(state, Verdict.KEEP) is state
def test_a_vanished_file_is_skipped() -> None:
state = review.initial_state(2, [_item("a"), _item("b")])
state = review.on_missing_locally(state)
assert state.current is not None
assert state.current.md5 == "b-md5"
assert state.kept == 0
assert state.passed == 0
def test_missing_with_nothing_showing_is_ignored() -> None:
state = review.initial_state(0, [])
assert review.on_missing_locally(state) is state
def test_events_fold_into_the_state() -> None:
state = review.initial_state(0, [])
state = review.on_event(state, Indexed(known=9028))
assert state.indexed == 9028
state = review.on_event(state, FileReady(item=_item("a")))
assert state.current is not None
state = review.on_event(state, Downloaded(count=7))
assert state.downloaded == 7
state = review.on_event(state, ScanFinished())
assert state.scan_complete
state = review.on_event(state, WorkerFailed(message="no route to host"))
assert state.error == "no route to host"

102
python_pkg/wsg_grabber/tests/test_review_render.py
View File

@ -1,102 +0,0 @@
"""Tests for what the reviewer puts on screen.
Split from ``test_review.py`` to keep every file under the 250-line cap; these
cover the pure state-to-text functions that now live in ``review._render``.
"""
from __future__ import annotations
from pathlib import Path
from python_pkg.wsg_grabber import review
from python_pkg.wsg_grabber.models import (
Downloaded,
Emptiness,
ScanFinished,
Verdict,
WorkerFailed,
)
from python_pkg.wsg_grabber.tests.conftest import item as _item
def test_emptiness_distinguishes_waiting_from_exhausted() -> None:
state = review.initial_state(0, [])
assert review.emptiness(state) is Emptiness.WAITING
state = review.on_event(state, ScanFinished())
assert review.emptiness(state) is Emptiness.EXHAUSTED
state = review.on_new_files(state, [_item("a")])
assert review.emptiness(state) is Emptiness.NOT_EMPTY
def test_status_line_reports_progress() -> None:
state = review.initial_state(9028, [_item("a")])
state = review.on_event(state, Downloaded(count=12))
state = review.on_verdict(state, Verdict.KEEP)
line = review.status_line(state)
assert "kept 1" in line
assert "passed 0" in line
assert "downloaded 12/9028" in line
def test_status_line_omits_the_total_before_indexing() -> None:
state = review.initial_state(0, [_item("a")])
assert "/0" not in review.status_line(state)
def test_status_line_shows_the_two_empty_moods() -> None:
state = review.initial_state(0, [])
assert "waiting for downloads" in review.status_line(state)
assert "all caught up" in review.status_line(
review.on_event(state, ScanFinished()),
)
def test_a_worker_failure_takes_over_the_status_line() -> None:
state = review.on_event(review.initial_state(0, []), WorkerFailed(message="boom"))
assert review.status_line(state) == "downloader stopped: boom"
def test_filename_line_describes_the_current_video() -> None:
state = review.initial_state(1, [_item("a")])
line = review.filename_line(state)
assert "a.webm" in line
assert "480x360" in line
assert "2.0 MiB" in line
def test_filename_line_has_a_placeholder_when_idle() -> None:
assert review.filename_line(review.initial_state(0, [])) == "—"
def test_title_line_carries_the_tally() -> None:
state = review.initial_state(1, [_item("a")])
state = review.on_verdict(state, Verdict.KEEP)
assert review.title_line(state) == "/wsg/ — kept 1, passed 0"
def test_render_drives_the_window_completely() -> None:
state = review.initial_state(5, [_item("a")])
command = review.render(state)
assert command.play == Path("/incoming/a.webm")
assert not command.stop
assert command.verdicts_enabled
assert not command.quit_app
assert command.status == review.status_line(state)
assert command.title == review.title_line(state)
assert command.filename == review.filename_line(state)
def test_render_stops_playback_when_nothing_is_queued() -> None:
command = review.render(review.initial_state(0, []))
assert command.play is None
assert command.stop
assert not command.verdicts_enabled
def test_render_signals_quit_and_disables_the_buttons() -> None:
state = review.on_quit(review.initial_state(1, [_item("a")]))
command = review.render(state)
assert command.quit_app
assert not command.verdicts_enabled

79
python_pkg/wsg_grabber/tests/test_review_undo.py
View File

@ -1,79 +0,0 @@
"""Tests for taking a verdict back.
Split from ``test_review.py`` to keep every file under the 250-line cap; these
cover the group that now lives in ``review._undo``.
"""
from __future__ import annotations
from python_pkg.wsg_grabber import review
from python_pkg.wsg_grabber.models import Verdict
from python_pkg.wsg_grabber.tests.conftest import item as _item
from python_pkg.wsg_grabber.tests.conftest import reviewed as _reviewed
def test_undo_availability_comes_from_the_index_not_memory() -> None:
"""The count is seeded from the index, so undo survives a restart."""
resumed = review.initial_state(5, [_item("a")], undoable=3)
assert review.can_undo(resumed)
assert "u undoes (3)" in review.status_line(resumed)
assert not review.can_undo(review.initial_state(5, [_item("a")]))
def test_a_recorded_verdict_can_be_undone() -> None:
first = _item("a")
state = review.initial_state(2, [first, _item("b")])
state = review.on_verdict(state, Verdict.SKIP)
assert review.can_undo(state)
state = review.on_undo(state, _reviewed(first))
assert state.current is not None
assert state.current.md5 == "a-md5"
assert state.passed == 0
assert [item.md5 for item in state.pending] == ["b-md5"]
assert not review.can_undo(state)
def test_undo_restores_the_keep_counter() -> None:
first = _item("a")
state = review.initial_state(1, [first])
state = review.on_verdict(state, Verdict.KEEP)
assert state.kept == 1
assert review.on_undo(state, _reviewed(first, Verdict.KEEP)).kept == 0
def test_undo_works_when_the_queue_has_run_dry() -> None:
first = _item("a")
state = review.initial_state(1, [first])
state = review.on_verdict(state, Verdict.SKIP)
assert state.current is None
state = review.on_undo(state, _reviewed(first))
assert state.current is not None
assert state.current.md5 == "a-md5"
def test_forget_last_drops_one_undo_without_restoring() -> None:
first = _item("a")
state = review.initial_state(1, [first])
state = review.on_verdict(state, Verdict.SKIP)
dropped = review.forget_last(state)
assert not review.can_undo(dropped)
assert dropped.passed == 1
assert review.forget_last(dropped).undoable == 0
def test_the_status_line_advertises_undo_once_it_is_possible() -> None:
first = _item("a")
state = review.initial_state(1, [first])
assert "u undoes" not in review.status_line(state)
state = review.on_verdict(state, Verdict.SKIP)
assert "u undoes (1)" in review.status_line(state)
def test_render_enables_undo_only_when_something_can_be_undone() -> None:
first = _item("a")
state = review.initial_state(1, [first])
assert not review.render(state).undo_enabled
state = review.on_verdict(state, Verdict.KEEP)
assert review.render(state).undo_enabled
assert not review.render(review.on_quit(state)).undo_enabled

103
python_pkg/wsg_grabber/tests/test_scanner.py
View File

@ -1,103 +0,0 @@
"""Tests for the pure scheduling and retry helpers."""
from __future__ import annotations
import pytest
from python_pkg.wsg_grabber import scanner
from python_pkg.wsg_grabber.constants import (
BACKOFF_BASE_S,
BACKOFF_CAP_S,
MIN_REQUEST_INTERVAL_S,
)
from python_pkg.wsg_grabber.models import ResumePlan, TaskKind, ThreadRef
def test_downloads_are_preferred_so_a_video_appears_early() -> None:
"""The user watches while the rest load; fetching must outrank scanning."""
assert scanner.choose_task(5, 10, catalog_fresh=True) is TaskKind.DOWNLOAD
assert scanner.choose_task(1, 0, catalog_fresh=False) is TaskKind.DOWNLOAD
def test_scanning_happens_when_nothing_is_queued() -> None:
assert scanner.choose_task(0, 3, catalog_fresh=True) is TaskKind.SCAN
assert scanner.choose_task(0, 0, catalog_fresh=False) is TaskKind.SCAN
def test_idle_only_when_the_board_is_exhausted() -> None:
assert scanner.choose_task(0, 0, catalog_fresh=True) is TaskKind.IDLE
def test_stale_threads_picks_unseen_and_bumped_threads() -> None:
known = {1: 100, 2: 200}
live = [ThreadRef(1, 100), ThreadRef(2, 250), ThreadRef(3, 10)]
assert [ref.thread_no for ref in scanner.stale_threads(known, live)] == [2, 3]
def test_stale_threads_is_empty_on_an_unchanged_board() -> None:
known = {1: 100}
assert scanner.stale_threads(known, [ThreadRef(1, 100)]) == []
def test_next_delay_enforces_one_request_per_second() -> None:
assert scanner.next_delay(0.0, 0.0) == pytest.approx(MIN_REQUEST_INTERVAL_S)
assert scanner.next_delay(10.0, 10.5) == pytest.approx(0.5)
def test_next_delay_never_goes_negative() -> None:
assert scanner.next_delay(10.0, 99.0) == 0.0
def test_backoff_doubles_and_then_caps() -> None:
assert scanner.backoff_seconds(1) == pytest.approx(BACKOFF_BASE_S)
assert scanner.backoff_seconds(2) == pytest.approx(BACKOFF_BASE_S * 2)
assert scanner.backoff_seconds(3) == pytest.approx(BACKOFF_BASE_S * 4)
assert scanner.backoff_seconds(99) == pytest.approx(BACKOFF_CAP_S)
def test_backoff_treats_attempt_zero_as_the_first_failure() -> None:
assert scanner.backoff_seconds(0) == pytest.approx(BACKOFF_BASE_S)
def test_an_explicit_retry_after_wins_but_is_still_capped() -> None:
assert scanner.backoff_seconds(1, retry_after=7.0) == pytest.approx(7.0)
assert scanner.backoff_seconds(1, retry_after=9999.0) == pytest.approx(
BACKOFF_CAP_S,
)
def test_a_zero_retry_after_falls_back_to_exponential_backoff() -> None:
assert scanner.backoff_seconds(2, retry_after=0.0) == pytest.approx(
BACKOFF_BASE_S * 2,
)
def test_parse_retry_after_handles_the_forms_we_accept() -> None:
assert scanner.parse_retry_after("30") == pytest.approx(30.0)
assert scanner.parse_retry_after(" 12 ") == pytest.approx(12.0)
assert scanner.parse_retry_after(None) is None
assert scanner.parse_retry_after("Wed, 21 Oct 2026 07:28:00 GMT") is None
assert scanner.parse_retry_after("-5") is None
def test_resume_plan_starts_from_scratch_when_nothing_is_on_disk() -> None:
plan = scanner.resume_plan(0, 1000)
assert plan == ResumePlan(offset=0, discard=False)
def test_resume_plan_continues_a_genuine_partial() -> None:
plan = scanner.resume_plan(400, 1000)
assert plan.offset == 400
assert not plan.discard
def test_resume_plan_bins_an_oversized_leftover() -> None:
"""A .part at or beyond the expected size is corrupt, not resumable."""
assert scanner.resume_plan(1000, 1000).discard
assert scanner.resume_plan(1500, 1000).discard
def test_resume_plan_with_unknown_expected_size_still_resumes() -> None:
plan = scanner.resume_plan(400, 0)
assert plan.offset == 400
assert not plan.discard

95
python_pkg/wsg_grabber/tests/test_states.py
View File

@ -1,95 +0,0 @@
"""Tests for the file lifecycle transition function."""
from __future__ import annotations
import pytest
from python_pkg.wsg_grabber.constants import MAX_ATTEMPTS
from python_pkg.wsg_grabber.states import (
CLAIMABLE,
TERMINAL,
FileEvent,
FileState,
is_terminal,
next_state,
)
@pytest.mark.parametrize("state", sorted(CLAIMABLE))
def test_claiming_starts_a_download(state: FileState) -> None:
assert next_state(state, FileEvent.CLAIMED, 0) is FileState.DOWNLOADING
@pytest.mark.parametrize("state", sorted(CLAIMABLE))
def test_upstream_deletion_before_download_is_terminal(state: FileState) -> None:
assert next_state(state, FileEvent.DELETED_UPSTREAM, 0) is FileState.GONE
def test_completed_download_becomes_reviewable() -> None:
assert next_state(FileState.DOWNLOADING, FileEvent.COMPLETED, 1) is FileState.READY
def test_missing_upstream_file_is_never_retried() -> None:
assert next_state(FileState.DOWNLOADING, FileEvent.NOT_FOUND, 1) is FileState.GONE
def test_transient_error_retries_until_the_attempt_budget_runs_out() -> None:
assert (
next_state(FileState.DOWNLOADING, FileEvent.TRANSIENT_ERROR, 1)
is FileState.FAILED
)
assert (
next_state(FileState.DOWNLOADING, FileEvent.TRANSIENT_ERROR, MAX_ATTEMPTS)
is FileState.GONE
)
def test_checksum_mismatch_retries_until_the_attempt_budget_runs_out() -> None:
assert (
next_state(FileState.DOWNLOADING, FileEvent.CHECKSUM_MISMATCH, 1)
is FileState.CORRUPT
)
assert (
next_state(FileState.DOWNLOADING, FileEvent.CHECKSUM_MISMATCH, MAX_ATTEMPTS)
is FileState.GONE
)
def test_verdicts_are_terminal() -> None:
assert next_state(FileState.READY, FileEvent.KEPT, 1) is FileState.KEPT
assert next_state(FileState.READY, FileEvent.PASSED, 1) is FileState.PASSED
def test_a_vanished_local_file_is_written_off() -> None:
assert next_state(FileState.READY, FileEvent.MISSING_LOCALLY, 1) is FileState.GONE
def test_ready_ignores_upstream_deletion() -> None:
"""The bytes are already ours; the post being deleted is irrelevant."""
with pytest.raises(ValueError, match="ready"):
next_state(FileState.READY, FileEvent.DELETED_UPSTREAM, 1)
def test_illegal_transition_out_of_downloading_raises() -> None:
with pytest.raises(ValueError, match="downloading"):
next_state(FileState.DOWNLOADING, FileEvent.KEPT, 1)
def test_illegal_transition_from_a_terminal_state_raises() -> None:
with pytest.raises(ValueError, match="illegal transition"):
next_state(FileState.GONE, FileEvent.CLAIMED, 0)
def test_terminal_states_are_exactly_the_four_end_points() -> None:
assert {
FileState.KEPT,
FileState.PASSED,
FileState.GONE,
FileState.PURGED,
} == TERMINAL
for state in TERMINAL:
assert is_terminal(state)
for state in CLAIMABLE:
assert not is_terminal(state)
assert not is_terminal(FileState.READY)
assert not is_terminal(FileState.DOWNLOADING)

235
python_pkg/wsg_grabber/tests/test_store.py
View File

@ -1,235 +0,0 @@
"""Tests for the typed index operations."""
from __future__ import annotations
from typing import TYPE_CHECKING
from unittest.mock import MagicMock
import pytest
from python_pkg.wsg_grabber import db, store, store_threads, store_verdicts
from python_pkg.wsg_grabber.models import RemoteFile, ThreadRef
from python_pkg.wsg_grabber.states import CLAIMABLE, FileState
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
import sqlite3
_EXPECTED_CLAIMABLE_STATES = 3
@pytest.fixture
def conn(tmp_path: Path) -> Iterator[sqlite3.Connection]:
"""Open a migrated index for one test and close it afterwards.
Closing matters: pytest runs with ``filterwarnings = ["error", ...]`` and a
leaked connection surfaces as a ResourceWarning, i.e. a test failure.
Args:
tmp_path: Per-test temporary directory.
Yields:
sqlite3.Connection: Ready connection.
"""
connection = db.open_index(tmp_path / "index.db")
try:
yield connection
finally:
connection.close()
def _file(md5: str = "AAAA", tim: int = 111, thread_no: int = 9) -> RemoteFile:
"""Build a RemoteFile with sensible defaults.
Args:
md5: Identity.
tim: CDN filename stem.
thread_no: Owning thread.
Returns:
RemoteFile: Test fixture value.
"""
return RemoteFile(
md5=md5,
tim=tim,
ext=".webm",
orig_name="clip",
fsize=1234,
width=480,
height=360,
thread_no=thread_no,
post_no=tim,
)
def test_claimable_placeholder_count_matches_the_state_set() -> None:
"""The SQL spells out one '?' per claimable state; keep them in step."""
assert len(CLAIMABLE) == _EXPECTED_CLAIMABLE_STATES
assert len(store.claimable_values()) == _EXPECTED_CLAIMABLE_STATES
def test_local_name_ignores_the_poster_supplied_filename() -> None:
name = store.local_name("ab/cd+ef==", 555, ".webm")
assert name.startswith("555-")
assert name.endswith(".webm")
assert "/" not in name
assert "+" not in name
def test_record_files_counts_only_new_rows(conn: sqlite3.Connection) -> None:
assert store.record_files(conn, [_file("A"), _file("B", tim=2)]) == 2
assert store.record_files(conn, [_file("B", tim=2), _file("C", tim=3)]) == 1
assert store.known_md5s(conn) == {"A", "B", "C"}
def test_record_files_with_nothing_to_do(conn: sqlite3.Connection) -> None:
assert store.record_files(conn, []) == 0
def test_a_repost_of_identical_bytes_never_becomes_a_second_row(
conn: sqlite3.Connection,
) -> None:
"""Same md5 in another thread must not cost a second download."""
store.record_files(conn, [_file("SAME", tim=1, thread_no=100)])
added = store.record_files(conn, [_file("SAME", tim=999, thread_no=200)])
assert added == 0
assert store.pending_downloads(conn) == 1
def test_claim_next_takes_the_oldest_and_marks_it_in_flight(
conn: sqlite3.Connection,
) -> None:
store.record_files(conn, [_file("A", tim=1)])
claimed = store.claim_next(conn)
assert claimed is not None
assert claimed.md5 == "A"
assert store.attempts_for(conn, "A") == 1
assert store.counts(conn)[FileState.DOWNLOADING.value] == 1
def test_claim_next_returns_none_when_idle(conn: sqlite3.Connection) -> None:
assert store.claim_next(conn) is None
def test_claim_next_rolls_back_on_failure() -> None:
"""A mid-transaction failure must not leave the claim half-applied.
``sqlite3.Connection.execute`` is read-only so it cannot be patched on a
real connection; a mock stands in purely to inject the failure.
"""
fake = MagicMock()
# BEGIN IMMEDIATE, then the SELECT blows up, then ROLLBACK must still run.
fake.execute.side_effect = [MagicMock(), RuntimeError("simulated"), MagicMock()]
with pytest.raises(RuntimeError, match="simulated"):
store.claim_next(fake)
executed = [call.args[0] for call in fake.execute.call_args_list]
assert executed[0] == "BEGIN IMMEDIATE"
assert executed[-1] == "ROLLBACK"
def test_attempts_for_unknown_file_is_zero(conn: sqlite3.Connection) -> None:
assert store.attempts_for(conn, "nope") == 0
def test_set_state_records_the_error(conn: sqlite3.Connection) -> None:
store.record_files(conn, [_file("A")])
store.set_state(conn, "A", FileState.GONE, error="404")
row = conn.execute("SELECT state, last_error FROM files").fetchone()
assert row["state"] == FileState.GONE.value
assert row["last_error"] == "404"
def test_mark_downloaded_makes_the_file_reviewable(
conn: sqlite3.Connection,
tmp_path: Path,
) -> None:
store.record_files(conn, [_file("A", tim=42)])
store.mark_downloaded(conn, "A", "42-x.webm")
items = store.ready_items(conn, tmp_path)
assert len(items) == 1
assert items[0].md5 == "A"
assert items[0].path == tmp_path / "42-x.webm"
assert items[0].orig_name == "clip.webm"
def test_ready_items_is_empty_before_any_download(
conn: sqlite3.Connection,
tmp_path: Path,
) -> None:
store.record_files(conn, [_file("A")])
assert store.ready_items(conn, tmp_path) == []
def test_record_progress_persists_partial_bytes(conn: sqlite3.Connection) -> None:
store.record_files(conn, [_file("A")])
store.record_progress(conn, "A", 4096)
assert conn.execute("SELECT bytes_done FROM files").fetchone()[0] == 4096
def test_record_verdict_is_terminal_and_stamps_the_time(
conn: sqlite3.Connection,
tmp_path: Path,
) -> None:
store.record_files(conn, [_file("A")])
store.mark_downloaded(conn, "A", "a.webm")
store_verdicts.record_verdict(conn, "A", FileState.PASSED)
assert store.ready_items(conn, tmp_path) == []
assert conn.execute("SELECT reviewed_at FROM files").fetchone()[0] is not None
assert store.pending_downloads(conn) == 0
def test_a_reviewed_file_is_still_remembered_so_it_never_returns(
conn: sqlite3.Connection,
) -> None:
store.record_files(conn, [_file("A")])
store_verdicts.record_verdict(conn, "A", FileState.PASSED)
assert "A" in store.known_md5s(conn)
assert store.record_files(conn, [_file("A")]) == 0
def test_reset_in_flight_recovers_an_interrupted_download(
conn: sqlite3.Connection,
) -> None:
store.record_files(conn, [_file("A")])
store.claim_next(conn)
assert store.reset_in_flight(conn) == 1
assert store.pending_downloads(conn) == 1
assert store.reset_in_flight(conn) == 0
def test_counts_reports_every_state_and_a_total(conn: sqlite3.Connection) -> None:
store.record_files(conn, [_file("A"), _file("B", tim=2)])
store.set_state(conn, "A", FileState.GONE)
result = store.counts(conn)
assert result["total"] == 2
assert result[FileState.GONE.value] == 1
assert result[FileState.NEW.value] == 1
def test_thread_bookkeeping_round_trip(conn: sqlite3.Connection) -> None:
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
store_threads.record_thread(conn, ThreadRef(7, 1000), stamp)
assert store_threads.known_threads(conn) == {7: 1000}
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
assert store_threads.http_last_modified(conn, 7) == stamp
store_threads.record_thread(conn, ThreadRef(7, 2000), None)
assert store_threads.known_threads(conn) == {7: 2000}
stamp = "Mon, 27 Jul 2026 08:00:00 GMT"
assert store_threads.http_last_modified(conn, 7) == stamp
def test_http_last_modified_absent_cases(conn: sqlite3.Connection) -> None:
assert store_threads.http_last_modified(conn, 404) is None
store_threads.record_thread(conn, ThreadRef(8, 1), None)
assert store_threads.http_last_modified(conn, 8) is None
def test_mark_thread_gone(conn: sqlite3.Connection) -> None:
store_threads.record_thread(conn, ThreadRef(7, 1000), None)
store_threads.mark_thread_gone(conn, 7)
row = conn.execute("SELECT status FROM threads WHERE thread_no = 7").fetchone()
assert row["status"] == "gone"

213
python_pkg/wsg_grabber/tests/test_ui.py
View File

@ -1,213 +0,0 @@
"""Tests for the Tk window.
``tkinter`` is replaced wholesale in the module's namespace, so every widget is
a mock and no real ``Tk()`` is ever constructed. That matters beyond speed: CI
has no display, and instantiating Tk there would fail outright.
There is deliberately little to test here, because the window contains no
decisions -- those all live in ``review`` and are covered by ``test_review``.
"""
from __future__ import annotations
import queue
from typing import TYPE_CHECKING, cast
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import review, ui
from python_pkg.wsg_grabber.constants import (
KEEP_KEYS,
PASS_KEYS,
QUIT_KEYS,
UNDO_KEYS,
)
from python_pkg.wsg_grabber.models import (
ReviewedItem,
ReviewItem,
Verdict,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.review import SessionState
@pytest.fixture
def toolkit() -> Iterator[MagicMock]:
"""Replace tkinter in the ui module's namespace.
Yields:
MagicMock: The stand-in toolkit.
"""
fake = MagicMock()
# a fresh mock per widget, else status and filename are the same object
fake.Frame.side_effect = lambda *_a, **_k: MagicMock()
fake.Label.side_effect = lambda *_a, **_k: MagicMock()
fake.Button.side_effect = lambda *_a, **_k: MagicMock()
fake.BOTH = "both"
fake.X = "x"
fake.LEFT = "left"
fake.RIGHT = "right"
fake.BOTTOM = "bottom"
with patch.object(ui, "tk", fake):
yield fake
def _mock(widget: object) -> MagicMock:
"""View a mocked widget as the MagicMock it really is.
The Widgets dataclass is typed with real Tk classes, so mypy will not let
the tests reach ``call_args`` without this.
Args:
widget: The stand-in widget.
Returns:
MagicMock: The same object, typed for assertions.
"""
return cast("MagicMock", widget)
def _item(tmp_path: Path, name: str = "a") -> ReviewItem:
"""Build a ReviewItem backed by a real file.
Args:
tmp_path: Per-test temporary directory.
name: Distinguishes items.
Returns:
ReviewItem: Fixture value.
"""
path = tmp_path / f"{name}.webm"
path.write_bytes(b"video")
return ReviewItem(
md5=f"{name}-md5",
path=path,
orig_name=f"{name}.webm",
fsize=5,
width=4,
height=3,
)
def _callbacks() -> tuple[ui.Callbacks, MagicMock]:
"""Build callbacks that record what the window asked for.
Returns:
tuple: The callbacks and the recorder behind them.
"""
recorder = MagicMock()
_undone: dict[str, ReviewedItem | None] = {"item": None}
def commit(state: SessionState, choice: Verdict) -> SessionState:
recorder.commit(choice)
if state.current is not None:
_undone["item"] = ReviewedItem(
item=state.current,
choice=choice,
reviewed_name=state.current.path.name,
)
return review.on_verdict(state, choice)
def missing(state: SessionState) -> SessionState:
recorder.missing()
return review.on_missing_locally(state)
def undo(state: SessionState) -> SessionState:
recorder.undo()
action = _undone["item"]
if action is None or not review.can_undo(state):
return state
return review.on_undo(state, action)
return (
ui.Callbacks(
commit=commit,
missing=missing,
undo=undo,
shutdown=recorder.shutdown,
),
recorder,
)
def _window(
state: SessionState,
events: queue.SimpleQueue[DownloadEvent] | None = None,
) -> tuple[ui.ReviewWindow, MagicMock]:
"""Build a window over a mocked toolkit.
Args:
state: Starting state.
events: Optional event queue.
Returns:
tuple: The window and the callback recorder.
"""
callbacks, recorder = _callbacks()
window = ui.ReviewWindow(state, events or queue.SimpleQueue(), callbacks)
return window, recorder
def test_the_window_has_a_video_area_and_a_control_bar(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
assert toolkit.Tk.called
assert toolkit.Button.call_count == 3 # keep, pass, undo
# two frames: the video area and the control bar below it
assert toolkit.Frame.call_count == 2
# the video area expands; the bar sits under it rather than over it, because
# mpv's --wid child window covers whatever shares its frame
assert _mock(window.widgets.video).pack.call_args.kwargs["expand"]
def test_every_shortcut_is_bound(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
calls = _mock(window.widgets.root).bind.call_args_list
bound = {call.args[0] for call in calls}
assert set(KEEP_KEYS) <= bound
assert set(PASS_KEYS) <= bound
assert set(QUIT_KEYS) <= bound
assert set(UNDO_KEYS) <= bound
def test_keep_and_pass_shortcuts_reach_the_callbacks(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
state = review.initial_state(2, [_item(tmp_path, "a"), _item(tmp_path, "b")])
window, recorder = _window(state)
window.attach_player(MagicMock())
bound = _mock(window.widgets.root).bind.call_args_list
handlers = {call.args[0]: call.args[1] for call in bound}
handlers[KEEP_KEYS[0]](object())
first: object = recorder.commit.call_args.args[0]
assert first is Verdict.KEEP
handlers[PASS_KEYS[0]](object())
second: object = recorder.commit.call_args.args[0]
assert second is Verdict.SKIP
def test_the_quit_shortcut_shuts_down(toolkit: MagicMock) -> None:
window, recorder = _window(review.initial_state(0, []))
bound = _mock(window.widgets.root).bind.call_args_list
handlers = {call.args[0]: call.args[1] for call in bound}
handlers[QUIT_KEYS[0]](object())
recorder.shutdown.assert_called_once()
_mock(window.widgets.root).destroy.assert_called_once()
def test_the_buttons_call_the_same_handlers(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
state = review.initial_state(1, [_item(tmp_path)])
window, recorder = _window(state)
window.attach_player(MagicMock())
window.on_keep()
assert recorder.commit.call_args.args[0] is Verdict.KEEP

204
python_pkg/wsg_grabber/tests/test_ui_idle.py
View File

@ -1,204 +0,0 @@
"""Tests for the review window when the queue runs dry."""
from __future__ import annotations
import queue
from typing import TYPE_CHECKING, cast
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import review, ui
from python_pkg.wsg_grabber.models import (
ReviewCommand,
ReviewedItem,
ReviewItem,
Verdict,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.review import SessionState
@pytest.fixture
def toolkit() -> Iterator[MagicMock]:
"""Replace tkinter in the ui module's namespace.
Yields:
MagicMock: The stand-in toolkit.
"""
fake = MagicMock()
# a fresh mock per widget, else status and filename are the same object
fake.Frame.side_effect = lambda *_a, **_k: MagicMock()
fake.Label.side_effect = lambda *_a, **_k: MagicMock()
fake.Button.side_effect = lambda *_a, **_k: MagicMock()
fake.BOTH = "both"
fake.X = "x"
fake.LEFT = "left"
fake.RIGHT = "right"
fake.BOTTOM = "bottom"
with patch.object(ui, "tk", fake):
yield fake
def _mock(widget: object) -> MagicMock:
"""View a mocked widget as the MagicMock it really is.
The Widgets dataclass is typed with real Tk classes, so mypy will not let
the tests reach ``call_args`` without this.
Args:
widget: The stand-in widget.
Returns:
MagicMock: The same object, typed for assertions.
"""
return cast("MagicMock", widget)
def _item(tmp_path: Path, name: str = "a") -> ReviewItem:
"""Build a ReviewItem backed by a real file.
Args:
tmp_path: Per-test temporary directory.
name: Distinguishes items.
Returns:
ReviewItem: Fixture value.
"""
path = tmp_path / f"{name}.webm"
path.write_bytes(b"video")
return ReviewItem(
md5=f"{name}-md5",
path=path,
orig_name=f"{name}.webm",
fsize=5,
width=4,
height=3,
)
def _callbacks() -> tuple[ui.Callbacks, MagicMock]:
"""Build callbacks that record what the window asked for.
Returns:
tuple: The callbacks and the recorder behind them.
"""
recorder = MagicMock()
_undone: dict[str, ReviewedItem | None] = {"item": None}
def commit(state: SessionState, choice: Verdict) -> SessionState:
recorder.commit(choice)
if state.current is not None:
_undone["item"] = ReviewedItem(
item=state.current,
choice=choice,
reviewed_name=state.current.path.name,
)
return review.on_verdict(state, choice)
def missing(state: SessionState) -> SessionState:
recorder.missing()
return review.on_missing_locally(state)
def undo(state: SessionState) -> SessionState:
recorder.undo()
action = _undone["item"]
if action is None or not review.can_undo(state):
return state
return review.on_undo(state, action)
return (
ui.Callbacks(
commit=commit,
missing=missing,
undo=undo,
shutdown=recorder.shutdown,
),
recorder,
)
def _window(
state: SessionState,
events: queue.SimpleQueue[DownloadEvent] | None = None,
) -> tuple[ui.ReviewWindow, MagicMock]:
"""Build a window over a mocked toolkit.
Args:
state: Starting state.
events: Optional event queue.
Returns:
tuple: The window and the callback recorder.
"""
callbacks, recorder = _callbacks()
window = ui.ReviewWindow(state, events or queue.SimpleQueue(), callbacks)
return window, recorder
def test_apply_stops_and_disables_when_idle(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
player = MagicMock()
window.attach_player(player)
window.apply(
ReviewCommand(
play=None,
stop=True,
status="s",
title="t",
filename="f",
verdicts_enabled=False,
undo_enabled=False,
quit_app=False,
),
)
# nothing was playing, so there is nothing to stop; the buttons still go dead
player.stop.assert_not_called()
_mock(window.widgets.keep).configure.assert_called_with(state="disabled")
def test_apply_quits_when_told_to(toolkit: MagicMock) -> None:
window, recorder = _window(review.initial_state(0, []))
window.apply(
ReviewCommand(
play=None,
stop=False,
status="s",
title="t",
filename="f",
verdicts_enabled=False,
undo_enabled=False,
quit_app=True,
),
)
recorder.shutdown.assert_called_once()
def test_playing_without_a_player_is_a_no_op(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
window, _ = _window(review.initial_state(1, [_item(tmp_path)]))
window.refresh()
def test_a_video_that_vanished_is_skipped(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
"""A file can disappear between download and review; don't hang on it."""
first = _item(tmp_path, "a")
state = review.initial_state(2, [first, _item(tmp_path, "b")])
window, recorder = _window(state)
first.path.unlink()
player = MagicMock()
window.attach_player(player)
recorder.missing.assert_called_once()
assert window.state.current is not None
assert window.state.current.md5 == "b-md5"

217
python_pkg/wsg_grabber/tests/test_ui_playback.py
View File

@ -1,217 +0,0 @@
"""Tests for the review window's playback and advance behaviour."""
from __future__ import annotations
import queue
from typing import TYPE_CHECKING, cast
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import review, ui
from python_pkg.wsg_grabber.models import (
ReviewCommand,
ReviewedItem,
ReviewItem,
Verdict,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.review import SessionState
@pytest.fixture
def toolkit() -> Iterator[MagicMock]:
"""Replace tkinter in the ui module's namespace.
Yields:
MagicMock: The stand-in toolkit.
"""
fake = MagicMock()
# a fresh mock per widget, else status and filename are the same object
fake.Frame.side_effect = lambda *_a, **_k: MagicMock()
fake.Label.side_effect = lambda *_a, **_k: MagicMock()
fake.Button.side_effect = lambda *_a, **_k: MagicMock()
fake.BOTH = "both"
fake.X = "x"
fake.LEFT = "left"
fake.RIGHT = "right"
fake.BOTTOM = "bottom"
with patch.object(ui, "tk", fake):
yield fake
def _mock(widget: object) -> MagicMock:
"""View a mocked widget as the MagicMock it really is.
The Widgets dataclass is typed with real Tk classes, so mypy will not let
the tests reach ``call_args`` without this.
Args:
widget: The stand-in widget.
Returns:
MagicMock: The same object, typed for assertions.
"""
return cast("MagicMock", widget)
def _item(tmp_path: Path, name: str = "a") -> ReviewItem:
"""Build a ReviewItem backed by a real file.
Args:
tmp_path: Per-test temporary directory.
name: Distinguishes items.
Returns:
ReviewItem: Fixture value.
"""
path = tmp_path / f"{name}.webm"
path.write_bytes(b"video")
return ReviewItem(
md5=f"{name}-md5",
path=path,
orig_name=f"{name}.webm",
fsize=5,
width=4,
height=3,
)
def _callbacks() -> tuple[ui.Callbacks, MagicMock]:
"""Build callbacks that record what the window asked for.
Returns:
tuple: The callbacks and the recorder behind them.
"""
recorder = MagicMock()
_undone: dict[str, ReviewedItem | None] = {"item": None}
def commit(state: SessionState, choice: Verdict) -> SessionState:
recorder.commit(choice)
if state.current is not None:
_undone["item"] = ReviewedItem(
item=state.current,
choice=choice,
reviewed_name=state.current.path.name,
)
return review.on_verdict(state, choice)
def missing(state: SessionState) -> SessionState:
recorder.missing()
return review.on_missing_locally(state)
def undo(state: SessionState) -> SessionState:
recorder.undo()
action = _undone["item"]
if action is None or not review.can_undo(state):
return state
return review.on_undo(state, action)
return (
ui.Callbacks(
commit=commit,
missing=missing,
undo=undo,
shutdown=recorder.shutdown,
),
recorder,
)
def _window(
state: SessionState,
events: queue.SimpleQueue[DownloadEvent] | None = None,
) -> tuple[ui.ReviewWindow, MagicMock]:
"""Build a window over a mocked toolkit.
Args:
state: Starting state.
events: Optional event queue.
Returns:
tuple: The window and the callback recorder.
"""
callbacks, recorder = _callbacks()
window = ui.ReviewWindow(state, events or queue.SimpleQueue(), callbacks)
return window, recorder
def test_apply_plays_and_labels(toolkit: MagicMock, tmp_path: Path) -> None:
window, _ = _window(review.initial_state(0, []))
player = MagicMock()
window.attach_player(player)
path = tmp_path / "x.webm"
path.write_bytes(b"v")
window.apply(
ReviewCommand(
play=path,
stop=False,
status="status text",
title="title text",
filename="file text",
verdicts_enabled=True,
undo_enabled=False,
quit_app=False,
),
)
player.play.assert_called_once_with(path)
_mock(window.widgets.status).configure.assert_called_with(text="status text")
_mock(window.widgets.filename).configure.assert_called_with(text="file text")
_mock(window.widgets.root).title.assert_called_with("title text")
_mock(window.widgets.keep).configure.assert_called_with(state="normal")
def test_the_same_video_is_only_loaded_once(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
"""refresh() runs every tick; re-sending loadfile restarts the video."""
window, _ = _window(review.initial_state(1, [_item(tmp_path)]))
player = MagicMock()
window.attach_player(player)
assert player.play.call_count == 1
for _ in range(10):
window.tick()
assert player.play.call_count == 1
def test_advancing_loads_the_next_video(toolkit: MagicMock, tmp_path: Path) -> None:
state = review.initial_state(2, [_item(tmp_path, "a"), _item(tmp_path, "b")])
window, _ = _window(state)
player = MagicMock()
window.attach_player(player)
window.on_keep()
assert player.play.call_count == 2
assert player.play.call_args.args[0].name == "b.webm"
def test_stopping_lets_the_same_video_load_again(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
item = _item(tmp_path)
window, _ = _window(review.initial_state(1, [item]))
player = MagicMock()
window.attach_player(player)
window.apply(
ReviewCommand(
play=None,
stop=True,
status="s",
title="t",
filename="f",
verdicts_enabled=False,
undo_enabled=False,
quit_app=False,
),
)
player.stop.assert_called_once()
window.refresh()
assert player.play.call_count == 2

243
python_pkg/wsg_grabber/tests/test_ui_tick.py
View File

@ -1,243 +0,0 @@
"""Tests for the window's event-drain tick and shutdown."""
from __future__ import annotations
import queue
from typing import TYPE_CHECKING, cast
from unittest.mock import MagicMock, patch
import pytest
from python_pkg.wsg_grabber import review, ui
from python_pkg.wsg_grabber.constants import (
UNDO_KEYS,
)
from python_pkg.wsg_grabber.models import (
ReviewedItem,
ReviewItem,
Verdict,
)
if TYPE_CHECKING:
from collections.abc import Iterator
from pathlib import Path
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.review import SessionState
@pytest.fixture
def toolkit() -> Iterator[MagicMock]:
"""Replace tkinter in the ui module's namespace.
Yields:
MagicMock: The stand-in toolkit.
"""
fake = MagicMock()
# a fresh mock per widget, else status and filename are the same object
fake.Frame.side_effect = lambda *_a, **_k: MagicMock()
fake.Label.side_effect = lambda *_a, **_k: MagicMock()
fake.Button.side_effect = lambda *_a, **_k: MagicMock()
fake.BOTH = "both"
fake.X = "x"
fake.LEFT = "left"
fake.RIGHT = "right"
fake.BOTTOM = "bottom"
with patch.object(ui, "tk", fake):
yield fake
def _mock(widget: object) -> MagicMock:
"""View a mocked widget as the MagicMock it really is.
The Widgets dataclass is typed with real Tk classes, so mypy will not let
the tests reach ``call_args`` without this.
Args:
widget: The stand-in widget.
Returns:
MagicMock: The same object, typed for assertions.
"""
return cast("MagicMock", widget)
def _item(tmp_path: Path, name: str = "a") -> ReviewItem:
"""Build a ReviewItem backed by a real file.
Args:
tmp_path: Per-test temporary directory.
name: Distinguishes items.
Returns:
ReviewItem: Fixture value.
"""
path = tmp_path / f"{name}.webm"
path.write_bytes(b"video")
return ReviewItem(
md5=f"{name}-md5",
path=path,
orig_name=f"{name}.webm",
fsize=5,
width=4,
height=3,
)
def _callbacks() -> tuple[ui.Callbacks, MagicMock]:
"""Build callbacks that record what the window asked for.
Returns:
tuple: The callbacks and the recorder behind them.
"""
recorder = MagicMock()
_undone: dict[str, ReviewedItem | None] = {"item": None}
def commit(state: SessionState, choice: Verdict) -> SessionState:
recorder.commit(choice)
if state.current is not None:
_undone["item"] = ReviewedItem(
item=state.current,
choice=choice,
reviewed_name=state.current.path.name,
)
return review.on_verdict(state, choice)
def missing(state: SessionState) -> SessionState:
recorder.missing()
return review.on_missing_locally(state)
def undo(state: SessionState) -> SessionState:
recorder.undo()
action = _undone["item"]
if action is None or not review.can_undo(state):
return state
return review.on_undo(state, action)
return (
ui.Callbacks(
commit=commit,
missing=missing,
undo=undo,
shutdown=recorder.shutdown,
),
recorder,
)
def _window(
state: SessionState,
events: queue.SimpleQueue[DownloadEvent] | None = None,
) -> tuple[ui.ReviewWindow, MagicMock]:
"""Build a window over a mocked toolkit.
Args:
state: Starting state.
events: Optional event queue.
Returns:
tuple: The window and the callback recorder.
"""
callbacks, recorder = _callbacks()
window = ui.ReviewWindow(state, events or queue.SimpleQueue(), callbacks)
return window, recorder
def test_tick_drains_events_and_reschedules(toolkit: MagicMock) -> None:
from python_pkg.wsg_grabber.models import Indexed
events: queue.SimpleQueue[DownloadEvent] = queue.SimpleQueue()
events.put(Indexed(known=42))
window, _ = _window(review.initial_state(0, []), events)
window.tick()
assert window.state.indexed == 42
_mock(window.widgets.root).after.assert_called_once()
assert _mock(window.widgets.root).after.call_args.args[1] == window.tick
def test_drain_on_an_empty_queue_is_harmless(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
window.drain()
assert window.state.indexed == 0
def test_shutdown_cancels_the_tick_and_closes_the_player(
toolkit: MagicMock,
) -> None:
"""A leaked after-callback or an orphan mpv would outlive the window."""
window, recorder = _window(review.initial_state(0, []))
player = MagicMock()
window.attach_player(player)
window.tick()
window.shutdown()
_mock(window.widgets.root).after_cancel.assert_called_once()
player.close.assert_called_once()
recorder.shutdown.assert_called_once()
_mock(window.widgets.root).destroy.assert_called_once()
def test_shutdown_without_a_pending_tick(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
window.shutdown()
_mock(window.widgets.root).after_cancel.assert_not_called()
def test_video_wid_realises_the_window_first(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
_mock(window.widgets.video).winfo_id.return_value = 4321
assert window.video_wid() == 4321
_mock(window.widgets.root).update.assert_called()
def test_run_starts_ticking_then_enters_the_loop(toolkit: MagicMock) -> None:
window, _ = _window(review.initial_state(0, []))
window.run()
_mock(window.widgets.root).mainloop.assert_called_once()
_mock(window.widgets.root).after.assert_called_once()
def test_undo_takes_back_the_last_verdict(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
"""The misclick case: passed something you meant to keep."""
state = review.initial_state(2, [_item(tmp_path, "a"), _item(tmp_path, "b")])
window, recorder = _window(state)
player = MagicMock()
window.attach_player(player)
window.on_pass()
assert window.state.current is not None
assert window.state.current.md5 == "b-md5"
assert window.state.passed == 1
window.on_undo()
recorder.undo.assert_called_once()
assert window.state.current is not None
assert window.state.current.md5 == "a-md5"
assert window.state.passed == 0
# and the video that was on screen is not lost
assert [item.md5 for item in window.state.pending] == ["b-md5"]
def test_the_undo_shortcut_is_wired(toolkit: MagicMock, tmp_path: Path) -> None:
state = review.initial_state(1, [_item(tmp_path)])
window, recorder = _window(state)
window.attach_player(MagicMock())
window.on_keep()
bound = _mock(window.widgets.root).bind.call_args_list
handlers = {call.args[0]: call.args[1] for call in bound}
handlers[UNDO_KEYS[0]](object())
recorder.undo.assert_called_once()
def test_the_undo_button_is_disabled_until_there_is_something_to_undo(
toolkit: MagicMock,
tmp_path: Path,
) -> None:
state = review.initial_state(1, [_item(tmp_path)])
window, _ = _window(state)
window.attach_player(MagicMock())
assert _mock(window.widgets.undo).configure.call_args.kwargs["state"] == "disabled"
window.on_keep()
assert _mock(window.widgets.undo).configure.call_args.kwargs["state"] == "normal"

126
python_pkg/wsg_grabber/tests/test_verdict.py
View File

@ -1,126 +0,0 @@
"""Tests for verdict planning and the one module that moves files."""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import files, verdict
from python_pkg.wsg_grabber.models import FileMove, ReviewItem, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from pathlib import Path
def _item(path: Path) -> ReviewItem:
"""Build a ReviewItem pointing at *path*.
Args:
path: Location of the video.
Returns:
ReviewItem: Fixture value.
"""
return ReviewItem(
md5="md5",
path=path,
orig_name="clip.webm",
fsize=10,
width=4,
height=3,
)
def test_verdicts_map_to_terminal_states() -> None:
assert verdict.target_state(Verdict.KEEP) is FileState.KEPT
assert verdict.target_state(Verdict.SKIP) is FileState.PASSED
def test_verdicts_map_to_directories(tmp_path: Path) -> None:
keep, trash = tmp_path / "keep", tmp_path / "trash"
assert verdict.target_dir(Verdict.KEEP, keep, trash) == keep
assert verdict.target_dir(Verdict.SKIP, keep, trash) == trash
def test_unique_destination_passes_through_when_free(tmp_path: Path) -> None:
assert verdict.unique_destination(tmp_path, "a.webm", set()) == tmp_path / "a.webm"
def test_unique_destination_suffixes_a_collision(tmp_path: Path) -> None:
result = verdict.unique_destination(tmp_path, "a.webm", {"a.webm"})
assert result == tmp_path / "a-2.webm"
def test_unique_destination_keeps_counting(tmp_path: Path) -> None:
taken = {"a.webm", "a-2.webm", "a-3.webm"}
assert (
verdict.unique_destination(tmp_path, "a.webm", taken) == tmp_path / "a-4.webm"
)
def test_unique_destination_handles_a_name_without_an_extension(
tmp_path: Path,
) -> None:
assert verdict.unique_destination(tmp_path, "a", {"a"}) == tmp_path / "a-2"
def test_plan_move_sends_a_keep_to_the_keep_directory(tmp_path: Path) -> None:
keep, trash = tmp_path / "keep", tmp_path / "trash"
move = verdict.plan_move(
_item(tmp_path / "in" / "x.webm"), Verdict.KEEP, keep, trash
)
assert move.dst == keep / "x.webm"
assert move.src == tmp_path / "in" / "x.webm"
assert move.md5 == "md5"
def test_plan_move_sends_a_pass_to_the_trash_directory(tmp_path: Path) -> None:
keep, trash = tmp_path / "keep", tmp_path / "trash"
move = verdict.plan_move(
_item(tmp_path / "in" / "x.webm"), Verdict.SKIP, keep, trash
)
assert move.dst == trash / "x.webm"
def test_apply_move_relocates_rather_than_deleting(tmp_path: Path) -> None:
"""A pass must never destroy the file; trash is a destination, not a bin."""
source = tmp_path / "in" / "x.webm"
source.parent.mkdir()
source.write_bytes(b"data")
move = FileMove(md5="md5", src=source, dst=tmp_path / "trash" / "x.webm")
assert files.apply_move(move)
assert not source.exists()
assert move.dst.read_bytes() == b"data"
def test_apply_move_reports_a_source_that_has_vanished(tmp_path: Path) -> None:
move = FileMove(md5="md5", src=tmp_path / "gone.webm", dst=tmp_path / "t.webm")
assert not files.apply_move(move)
def test_existing_names_lists_the_directory(tmp_path: Path) -> None:
# a subdirectory, because the isolation fixture puts a fake home in tmp_path
keep = tmp_path / "keep"
keep.mkdir()
(keep / "a.webm").write_bytes(b"")
(keep / "b.webm").write_bytes(b"")
assert files.existing_names(keep) == {"a.webm", "b.webm"}
def test_existing_names_of_a_missing_directory_is_empty(tmp_path: Path) -> None:
assert files.existing_names(tmp_path / "nope") == set()
def test_apply_move_refuses_to_clobber_the_destination(tmp_path: Path) -> None:
"""The structural backstop: no move may ever destroy an existing file."""
source = tmp_path / "in" / "x.webm"
source.parent.mkdir()
source.write_bytes(b"new")
victim = tmp_path / "keep" / "x.webm"
victim.parent.mkdir()
victim.write_bytes(b"ALREADY HERE")
assert not files.apply_move(FileMove(md5="m", src=source, dst=victim))
assert victim.read_bytes() == b"ALREADY HERE"
assert source.read_bytes() == b"new"

242
python_pkg/wsg_grabber/ui.py
View File

@ -1,242 +0,0 @@
"""The Tk window: video on top, keep/pass bar underneath.
The only module in this package that imports ``tkinter``. Almost every callback
hands off to :mod:`python_pkg.wsg_grabber.review`, gets a
:class:`ReviewCommand` back and applies it verbatim, so the behaviour is
specified and covered without a display ever being opened.
Two decisions do live here, both because a bug proved they had to: ``_play``
skips a reload when the file is already playing, and drops an item whose file
has vanished. Everything else is wiring.
The control bar sits *below* the video rather than over it: with ``--wid`` mpv
reparents its own child window onto the frame and covers it completely, so a
widget drawn on top would be hidden.
Nothing here tracks the window size. mpv resizes itself to fill the frame it was
given, and telling it otherwise via ``geometry`` leaves the video drawn at the
wrong size -- which is exactly what happened the first time this ran.
"""
from __future__ import annotations
from dataclasses import dataclass
import tkinter as tk
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber import logs
from python_pkg.wsg_grabber._display import DisplayMixin
from python_pkg.wsg_grabber.constants import (
CONTROL_BAR_HEIGHT_PX,
KEEP_KEYS,
PASS_KEYS,
QUIT_KEYS,
UNDO_KEYS,
WINDOW_HEIGHT_PX,
WINDOW_WIDTH_PX,
)
from python_pkg.wsg_grabber.models import Verdict
if TYPE_CHECKING:
from collections.abc import Callable
from pathlib import Path
import queue
from python_pkg.wsg_grabber.models import DownloadEvent
from python_pkg.wsg_grabber.player import Player
from python_pkg.wsg_grabber.review import SessionState
_BACKGROUND = "#1B1D21"
_ACCENT = "#B8862E"
_TEXT = "#E8E6E3"
@dataclass(frozen=True, slots=True)
class Callbacks:
"""What the window asks the rest of the application to do."""
commit: Callable[[SessionState, Verdict], SessionState]
missing: Callable[[SessionState], SessionState]
undo: Callable[[SessionState], SessionState]
shutdown: Callable[[], None]
@dataclass
class Widgets:
"""The widgets, grouped so the window stays within the attribute limit."""
root: tk.Tk
video: tk.Frame
status: tk.Label
filename: tk.Label
keep: tk.Button
skip: tk.Button
undo: tk.Button
class ReviewWindow(DisplayMixin):
"""Presents downloaded videos one at a time for a keep/pass decision.
The screen-updating and state-transition halves live in _display to keep
this module under the 250-line cap. Widget construction stays here: this
is the only module that imports tkinter, and the tests fake the toolkit
by patching ``ui.tk``.
"""
def __init__(
self,
state: SessionState,
events: queue.SimpleQueue[DownloadEvent],
callbacks: Callbacks,
) -> None:
"""Build the window.
The player is attached afterwards with :meth:`attach_player`, because
it needs the X11 id of a frame that must exist first.
Args:
state: Starting session state.
events: Queue the download worker publishes to.
callbacks: Hooks back into the application.
"""
self._state = state
self._events = events
self._callbacks = callbacks
self._player: Player | None = None
self._after_id: str | None = None
self._playing: Path | None = None
self._widgets = self._build()
self._bind_keys()
@property
def widgets(self) -> Widgets:
"""Return the constructed widgets.
Returns:
Widgets: The window's parts.
"""
return self._widgets
@property
def state(self) -> SessionState:
"""Return the current session state.
Returns:
SessionState: What the window is showing.
"""
return self._state
def _build(self) -> Widgets:
"""Construct the window and its widgets.
Returns:
Widgets: The constructed parts.
"""
root = tk.Tk()
root.title("/wsg/")
root.geometry(f"{WINDOW_WIDTH_PX}x{WINDOW_HEIGHT_PX}")
root.configure(bg=_BACKGROUND)
video = tk.Frame(root, bg="black")
video.pack(fill=tk.BOTH, expand=True)
controls = tk.Frame(root, bg=_BACKGROUND, height=CONTROL_BAR_HEIGHT_PX)
controls.pack(fill=tk.X, side=tk.BOTTOM)
keep = tk.Button(
controls,
text="✔ KEEP (k)",
bg=_ACCENT,
fg=_BACKGROUND,
command=self.on_keep,
)
keep.pack(side=tk.LEFT, padx=16, pady=10)
skip = tk.Button(
controls,
text="✘ PASS (j)",
bg=_BACKGROUND,
fg=_TEXT,
command=self.on_pass,
)
skip.pack(side=tk.RIGHT, padx=16, pady=10)
undo = tk.Button(
controls,
text="↶ UNDO (u)",
bg=_BACKGROUND,
fg=_TEXT,
command=self.on_undo,
)
undo.pack(side=tk.LEFT, padx=8, pady=10)
status = tk.Label(controls, text="", bg=_BACKGROUND, fg=_ACCENT)
status.pack()
filename = tk.Label(controls, text="", bg=_BACKGROUND, fg=_TEXT)
filename.pack()
root.protocol("WM_DELETE_WINDOW", self.on_quit)
return Widgets(
root=root,
video=video,
status=status,
filename=filename,
keep=keep,
skip=skip,
undo=undo,
)
def _bind_keys(self) -> None:
"""Bind the keyboard shortcuts onto the toplevel."""
for sequence in KEEP_KEYS:
self._widgets.root.bind(sequence, lambda _event: self.on_keep())
for sequence in PASS_KEYS:
self._widgets.root.bind(sequence, lambda _event: self.on_pass())
for sequence in UNDO_KEYS:
self._widgets.root.bind(sequence, lambda _event: self.on_undo())
for sequence in QUIT_KEYS:
self._widgets.root.bind(sequence, lambda _event: self.on_quit())
def video_wid(self) -> int:
"""Return the X11 id of the video frame.
Returns:
int: Window id to hand to mpv's ``--wid``.
"""
self._widgets.root.update()
return int(self._widgets.video.winfo_id())
def on_keep(self) -> None:
"""Keep the current video."""
self._verdict(Verdict.KEEP)
def on_pass(self) -> None:
"""Pass on the current video."""
self._verdict(Verdict.SKIP)
def shutdown(self) -> None:
"""Cancel the tick, stop the worker and destroy the window."""
logs.event(
"review.shutdown",
kept=self._state.kept,
passed=self._state.passed,
queued=len(self._state.pending),
)
try:
if self._after_id is not None:
self._widgets.root.after_cancel(self._after_id)
self._after_id = None
self._callbacks.shutdown()
player, self._player = self._player, None
if player is not None:
player.close()
finally:
# Whatever failed above -- a Ctrl-C landing in the worker join, an
# mpv that will not die -- the window must not be left up and
# frozen with its tick already cancelled.
self._widgets.root.destroy()
def run(self) -> None:
"""Start ticking and hand control to Tk."""
self.tick()
self._widgets.root.mainloop()

99
python_pkg/wsg_grabber/verdict.py
View File

@ -1,99 +0,0 @@
"""What a keep/pass decision means, expressed as data.
Planning a move and performing it are split deliberately: everything here is
pure and exhaustively testable, and :mod:`python_pkg.wsg_grabber.files` is the
only code that touches the filesystem.
Nothing in this package deletes a video. A pass relocates it to ``trash/`` and
leaves clearing that directory to the user.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from python_pkg.wsg_grabber.models import FileMove, Verdict
from python_pkg.wsg_grabber.states import FileState
if TYPE_CHECKING:
from collections.abc import Container
from pathlib import Path
from python_pkg.wsg_grabber.models import ReviewItem
def target_state(choice: Verdict) -> FileState:
"""Return the terminal state a verdict leads to.
Args:
choice: The user's decision.
Returns:
FileState: ``KEPT`` or ``PASSED``.
"""
return FileState.KEPT if choice is Verdict.KEEP else FileState.PASSED
def target_dir(choice: Verdict, keep: Path, trash: Path) -> Path:
"""Return the directory a verdict sends a file to.
Args:
choice: The user's decision.
keep: Directory for kept videos.
trash: Directory for passed videos.
Returns:
Path: Destination directory.
"""
return keep if choice is Verdict.KEEP else trash
def unique_destination(directory: Path, name: str, taken: Container[str]) -> Path:
"""Return a collision-free destination inside *directory*.
``taken`` is supplied by the caller rather than read from disk, keeping this
function pure and its collision handling directly testable.
Args:
directory: Destination directory.
name: Preferred filename.
taken: Names already present.
Returns:
Path: A path whose name is not in *taken*.
"""
if name not in taken:
return directory / name
stem, _, suffix = name.rpartition(".")
base, ext = (stem, f".{suffix}") if stem else (name, "")
index = 2
while f"{base}-{index}{ext}" in taken:
index += 1
return directory / f"{base}-{index}{ext}"
def plan_move(
item: ReviewItem,
choice: Verdict,
keep: Path,
trash: Path,
taken: Container[str] = (),
) -> FileMove:
"""Work out where a reviewed video should end up.
Args:
item: The video under review.
choice: The user's decision.
keep: Directory for kept videos.
trash: Directory for passed videos.
taken: Names already used in the destination directory.
Returns:
FileMove: Source, destination and the file's identity.
"""
destination = target_dir(choice, keep, trash)
return FileMove(
md5=item.md5,
src=item.path,
dst=unique_destination(destination, item.path.name, taken),
)