testsAndMisc/python_pkg/wsg_grabber/models.py
Krzysztof kuhy Rudnicki 89ed9bc95d
Some checks failed
Pre-commit checks / pre-commit (push) Has been cancelled
Python tests / test (push) Has been cancelled
Shell tests / shell-tests (push) Has been cancelled
Shell tests / arch-tests (push) Has been cancelled
wsg_grabber: scrape /wsg/, download every new video, triage with keep/pass
New package. Walks every thread on 4chan's /wsg/, downloads each video it has
not seen, and reviews them one at a time: mpv embedded in a Tk window via --wid
with Keep/Pass/Undo directly beneath the picture.

Dedupe is structural rather than a convention. Every post carrying a file also
carries md5 -- base64 MD5 of the whole file -- so it is the sqlite primary key.
An already-seen video costs zero bytes to skip and a byte-identical repost in
another thread collapses onto the existing row. Verified against a live file:
the API's Nz9OEKdMuMZEdYE6eLTKmA== matched base64(md5(bytes)) exactly.

Undo is unbounded and survives quitting: the verdict trail lives in the index,
not in memory, so yesterday's decisions can still be taken back.

No downloaded video is ever deleted. A pass moves the file to trash/ and leaves
it there; files.apply_move refuses to overwrite an existing destination, so the
guarantee holds by construction rather than per caller.

Three bugs only running it could find, each caught from evidence rather than
guesswork:
  - the video stayed black: a <Configure> handler set mpv's geometry, fighting
    the automatic sizing --wid already does
  - playback never advanced: refresh() ran every tick and re-sent loadfile ~7x
    a second, restarting the video continuously
  - it froze on one video around the 26th while filenames kept advancing: mpv
    pushes an event stream down the same socket and nothing drained it, so the
    buffer filled and mpv stopped reading commands. Isolated to command #35
    without a drain thread; all 60 honoured with one.

A five-lens review then found nine more, all reproduced before fixing: a failed
mpv start stranded the non-daemon worker forever; undo could clobber a file in
incoming/; a NUL-bearing md5 poisoned the index permanently; an oversized int
killed the worker with no event; gzip bombs on both hosts were accepted; the
board endpoints were fetched unconditionally forever; and Session.missing could
corrupt the undo trail.

mpv runs as a subprocess over its JSON IPC socket rather than via python-mpv,
which resolves libmpv at import time and would fail on CI while passing here.
Third-party surface is requests alone, already declared.

277 package tests, 100% branch coverage on all 22 modules, no suppressions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 14:55:30 +02:00

210 lines
4.6 KiB
Python

"""Plain-data records shared across the package.
Everything crossing a module boundary is a frozen dataclass or an enum, so the
pure planning code can be driven from hand-built literals and the GUI never has
to be constructed to test a decision.
"""
from __future__ import annotations
from dataclasses import dataclass
from enum import Enum
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from pathlib import Path
# ``str, Enum`` rather than ``StrEnum`` throughout: mypy runs with
# ``python_version = "3.10"`` and ``StrEnum`` only exists from 3.11, so it would
# degrade to ``Any`` and silently disable type checking on every enum here.
class Verdict(str, Enum):
"""What the user decided about a video.
The member is ``SKIP`` rather than ``PASS`` only because ruff's S105 reads a
name of ``PASS`` as a hardcoded password and this repo forbids ``noqa``. The
stored value stays ``"pass"``, which is the word the UI shows.
"""
KEEP = "keep"
SKIP = "pass"
class TaskKind(str, Enum):
"""What the background worker should do next."""
DOWNLOAD = "download"
SCAN = "scan"
IDLE = "idle"
class Emptiness(str, Enum):
"""Why the reviewer has nothing to show."""
NOT_EMPTY = "not_empty"
WAITING = "waiting"
EXHAUSTED = "exhausted"
@dataclass(frozen=True, slots=True)
class Size:
"""A pixel size."""
width: int
height: int
@dataclass(frozen=True, slots=True)
class ThreadRef:
"""A thread as advertised by ``threads.json``."""
thread_no: int
api_last_modified: int
@dataclass(frozen=True, slots=True)
class RemoteFile:
"""An attachment described by a post, before anything is downloaded.
``md5`` is the API's own ``base64(md5(file_bytes))`` and is the identity of
the file everywhere in this package.
"""
md5: str
tim: int
ext: str
orig_name: str
fsize: int
width: int
height: int
thread_no: int
post_no: int
@dataclass(frozen=True, slots=True)
class ReviewItem:
"""A downloaded video waiting for a verdict."""
md5: str
path: Path
orig_name: str
fsize: int
width: int
height: int
@dataclass(frozen=True, slots=True)
class FileMove:
"""A planned relocation. Applying it is the only filesystem write."""
md5: str
src: Path
dst: Path
@dataclass(frozen=True, slots=True)
class ReviewedItem:
"""A verdict recorded in the index, and therefore reversible.
``reviewed_name`` is the name the file was given in keep/ or trash/, which
differs from ``item.path.name`` whenever a collision forced a rename. Undo
needs it to find the file again, which is why it is persisted rather than
held in memory.
"""
item: ReviewItem
choice: Verdict
reviewed_name: str
@dataclass(frozen=True, slots=True)
class ResumePlan:
"""How to continue a partial download."""
offset: int
discard: bool
class Outcome(str, Enum):
"""How a download ended."""
COMPLETED = "completed"
NOT_FOUND = "not_found"
TRANSIENT = "transient"
CHECKSUM_MISMATCH = "checksum_mismatch"
ABORTED = "aborted"
@dataclass(frozen=True, slots=True)
class JsonResponse:
"""The result of a conditional GET against the API."""
payload: object | None
last_modified: str | None
not_modified: bool
not_found: bool
@dataclass(frozen=True, slots=True)
class DownloadResult:
"""The result of streaming one file to disk."""
outcome: Outcome
bytes_done: int
retry_after: float | None = None
message: str | None = None
@dataclass(frozen=True, slots=True)
class ReviewCommand:
"""Everything the GUI must render or do, already decided.
The window applies this verbatim; it never formats a string or picks a file.
"""
play: Path | None
stop: bool
status: str
title: str
filename: str
verdicts_enabled: bool
undo_enabled: bool
quit_app: bool
@dataclass(frozen=True, slots=True)
class FileReady:
"""A newly downloaded video is available for review."""
item: ReviewItem
@dataclass(frozen=True, slots=True)
class Indexed:
"""The catalogue scan learned about more files."""
known: int
@dataclass(frozen=True, slots=True)
class Downloaded:
"""Running count of files fetched this session."""
count: int
@dataclass(frozen=True, slots=True)
class ScanFinished:
"""The board has been walked and every known file downloaded."""
@dataclass(frozen=True, slots=True)
class WorkerFailed:
"""The background worker died; the message is shown in the status bar."""
message: str
DownloadEvent = FileReady | Indexed | Downloaded | ScanFinished | WorkerFailed