testsAndMisc/python_pkg/wsg_grabber/README.md

wsg_grabber

Scrapes 4chan's /wsg/ board for videos, downloads every one it has not seen before, and shows them one at a time for a keep/pass decision — mpv embedded in a Tk window with the buttons directly underneath.

PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber          # review
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber scrape   # no UI
PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber stats

k / → keeps, j / space / ← passes, u / Backspace undoes, q / Esc quits. The buttons do the same thing for the mouse.

Undo

Misclicked? u takes the last verdict back: the file returns from the keep folder or trash/ to incoming/, the index row goes back to ready, the counter is corrected, and that video is on screen again. Whatever was showing when you hit undo goes to the front of the queue, so nothing is skipped.

It survives quitting — the trail lives in the index, not in memory, so reopening the reviewer can still undo what you decided yesterday. The status bar shows how many are left (u undoes (17)), and the button greys out when there is nothing to take back. Keeps are undoable for as long as the file is still in any place a keep can end up (housekeeping.locate_reviewed looks in this year's cloud folder, earlier years', ~/Downloads, the old keep/, and wherever the media sync swept it); passes are undoable while they are among the newest TRASH_RETAIN (200). If a file has gone anyway, undo says so in the log and drops that entry rather than failing on it repeatedly.

Storage

~/data/cloud/Media/<year>/wsg/   you kept these (falls back to ~/Downloads
                                 on a machine without the cloud folder)
~/.local/share/wsg_grabber/
├── index.db     sqlite, remembers every file ever seen
├── incoming/    downloaded, awaiting your verdict
└── trash/       you passed on these — only the newest 200 passes stay

A keep goes where you actually look: the dufs cloud folder, so it shows up in the gallery and on the phone. The first start after this change moved the old keep/ contents across (_migrate_keep.py).

A pass goes into trash/, and _prune.py deletes everything there except the 200 most recent passes — at startup and after every pass. Their index rows become purged: still known, so the file is never downloaded again, but no longer undoable. Before this cap existed 23k files / 77 GB had piled up. files.apply_move still refuses to overwrite an existing destination, so a move can never destroy a video you already decided on.

Why it never shows you the same video twice

Every 4chan post that carries a file also carries md5 — the base64 MD5 of the whole file, published by the API before you fetch anything. That is the primary key of the files table, so:

• a video you have already reviewed is never downloaded again, at a cost of zero bytes;
• the same clip reposted in another thread collapses onto the existing row, which on /wsg/ is the common case;
• a truncated or mangled download is caught, because the bytes are checked against that same value before the file is offered for review.

Verified against a live file during development: the API's Nz9OEKdMuMZEdYE6eLTKmA== matched base64(md5(bytes)) exactly.

Being a good citizen

• One request per second to a.4cdn.org, which is what the API rules ask for.
• If-Modified-Since on every API request — thread bodies, threads.json and archive.json alike — plus a last_modified comparison against the stored value, so an unchanged thread costs a 304 and nothing else. A steady-state rescan is a couple of conditional requests that return 304.
• Accept-Encoding: identity on both hosts, and hard caps on response and download size. A compressed response is refused outright: a few hundred KB on the wire can decode to gigabytes, and the md5 check does not help when the same party publishes both the bytes and the digest.
• One long-lived requests.Session. This is required rather than merely tidy: i.4cdn.org answers with 429 under load and sets __cf_bm / _cfuvid cookies that have to persist across requests.
• 429 and 5xx back off exponentially, honouring Retry-After.

Downloading is prioritised over indexing

While anything is queued the worker downloads rather than walking more threads. That is deliberate: it is what puts a playable video on screen seconds after launch instead of after the whole board has been catalogued. The consequence is that stats shows the known-file count climbing in bursts rather than smoothly — the board is fully indexed once downloads drain. A first full run is roughly 9,000 files and 20–25 GB, so expect a couple of hours; it is resumable and restarting picks up mid-file via HTTP Range.

Logs

Every run writes one JSON object per line to ~/.local/share/wsg_grabber/logs/session-<time>-<pid>.jsonl.

wsg-grabber logs                # where they are, plus a tally of the newest
wsg-grabber --log-level debug   # record every IPC command as well
wsg-grabber --echo-log          # mirror records to stderr while running

The point is after-the-fact diagnosis. If the reviewer ever looks stuck on one video while the filenames keep advancing, one query settles whether mpv was handed the new file and was actually decoding it:

LOG=$(ls -t ~/.local/share/wsg_grabber/logs/*.jsonl | head -1)
jq -r 'select(.event=="review.show")
       | "show \(.file)  mpv_was: \(.mpv_before.path) t=\(.mpv_before["playback-time"]) \(.mpv_before.width)x\(.mpv_before.height)"' "$LOG"

A healthy run shows playback-time advancing and the dimensions changing per video. A frozen one shows them static, which points at the player rather than at the queue. Other useful queries:

jq -c 'select(.level=="error")' "$LOG"                          # what went wrong
jq -r 'select(.event=="review.verdict")|"\(.choice) \(.on_file)"' "$LOG"
jq -c 'select(.event|startswith("player."))' "$LOG"

Field names are stable; every record carries ts, seq, level, event and thread.

Debugging without touching your queue

paths honours XDG_DATA_HOME, so a throwaway instance costs nothing and can neither consume real videos nor record real verdicts:

XDG_DATA_HOME=/tmp/wsg-sandbox python3 -m python_pkg.wsg_grabber stats

Layout

Module	Responsibility
catalog.py	Parses the board's JSON. Pure.
scanner.py	Scheduling, backoff, resume decisions. Pure.
review.py	Everything the reviewer does: queue, verdicts, status text. Pure.
verdict.py	What keep/pass means, as a planned file move. Pure.
net.py	The only module that speaks HTTP.
db.py	sqlite schema, connection setup, migrations.
store.py	The download lifecycle of a file.
store_threads.py	Which threads were visited, and their HTTP stamps.
store_verdicts.py	The review trail that undo walks back.
downloader.py	The background worker thread.
files.py	The only module that moves or deletes files.
housekeeping.py	Around a verdict: keep migration, trash prune, finding a keep.
_migrate_keep.py	One-off move of the old keep/ into the cloud folder.
_prune.py	Caps trash/ at the newest 200 passes.
player.py	mpv as a subprocess over its JSON IPC socket.
ui.py	The only module that imports tkinter.
logs.py	Structured JSONL event log.
app.py / cli.py	Composition and argument parsing.

The split is what makes 100% branch coverage achievable: ui.py and player.py are close to pure wiring, and every decision they would otherwise make lives in a pure function tested from plain values with no display and no network. Two decisions do remain in ui.py, both because a bug proved they had to: the already-playing guard (without it loadfile fires every tick and the video restarts continuously) and dropping an item whose file has vanished.

mpv is a subprocess, not python-mpv

python-mpv resolves libmpv at import time and raises OSError wherever it is missing — including this repo's CI runner, so it would pass locally and fail on push. Driving mpv --wid=<frame> --input-ipc-server=<socket> over a unix socket needs only subprocess, socket and json.

The socket must be drained. mpv pushes an asynchronous event stream plus a reply per command down the same connection. A client that only ever writes lets that backlog fill the kernel buffer, at which point mpv's writer for that connection blocks and it stops reading commands. Nothing errors: sendall still succeeds, mpv stays alive and keeps playing whatever it had, so the reviewer advances through filenames while the picture never changes. Measured: commands were silently ignored from about the thirty-fifth video onwards, and the very same command sent on a fresh connection worked instantly. player.py therefore runs a reader thread whose only job is to drain — deleting it reintroduces the freeze roughly half an hour into a session.

Two more mpv details worth knowing if you touch ui.py:

• --wid does not draw into your widget. mpv creates its own X window and reparents it onto the frame, covering it completely — which is why the control bar is packed below the video rather than over it.
• --input-vo-keyboard=no is what lets the Tk key bindings fire at all. Without it mpv's child window takes keyboard focus and Tk never sees a keypress.

Requirements

mpv and tk (system packages, installed by install.sh) and requests, which the repo already declares. Nothing else.