# wsg_grabber Scrapes 4chan's /wsg/ board for videos, downloads every one it has not seen before, and shows them one at a time for a keep/pass decision — mpv embedded in a Tk window with the buttons directly underneath. ```bash PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber # review PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber scrape # no UI PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber stats ``` `k` / `→` keeps, `j` / `space` / `←` passes, `u` / `Backspace` undoes, `q` / `Esc` quits. The buttons do the same thing for the mouse. ## Undo Misclicked? `u` takes the last verdict back: the file returns from the keep folder or `trash/` to `incoming/`, the index row goes back to _ready_, the counter is corrected, and that video is on screen again. Whatever was showing when you hit undo goes to the front of the queue, so nothing is skipped. It **survives quitting** — the trail lives in the index, not in memory, so reopening the reviewer can still undo what you decided yesterday. The status bar shows how many are left (`u undoes (17)`), and the button greys out when there is nothing to take back. Keeps are undoable for as long as the file is still in any place a keep can end up (`housekeeping.locate_reviewed` looks in this year's cloud folder, earlier years', `~/Downloads`, the old `keep/`, and wherever the media sync swept it); passes are undoable while they are among the newest `TRASH_RETAIN` (200). If a file has gone anyway, undo says so in the log and drops that entry rather than failing on it repeatedly. ## Storage ``` ~/data/cloud/Media//wsg/ you kept these (falls back to ~/Downloads on a machine without the cloud folder) ~/.local/share/wsg_grabber/ ├── index.db sqlite, remembers every file ever seen ├── incoming/ downloaded, awaiting your verdict └── trash/ you passed on these — only the newest 200 passes stay ``` A keep goes where you actually look: the dufs cloud folder, so it shows up in the gallery and on the phone. The first start after this change moved the old `keep/` contents across (`_migrate_keep.py`). A pass goes into `trash/`, and `_prune.py` deletes everything there except the 200 most recent passes — at startup and after every pass. Their index rows become _purged_: still known, so the file is never downloaded again, but no longer undoable. Before this cap existed 23k files / 77 GB had piled up. `files.apply_move` still refuses to overwrite an existing destination, so a move can never destroy a video you already decided on. ## Why it never shows you the same video twice Every 4chan post that carries a file also carries `md5` — the base64 MD5 of the whole file, published by the API before you fetch anything. That is the primary key of the `files` table, so: - a video you have already reviewed is never downloaded again, at a cost of zero bytes; - the same clip reposted in another thread collapses onto the existing row, which on /wsg/ is the common case; - a truncated or mangled download is caught, because the bytes are checked against that same value before the file is offered for review. Verified against a live file during development: the API's `Nz9OEKdMuMZEdYE6eLTKmA==` matched `base64(md5(bytes))` exactly. ## Being a good citizen - One request per second to `a.4cdn.org`, which is what the API rules ask for. - `If-Modified-Since` on every API request — thread bodies, `threads.json` and `archive.json` alike — plus a `last_modified` comparison against the stored value, so an unchanged thread costs a 304 and nothing else. A steady-state rescan is a couple of conditional requests that return 304. - `Accept-Encoding: identity` on both hosts, and hard caps on response and download size. A compressed response is refused outright: a few hundred KB on the wire can decode to gigabytes, and the md5 check does not help when the same party publishes both the bytes and the digest. - One long-lived `requests.Session`. This is required rather than merely tidy: `i.4cdn.org` answers with 429 under load and sets `__cf_bm` / `_cfuvid` cookies that have to persist across requests. - 429 and 5xx back off exponentially, honouring `Retry-After`. ## Downloading is prioritised over indexing While anything is queued the worker downloads rather than walking more threads. That is deliberate: it is what puts a playable video on screen seconds after launch instead of after the whole board has been catalogued. The consequence is that `stats` shows the known-file count climbing in bursts rather than smoothly — the board is fully indexed once downloads drain. A first full run is roughly 9,000 files and 20–25 GB, so expect a couple of hours; it is resumable and restarting picks up mid-file via HTTP `Range`. ## Logs Every run writes one JSON object per line to `~/.local/share/wsg_grabber/logs/session-