testsAndMisc/docs/superpowers/evidence/wsg-grabber-2026-07-27.json

{
  "intent": "Add python_pkg/wsg_grabber: scrapes 4chan /wsg/ for videos, downloads every one not seen before in the background, and reviews them one at a time with mpv embedded in a Tk window and Keep/Pass controls directly beneath the video. A reviewed video is never shown or downloaded again.",
  "scope": [
    "New package python_pkg/wsg_grabber/ (19 modules + tests, README.md, install.sh)",
    "New docs/superpowers/contracts/wsg-grabber-2026-07-27.json",
    "No change to meta/requirements.txt: the only third-party dependency is requests, already declared. mpv and tk are system packages installed by install.sh",
    "Non-goals: any board but /wsg/, a download cap, automatic deletion of anything, perceptual dedup"
  ],
  "changes": [
    "Dedup is structural rather than conventional: the API publishes `md5` (base64 MD5 of the entire file) on every post carrying an attachment, so it is used as the sqlite primary key. An already-seen file is skipped before a byte is transferred, and a repost of identical bytes in another thread collapses onto the existing row. Verified byte-exact against a live download: the API's Nz9OEKdMuMZEdYE6eLTKmA== matched base64(md5(bytes)) computed locally.",
    "sqlite (stdlib) instead of this repo's usual JSON state files, because two threads write concurrently -- the downloader appending rows, the UI recording verdicts -- and the seen-set grows without bound. Noted as a deliberate deviation.",
    "mpv is driven as a subprocess over its JSON IPC socket, NOT via python-mpv. python-mpv calls ctypes.util.find_library('mpv') at module scope (mpv.py:70-73) and raises OSError wherever libmpv is absent, which includes .github/workflows/python-tests.yml's ubuntu-latest runner: it would pass locally and fail on push. The lazy-import workaround trips PLC0415, which cannot be suppressed here.",
    "Strict pure-logic/IO-shell split: ui.py (the only module importing tkinter) and player.py contain no decisions. review.py produces a ReviewCommand dataclass that the window applies verbatim, which is what makes 100% branch coverage of the GUI achievable without a display.",
    "A terminal `gone` state stops a 404 or upstream-deleted file being retried on every future run -- necessary because threads fall off page 10 during a multi-hour first pass. An already-downloaded file keeps its state when the post is deleted, since the bytes are already ours.",
    "Politeness: <=1 request/second to a.4cdn.org, If-Modified-Since on every API GET (verified 304), one persistent requests.Session (required because i.4cdn.org 429s under load and sets __cf_bm/_cfuvid cookies), Retry-After-aware exponential backoff, and Range-based resume of partial downloads verified against the API md5 before the file is offered.",
    "Downloads outrank further indexing so the first playable video appears seconds after launch while the rest of the board loads behind it.",
    "tests/conftest.py adds the repo's first autouse path-isolation fixture, redirecting Path.home, HOME and XDG_DATA_HOME, plus an AST guard in test_paths.py asserting no module resolves a home directory at import time -- the failure mode that previously leaked test writes into real user data."
  ],
  "verification": [
    {
      "command": "PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber scrape --seconds 100",
      "result": "pass",
      "evidence": "Indexed 123 files and downloaded 97 of them (472 MB) in 100s at the intended ~1/s. Progress line advanced known 123 / ready 11 -> 96 / pending 111 -> 26."
    },
    {
      "command": "PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber  (reviewer window, screenshotted on HDMI-0)",
      "result": "pass",
      "evidence": "Video renders INSIDE the Tk frame with the Keep/Pass bar beneath it in the same window; status line read 'passed 0 - queued 165 - downloaded 10/2..' and the filename line 'Hinamatsuri 2.webm - 1920x1080 - 5.9 MiB', i.e. playback and downloading proceeding together. mpv reported vo-configured=True, geometry='' and osd-dimensions 636x1337, matching the frame."
    },
    {
      "command": "First reviewer run exposed a real bug not caught by 213 passing tests",
      "result": "fixed",
      "evidence": "The video area was black. Querying mpv over IPC showed playback-time=10.1 and vo-configured=True but geometry='329x180': the <Configure> handler was sending set_property geometry, which fights the automatic sizing that --wid already performs ('the window will always be resized to cover the parent window fully', man mpv). Removed the resize plumbing from player.py, ui.py and review.py; re-ran and the video fills the frame."
    },
    {
      "command": "Second reviewer run exposed a second real bug, also invisible to the tests",
      "result": "fixed",
      "evidence": "mpv reported osd-width=0 and playback that never advanced. Cause: tick() calls refresh() -> apply() -> _play() unconditionally, so `loadfile ... replace` was being sent roughly seven times a second and the video restarted continuously instead of playing. ui.py now tracks the currently-playing path and only loads when it changes; stop is likewise only sent when something is playing. Three regression tests added (test_the_same_video_is_only_loaded_once drives ten ticks and asserts play was called once). Re-ran: osd-width=849, and three screenshots taken three seconds apart differ by 2.6M and 5.2M pixels, i.e. playback demonstrably advancing."
    },
    {
      "command": "Third bug, found from a user-reported session log rather than by reproduction",
      "result": "fixed",
      "evidence": "User reported the reviewer freezing on one video around the 26th while filenames kept advancing. The session log showed it exactly: four consecutive review.show records with mpv_before.path pinned to one file and playback-time tracking wall-clock (13.0s gap -> t=12.98, +1.4 -> 14.41, +1.8 -> 16.18, +1.3 -> 17.48), core-idle=false, every later file present on disk and every loadfile sent without error. Root cause: mpv pushes an async event stream plus a per-command reply down the same socket, and the client never read from it; the kernel buffer filled, mpv's writer for that connection blocked, and it stopped reading commands. Isolated with a GUI-free harness: without draining, commands were silently ignored from #35 while the identical command on a FRESH socket was honoured instantly; with a drain thread, all 60 were honoured. Fixed by a reader thread in player.py that drains the socket and logs mpv's error replies. Verified through the real app: 50 consecutive passes, zero mismatches between what the UI loaded and what mpv held. Two earlier hypotheses (a run of visually-identical reposts, and an mpv repaint failure) were tested and disproved -- the 31 files the user passed had zero near-duplicate first frames, and a minimal mpv --wid harness switched files correctly."
    },
    {
      "command": "Undo, added on request after a misclick lost a video to the wrong pile",
      "result": "pass",
      "evidence": "`u` / Backspace and an UNDO button take back the last verdict: the file moves back from keep/ or trash/ into incoming/, the index row returns to ready, the counter is corrected, and the video is shown again -- with whatever was on screen pushed to the front of the queue so nothing is skipped. Multi-level within a session. Verified against real files: KEEP -> undo -> PASS -> PASS -> undo -> undo returned kept/passed to 0/0 with all four files back in incoming/ and keep/ and trash/ empty. Edge cases covered by tests: a destination that was renamed to avoid a collision (undo follows the real name), an empty history (no-op), and a file the user deleted from trash/ by hand (logged, entry dropped rather than failing repeatedly)."
    },
    {
      "command": "Clean-shutdown check after closing the reviewer window",
      "result": "pass",
      "evidence": "ps shows no reviewer process and no orphan mpv; the IPC socket is removed; one .part file remains, which is correct and resumable."
    },
    {
      "command": "Dedup acceptance: mark 189 downloaded files reviewed, then rescan",
      "result": "pass",
      "evidence": "Marked 94 kept / 95 passed. After a 45s rescan, kept=94 and passed=95 were unchanged while known files rose 221 -> 236, i.e. 15 genuinely new files were picked up and not one reviewed file was re-offered or re-downloaded."
    },
    {
      "command": "systemd-run --user --scope -p MemoryMax=2G -p MemorySwapMax=0 python -m pytest python_pkg/wsg_grabber --cov --cov-branch -W error",
      "result": "pass",
      "evidence": "272 passed. 100% branch coverage on all 22 modules, including the two that touch the untestable world: ui.py (160 stmts / 30 branches) and player.py (127 / 22). Run under `systemd-run --user --scope -p MemoryMax=2G -p MemorySwapMax=0` and with -W error, so a leaked thread, unclosed sqlite connection or undestroyed Tk object fails the suite."
    },
    {
      "command": "ruff check / mypy / pylint",
      "result": "pass",
      "evidence": "ruff (select=ALL): All checks passed. mypy strict: no issues in 36 source files. pylint: 10.00/10. grep confirms zero '# noqa' and zero '# type: ignore' in the package."
    },
    {
      "command": "shellcheck --severity=warning python_pkg/wsg_grabber/install.sh",
      "result": "pass",
      "evidence": "Clean after splitting declare/assign for SC2155; --help runs."
    },
    {
      "command": "Five-lens code review (undo/migration, threads, data safety, untrusted input, repo rules)",
      "result": "fixed",
      "evidence": "Reviewers found seven real defects, each reproduced before fixing and re-verified after. (1) threads.json and archive.json were fetched UNCONDITIONALLY every idle cycle -- two un-cached requests a minute forever against a host the README promises to be polite to; now conditional via a new http_cache table (schema v3), proven by probe: first sweep sends no stamp, second sends it back. (2) Session.undo built its destination directly, bypassing unique_destination, and Path.replace clobbers silently -- a video in incoming/ could be destroyed with no trash copy; undo now picks a collision-free name AND files.apply_move refuses any occupied destination, making the guarantee structural rather than per-caller. (3) A 24-char md5 containing NUL passed validation, became a PRIMARY KEY and a filename, and crashed the worker before the row could be marked gone -- poisoning the index permanently across restarts; _as_md5 now matches the true base64 shape [A-Za-z0-9+/]{22}==. (4) An API integer >= 2**63 raised OverflowError, which was NOT in the worker's recoverable set, killing the thread with no WorkerFailed event and a permanently idle UI; _as_int now bounds to sqlite's signed 64-bit range and ArithmeticError was added to _RECOVERABLE. (5) A hostile CDN could serve a gzip bomb: 200 KB on the wire decoded to 200 MB on disk and was ACCEPTED, because the same party publishes the md5; a non-identity Content-Encoding is now refused outright. (6) Nothing compared bytes written against the advertised fsize -- a lying server filled the disk (64 MiB written against fsize=4); the write loop now aborts past the advertised size plus slack. (7) The API session advertised gzip/deflate/br/zstd and json.loads had no size limit (6 KB zstd -> 200 MB decoded); Accept-Encoding is now identity and responses over 32 MB are refused. Session.missing could also overwrite an already-recorded verdict, silently corrupting the undo trail -- it now refuses rows in TERMINAL."
    },
    {
      "command": "Real-data-dir safety, verified rather than assumed",
      "result": "pass",
      "evidence": "A reviewer snapshotted the live ~/.local/share/wsg_grabber (3676 entries, 4.6 GB, index.db md5 b4857900c1a1fcb20e1e05de97a349b2), ran the full suite, and diffed: empty, at every checkpoint, including after five adversarial probes. Every storage path in the package derives from paths.data_dir(), so there is no other write target."
    }
  ],
  "risks": [
    "Keyboard delivery to the Tk bindings is UNVERIFIED. --input-vo-keyboard=no is set precisely so mpv declines the keyboard and X delivers keys to the focused Tk toplevel, but confirming it needs real key input, which requires stealing focus from the fullscreen game that was running throughout. Synthetic XSendEvent keys are ignored by both mpv and Tk and prove nothing. If it turns out mpv still swallows keys, the fallback is one IPC `keybind` command per shortcut plus a small socket reader, contained entirely in player.py. The mouse buttons are wired to the same handlers and are unaffected.",
    "The package now holds real user data in ~/.local/share/wsg_grabber. The autouse isolation fixture plus the AST guard are what keep the test suite out of it.",
    "A first full run is roughly 9,000 files and 20-25 GB over a couple of hours -- inherent to the explicit 'download everything, no cap' decision. It is resumable and restarting continues mid-file.",
    "md5 catches byte-identical reposts only. A re-encoded repost is a different file and will be shown again; perceptual dedup is out of scope.",
    "Keyboard delivery to the Tk bindings (k/j/u) is still UNVERIFIED. --input-vo-keyboard=no exists so mpv declines the keyboard and X delivers to the focused toplevel, but confirming it needs real key input, which means stealing focus from the fullscreen game that ran throughout. The mouse buttons are wired to the same handlers and are unaffected.",
    "Two reviewer instances against one index can still interfere: there is no lock file. The worst case is now bounded -- Session.missing refuses to clobber a recorded verdict -- but the queues are not coordinated."
  ],
  "rollback": [
    "git rm -r python_pkg/wsg_grabber docs/superpowers/contracts/wsg-grabber-2026-07-27.json docs/superpowers/evidence/wsg-grabber-2026-07-27.json; nothing else in the repo imports it and meta/requirements.txt was not modified",
    "sudo python_pkg/wsg_grabber/install.sh --uninstall removes /usr/local/bin/wsg-grabber and leaves downloaded videos alone",
    "~/.local/share/wsg_grabber is untouched by removal; delete it by hand if the downloaded videos are no longer wanted",
    "After rollback confirm the rest of the suite still reaches 100% coverage: python -m pytest python_pkg --cov=python_pkg --cov-branch"
  ]
}