mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 12:07:52 +01:00
Krzysztof kuhy Rudnicki
89ed9bc95d
New package. Walks every thread on 4chan's /wsg/, downloads each video it has
not seen, and reviews them one at a time: mpv embedded in a Tk window via --wid
with Keep/Pass/Undo directly beneath the picture.
Dedupe is structural rather than a convention. Every post carrying a file also
carries md5 -- base64 MD5 of the whole file -- so it is the sqlite primary key.
An already-seen video costs zero bytes to skip and a byte-identical repost in
another thread collapses onto the existing row. Verified against a live file:
the API's Nz9OEKdMuMZEdYE6eLTKmA== matched base64(md5(bytes)) exactly.
Undo is unbounded and survives quitting: the verdict trail lives in the index,
not in memory, so yesterday's decisions can still be taken back.
No downloaded video is ever deleted. A pass moves the file to trash/ and leaves
it there; files.apply_move refuses to overwrite an existing destination, so the
guarantee holds by construction rather than per caller.
Three bugs only running it could find, each caught from evidence rather than
guesswork:
- the video stayed black: a <Configure> handler set mpv's geometry, fighting
the automatic sizing --wid already does
- playback never advanced: refresh() ran every tick and re-sent loadfile ~7x
a second, restarting the video continuously
- it froze on one video around the 26th while filenames kept advancing: mpv
pushes an event stream down the same socket and nothing drained it, so the
buffer filled and mpv stopped reading commands. Isolated to command #35
without a drain thread; all 60 honoured with one.
A five-lens review then found nine more, all reproduced before fixing: a failed
mpv start stranded the non-daemon worker forever; undo could clobber a file in
incoming/; a NUL-bearing md5 poisoned the index permanently; an oversized int
killed the worker with no event; gzip bombs on both hosts were accepted; the
board endpoints were fetched unconditionally forever; and Session.missing could
corrupt the undo trail.
mpv runs as a subprocess over its JSON IPC socket rather than via python-mpv,
which resolves libmpv at import time and would fail on CI while passing here.
Third-party surface is requests alone, already declared.
277 package tests, 100% branch coverage on all 22 modules, no suppressions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
22 lines
3.7 KiB
JSON
22 lines
3.7 KiB
JSON
{
|
|
"title": "New package python_pkg/wsg_grabber: scrape 4chan /wsg/ for videos and triage them with an embedded-mpv keep/pass reviewer",
|
|
"objective": "There is no tool for pulling /wsg/ video content and deciding what to keep. Build one that walks every thread on the board, downloads every video it has not already seen, and presents them one at a time in a tinder-style reviewer -- mpv embedded in a Tk window via --wid with Keep/Pass controls directly beneath the video. The defining requirement is that a video reviewed once is never shown or downloaded again, across runs and across reposts. Success: running the reviewer plays a downloaded video within seconds while the rest of the board continues downloading behind it; keep and pass relocate the file; and a subsequent run offers only genuinely new videos.",
|
|
"acceptance_criteria": [
|
|
"Deduplication uses the API's own `md5` field (base64 MD5 of the whole file, verified byte-exact against a live download) as the sqlite primary key, so an already-seen file costs zero bytes to skip and a cross-thread repost of identical bytes collapses onto the existing row",
|
|
"A pass moves the file to trash/ and a keep moves it to keep/; there is no unlink() anywhere in the package except to discard a corrupt partial download, so nothing the user might want back is ever destroyed",
|
|
"The board is polled politely: at most one request per second to a.4cdn.org, If-Modified-Since on every API request so an unchanged thread costs a 304, one persistent requests.Session (required, because i.4cdn.org 429s under load and sets __cf_bm/_cfuvid cookies), and Retry-After-aware exponential backoff",
|
|
"A file that 404s or is deleted upstream reaches a terminal `gone` state and is never retried on any future run; a file already downloaded keeps its state even if the post is deleted upstream",
|
|
"Downloads are prioritised over further indexing, so the first playable video reaches the screen seconds after launch rather than after the whole board is catalogued",
|
|
"100% branch coverage across every module including ui.py and player.py, with zero `# noqa` and zero `# type: ignore` (both blocked by pre-commit), ruff `select = [\"ALL\"]` clean, mypy strict clean, pylint 10.00/10, and every file under the repo's 500-line cap",
|
|
"No test constructs a real tkinter.Tk(), starts a real mpv, opens a socket, or writes outside tmp_path; an autouse fixture redirects Path.home, HOME and XDG_DATA_HOME so the suite cannot touch the real ~/.local/share/wsg_grabber"
|
|
],
|
|
"out_of_scope": [
|
|
"Any board other than /wsg/ -- the board is a hardcoded constant, not a flag, per the explicit scope decision",
|
|
"A disk budget, cache eviction or download cap -- the requirement is to download everything, so a first run is deliberately ~9,000 files and 20-25GB",
|
|
"Automatic emptying of trash/ -- clearing it is the user's decision and the package must never do it",
|
|
"Perceptual/near-duplicate detection -- md5 catches byte-identical reposts only, which is the dominant case on /wsg/; a re-encoded repost is a different file and will be shown again",
|
|
"python-mpv as the player binding -- it resolves libmpv at import time and would break CI, so mpv is driven as a subprocess over its JSON IPC socket instead"
|
|
],
|
|
"verifier": "PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber scrape --seconds 100 (downloads real files); PYTHONPATH=~/testsAndMisc python3 -m python_pkg.wsg_grabber (reviewer window renders video inside the Tk frame with the control bar beneath); mark files reviewed and rescan, confirming kept/passed counts are unchanged; python -m pytest python_pkg/wsg_grabber --cov --cov-branch --cov-fail-under=100; pre-commit run --files <changed>"
|
|
}
|