Commit Graph

3 Commits

Author SHA1 Message Date
adf2e402df Grade against 3.14 too, not just run on it
`requires-python` and the CI pin already said 3.14, but ruff, mypy and pylint
were still grading `meta/` against 3.10 -- three tools judging an interpreter
nothing runs any more, and ruff withholding the pyupgrade fixes that only
become safe at the newer target. `pin_ci_python.py` rewrites these three knobs.

What the stale target was hiding: `datetime.timezone.utc` where `datetime.UTC`
is now correct (UP017), and ruff's stable formatter applying PEP 758 --
`except (A, B):` becomes `except A, B:`. That makes the source 3.14-only at the
syntax level, which is what `requires-python = ">=3.14"` already promised; kuhy
chose it deliberately over holding ruff a release back (2026-08-29).

Also drops a comment in `wsg_grabber/models.py` that the bump makes false: it
explained why the file used ``str, Enum`` instead of ``StrEnum`` under mypy at
3.10, sitting directly above a ``class Verdict(StrEnum)`` that had already
stopped obeying it.

ruff, ruff-format, mypy --strict and pylint all clean at the new target.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SfnJmxxsZ534AKk16rC2Gr
2026-08-29 11:50:39 +02:00
f58caea2ab Split wsg_grabber/store.py under the 250-line cap
Some checks are pending
Pre-commit checks / pre-commit (push) Waiting to run
Python tests / test (push) Waiting to run
Shell tests / shell-tests (push) Waiting to run
Shell tests / arch-tests (push) Waiting to run
store.py mixed the writes that move a file through its lifecycle with the
read-only projections over the same table. The reads move to _store_queries.py:
claimable_values, ready_items, ready_item, counts, pending_downloads,
known_md5s, state_of, attempts_for.

claim_next goes with them despite being a write. It is the one write that has
to be a single atomic transaction against exactly those reads, so splitting it
from them would have left BEGIN IMMEDIATE on one side of a module boundary and
its SELECT on the other.

_scalar becomes scalar: record_files still counts rows before and after its
insert, so store imports it back. The dependency runs one way only.

store.py re-exports through __all__, so downloader.py, cli.py and test_store.py
keep reaching everything at store.* unchanged -- all 15 names verified to
resolve after ruff's autofix pass.

store.py 382 -> 202, _store_queries.py 232.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011w2Pj4Ek3ZNMQVYcFHgXDY
2026-08-16 15:52:08 +02:00
89ed9bc95d wsg_grabber: scrape /wsg/, download every new video, triage with keep/pass
Some checks failed
Pre-commit checks / pre-commit (push) Has been cancelled
Python tests / test (push) Has been cancelled
Shell tests / shell-tests (push) Has been cancelled
Shell tests / arch-tests (push) Has been cancelled
New package. Walks every thread on 4chan's /wsg/, downloads each video it has
not seen, and reviews them one at a time: mpv embedded in a Tk window via --wid
with Keep/Pass/Undo directly beneath the picture.

Dedupe is structural rather than a convention. Every post carrying a file also
carries md5 -- base64 MD5 of the whole file -- so it is the sqlite primary key.
An already-seen video costs zero bytes to skip and a byte-identical repost in
another thread collapses onto the existing row. Verified against a live file:
the API's Nz9OEKdMuMZEdYE6eLTKmA== matched base64(md5(bytes)) exactly.

Undo is unbounded and survives quitting: the verdict trail lives in the index,
not in memory, so yesterday's decisions can still be taken back.

No downloaded video is ever deleted. A pass moves the file to trash/ and leaves
it there; files.apply_move refuses to overwrite an existing destination, so the
guarantee holds by construction rather than per caller.

Three bugs only running it could find, each caught from evidence rather than
guesswork:
  - the video stayed black: a <Configure> handler set mpv's geometry, fighting
    the automatic sizing --wid already does
  - playback never advanced: refresh() ran every tick and re-sent loadfile ~7x
    a second, restarting the video continuously
  - it froze on one video around the 26th while filenames kept advancing: mpv
    pushes an event stream down the same socket and nothing drained it, so the
    buffer filled and mpv stopped reading commands. Isolated to command #35
    without a drain thread; all 60 honoured with one.

A five-lens review then found nine more, all reproduced before fixing: a failed
mpv start stranded the non-daemon worker forever; undo could clobber a file in
incoming/; a NUL-bearing md5 poisoned the index permanently; an oversized int
killed the worker with no event; gzip bombs on both hosts were accepted; the
board endpoints were fetched unconditionally forever; and Session.missing could
corrupt the undo trail.

mpv runs as a subprocess over its JSON IPC socket rather than via python-mpv,
which resolves libmpv at import time and would fail on CI while passing here.
Third-party surface is requests alone, already declared.

277 package tests, 100% branch coverage on all 22 modules, no suppressions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 14:55:30 +02:00