diff --git a/AGENTS.md b/AGENTS.md index 174c844e..248b1e26 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,6 +41,8 @@ Extracted to their own repos: [`steam-backlog-enforcer`](https://github.com/kuhy `~/android-guardian`), [`phone-focus-mode`](https://github.com/kuhyx/phone-focus-mode) (with `python_pkg/focus_policy`), +[`wsg-grabber`](https://github.com/kuhyx/wsg-grabber) (private; lives at +`~/src/wsg-grabber`), [`hosts-blocker`](https://github.com/kuhyx/hosts-blocker) (lives at `~/src/hosts-blocker`), [`digital-wellbeing`](https://github.com/kuhyx/digital-wellbeing) (lives at diff --git a/docs/superpowers/contracts/remove-wsg-grabber-2026-09-28.json b/docs/superpowers/contracts/remove-wsg-grabber-2026-09-28.json new file mode 100644 index 00000000..c8ce7d2e --- /dev/null +++ b/docs/superpowers/contracts/remove-wsg-grabber-2026-09-28.json @@ -0,0 +1,16 @@ +{ + "title": "Remove wsg_grabber from the monorepo after extracting it", + "objective": "wsg_grabber lives only in kuhyx/wsg-grabber (private, ~/src/wsg-grabber); nothing in testsAndMisc still carries or runs it, and main's CI is green.", + "acceptance_criteria": [ + "python_pkg/wsg_grabber/ no longer exists in the tree.", + "kuhyx/wsg-grabber holds the package's history (14 commits) plus the standalone setup, and its suite passes at 100% branch coverage.", + "/usr/local/bin/wsg-grabber runs from ~/src/wsg-grabber and `wsg-grabber stats` reads the live index.", + "AGENTS.md lists wsg-grabber among the extracted repos.", + "Pre-commit checks and dependency freshness are green on the pushed commit." + ], + "out_of_scope": [ + "The metadata, duplicate-detection and report features; they are built in the new repo.", + "Rewriting historical docs that mention the old path." + ], + "verifier": "ls python_pkg/wsg_grabber fails; git -C ~/src/wsg-grabber log --oneline | wc -l; cat /usr/local/bin/wsg-grabber; wsg-grabber stats; gh run list --branch main" +} diff --git a/docs/superpowers/evidence/remove-wsg-grabber-2026-09-28.json b/docs/superpowers/evidence/remove-wsg-grabber-2026-09-28.json new file mode 100644 index 00000000..657367f1 --- /dev/null +++ b/docs/superpowers/evidence/remove-wsg-grabber-2026-09-28.json @@ -0,0 +1,37 @@ +{ + "intent": "Remove python_pkg/wsg_grabber from the monorepo now that it lives in its own private repo, kuhyx/wsg-grabber, with its full history; and turn main's CI green again (hypothesis released 6.168.3).", + "scope": [ + "python_pkg/wsg_grabber/ deleted (package, tests, README, install.sh)", + "AGENTS.md (CLAUDE.md symlink): wsg-grabber added to the extracted-repos list", + "meta/requirements.txt: hypothesis 6.168.2 -> 6.168.3, the only item the Pre-commit checks and dependency freshness jobs reported behind on 33f0fee3", + "Non-goal: the historical mentions in docs/DOCS-python-split-recipes.md and docs/DOCS-restructure-live-state.md stay; they record what happened here" + ], + "changes": [ + "git filter-repo kept the 14 commits that touched python_pkg/wsg_grabber; the standalone setup is kuhyx/wsg-grabber 6b6d4b3", + "/usr/local/bin/wsg-grabber regenerated by ~/src/wsg-grabber/install.sh, so the launcher no longer depends on this repo" + ], + "verification": [ + { + "command": "~/src/wsg-grabber: .venv/bin/python -m pytest -p no:cacheprovider -q", + "result": "pass", + "evidence": "303 passed, TOTAL 1645 stmts / 342 branches, 100.00% coverage; pre-commit run --all-files clean there" + }, + { + "command": "vm run wsg 'sudo bash ~/install.sh' then on the host: sudo bash ~/src/wsg-grabber/install.sh && wsg-grabber stats", + "result": "pass", + "evidence": "vmbox: packages installed, launcher written with PYTHONPATH=/home/arch, --uninstall removed it. Host: launcher PYTHONPATH=/home/kuhy/src/wsg-grabber; stats reads the live index (26898 known, 706 kept, 200 passed, 25992 purged)" + }, + { + "command": "gh run view 36399404250 --log-failed", + "result": "pass", + "evidence": "the only freshness finding was pypi:hypothesis 6.168.2 -> 6.168.3 at meta/requirements.txt:42, bumped here" + } + ], + "risks": [ + "Anything outside git that pointed at ~/src/testsAndMisc/python_pkg/wsg_grabber would break; the only such artefact found was /usr/local/bin/wsg-grabber, already repointed." + ], + "rollback": [ + "git revert this commit to restore the package, then rerun its install.sh to point the launcher back here", + "Validate with `wsg-grabber stats`" + ] +} diff --git a/docs/superpowers/evidence/wire-shell-coverage-ratchet-2026-08-23.json b/docs/superpowers/evidence/wire-shell-coverage-ratchet-2026-08-23.json deleted file mode 100644 index d7e64759..00000000 --- a/docs/superpowers/evidence/wire-shell-coverage-ratchet-2026-08-23.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "intent": "Wire check_shell_coverage.sh into pre-commit so the shell ratchet cannot silently rot, and give the coverage tooling itself the suite it was missing.", - "scope": [ - ".pre-commit-config.yaml (new shell-coverage-ratchet hook)", - "meta/scripts/check_shell_coverage.sh (stale message referencing a removed constant)", - "meta/scripts/lib/tests/ (new suite: harness + 3 test files + run_all.sh)" - ], - "changes": [ - "Added the shell-coverage-ratchet hook, types: [shell], excluding third_party/. It was previously never wired, so a new untested shell library could land freely.", - "FIX: check_shell_coverage.sh line 177 interpolated ${COVERAGE_BAR}, which no longer exists after 578c4a6c, producing 'COVERAGE_BAR: unbound variable' on every failure path. The message now describes the presence bar it actually enforces.", - "New suite for the three meta/scripts/lib coverage-tooling libs, which are the one corner of the repo the jail cannot measure itself: 48 assertions over is_covered, build_jail, write_cases, build_mount_script, reconcile_pass_failures and write_case_runner.", - "Assertions on unexpanded ${BASH_SOURCE}/$1 literals compose the dollar sign from an exported DOLLAR constant rather than writing single-quoted literals, so the files are SC2016-clean with no suppressions." - ], - "verification": [ - { - "command": "bash meta/scripts/lib/tests/run_all.sh", - "result": "pass", - "evidence": "48 assertions across 3 files, 0 failed (5 + 26 + 17)" - }, - { - "command": "pre-commit run shell-coverage-ratchet --all-files", - "result": "pass", - "evidence": "Passed in 0.16s; check_shell_coverage.sh --all now reports 0 BLOCK entries, down from 3, and runs in 0.31s" - }, - { - "command": "Created an untested lib at mtk_root/lib/zz_probe.sh and ran the hook on it", - "result": "pass", - "evidence": "Hook failed with 'needs: linux_configuration/scripts/mtk_root/lib/tests/run_all.sh'. Probe file removed; the ratchet is not toothless." - }, - { - "command": "shellcheck -x and shfmt -l over the new files", - "result": "pass", - "evidence": "clean, no noqa/disable directives added" - } - ], - "risks": [ - "The hook now runs on every commit touching a shell file. Measured at 0.16s repo-wide, so the cost is negligible.", - "A lib added to a directory that has no suite is now blocked at commit time. That is the intended ratchet behaviour, and all 8 such directories are already allowlisted." - ], - "rollback": [ - "Remove the shell-coverage-ratchet block from .pre-commit-config.yaml; the gate returns to being unenforced.", - "After rollback, confirm `pre-commit run --all-files` no longer lists the hook." - ] -} diff --git a/meta/requirements.txt b/meta/requirements.txt index aa6568ca..27538359 100644 --- a/meta/requirements.txt +++ b/meta/requirements.txt @@ -39,7 +39,7 @@ gatelock @ git+https://github.com/kuhyx/utils@gatelock-v0.8.2#subdirectory=gatel genanki==0.13.1 geopandas==1.1.4 howlongtobeatpy==1.0.23 -hypothesis==6.168.2 +hypothesis==6.168.3 importlib-metadata==9.0.1 interrogate==1.7.0 # isort is deliberately not pinned: nothing here imports it (ruff's isort diff --git a/python_pkg/wsg_grabber/README.md b/python_pkg/wsg_grabber/README.md deleted file mode 100644 index 9e776c0f..00000000 --- a/python_pkg/wsg_grabber/README.md +++ /dev/null @@ -1,199 +0,0 @@ -# wsg_grabber - -Scrapes 4chan's /wsg/ board for videos, downloads every one it has not seen -before, and shows them one at a time for a keep/pass decision — mpv embedded in -a Tk window with the buttons directly underneath. - -```bash -PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber # review -PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber scrape # no UI -PYTHONPATH=~/src/testsAndMisc python3 -m python_pkg.wsg_grabber stats -``` - -`k` / `→` keeps, `j` / `space` / `←` passes, `u` / `Backspace` undoes, `q` / -`Esc` quits. The buttons do the same thing for the mouse. - -## Undo - -Misclicked? `u` takes the last verdict back: the file returns from the keep -folder or `trash/` to `incoming/`, the index row goes back to _ready_, the -counter is corrected, and that video is on screen again. Whatever was showing -when you hit undo goes to the front of the queue, so nothing is skipped. - -It **survives quitting** — the trail lives in the index, not in memory, so -reopening the reviewer can still undo what you decided yesterday. The status -bar shows how many are left (`u undoes (17)`), and the button greys out when -there is nothing to take back. Keeps are undoable for as long as the file is -still in any place a keep can end up (`housekeeping.locate_reviewed` looks in -this year's cloud folder, earlier years', `~/Downloads`, the old `keep/`, and -wherever the media sync swept it); passes are undoable while they are among -the newest `TRASH_RETAIN` (200). If a file has gone anyway, undo says so in -the log and drops that entry rather than failing on it repeatedly. - -## Storage - -``` -~/data/cloud/Media//wsg/ you kept these (falls back to ~/Downloads - on a machine without the cloud folder) -~/.local/share/wsg_grabber/ -├── index.db sqlite, remembers every file ever seen -├── incoming/ downloaded, awaiting your verdict -└── trash/ you passed on these — only the newest 200 passes stay -``` - -A keep goes where you actually look: the dufs cloud folder, so it shows up in -the gallery and on the phone. The first start after this change moved the old -`keep/` contents across (`_migrate_keep.py`). - -A pass goes into `trash/`, and `_prune.py` deletes everything there except the -200 most recent passes — at startup and after every pass. Their index rows -become _purged_: still known, so the file is never downloaded again, but no -longer undoable. Before this cap existed 23k files / 77 GB had piled up. -`files.apply_move` still refuses to overwrite an existing destination, so a -move can never destroy a video you already decided on. - -## Why it never shows you the same video twice - -Every 4chan post that carries a file also carries `md5` — the base64 MD5 of the -whole file, published by the API before you fetch anything. That is the primary -key of the `files` table, so: - -- a video you have already reviewed is never downloaded again, at a cost of - zero bytes; -- the same clip reposted in another thread collapses onto the existing row, - which on /wsg/ is the common case; -- a truncated or mangled download is caught, because the bytes are checked - against that same value before the file is offered for review. - -Verified against a live file during development: the API's -`Nz9OEKdMuMZEdYE6eLTKmA==` matched `base64(md5(bytes))` exactly. - -## Being a good citizen - -- One request per second to `a.4cdn.org`, which is what the API rules ask for. -- `If-Modified-Since` on every API request — thread bodies, `threads.json` and - `archive.json` alike — plus a `last_modified` comparison against the stored - value, so an unchanged thread costs a 304 and nothing else. A steady-state - rescan is a couple of conditional requests that return 304. -- `Accept-Encoding: identity` on both hosts, and hard caps on response and - download size. A compressed response is refused outright: a few hundred KB on - the wire can decode to gigabytes, and the md5 check does not help when the - same party publishes both the bytes and the digest. -- One long-lived `requests.Session`. This is required rather than merely tidy: - `i.4cdn.org` answers with 429 under load and sets `__cf_bm` / `_cfuvid` - cookies that have to persist across requests. -- 429 and 5xx back off exponentially, honouring `Retry-After`. - -## Downloading is prioritised over indexing - -While anything is queued the worker downloads rather than walking more threads. -That is deliberate: it is what puts a playable video on screen seconds after -launch instead of after the whole board has been catalogued. The consequence is -that `stats` shows the known-file count climbing in bursts rather than smoothly -— the board is fully indexed once downloads drain. A first full run is roughly -9,000 files and 20–25 GB, so expect a couple of hours; it is resumable and -restarting picks up mid-file via HTTP `Range`. - -## Logs - -Every run writes one JSON object per line to -`~/.local/share/wsg_grabber/logs/session-