diff --git a/TODO-refactor-remaining.md b/TODO-refactor-remaining.md index 6632e906..a6d1ed04 100644 --- a/TODO-refactor-remaining.md +++ b/TODO-refactor-remaining.md @@ -2,86 +2,132 @@ REMOVE ME AFTER FINISH -What is done, what is not, and the real numbers. Companion to -`TODO-file-length-250.md`, which parks one specific blocking decision. +What is done, what is not, and the real numbers, re-measured 2026-08-28 after +the lint campaign started. Companion to `TODO-file-length-250.md`, which parks +one specific blocking decision. ## Done | Area | State | |---|---| -| Vendored manifest | `gates/vendored.txt`, mirrored into `~/utils` (`kuhyx/utils@a4a3e0a`) | +| Vendored manifest | `gates/vendored.txt`, 8 entries, mirrored into `~/utils` | | Build artifacts | 189 untracked, `.gitignore` closed, 0 tracked-but-ignored files | | no-binaries gate | Path-scoped `.binary-allowlist`; verified blocking AND passing | | Markdown | 4 namespaces, all under 250 lines, `CLAUDE.md` written | | pre-commit | 12 hooks, green on all files | | pre-push | `pre-commit install --hook-type pre-push` via `install.sh` | -| GitHub CI | `pre-commit` + `md-naming`, both green. Actions had been DISABLED at repo level | -| run.sh | Root harness + 44 course runners over one shared library | -| Runnability | 38 runnable / 5 no-code / 1 blocked, recorded in `DOCS-runnability.md` | +| GitHub CI | `pre-commit` + `md-naming`, both green | +| run.sh | Root harness + per-course runners over one shared library | +| `pyproject.toml` | `select = ["ALL"]`, mypy strict, coverage config. **No hook wired up yet** | +| LaTeX 250-line cap | 19 reports split, every rebuilt PDF verified identical. 1 left | +| ZPOB | Fixed: builds with `--no-bibtex`, exits 0 | -## Not done, with honest sizes +## The lint campaign -### 1. Python linting — 9,281 ruff findings +`select = ["ALL"]`, and by kuhy's decision on 2026-08-28 there are **no +per-file-ignores at all** -- not even the `S101` block for tests that ruff's +own docs and the rest of this fleet carry. A test asserts by raising. -Measured with the fleet config (`select = ["ALL"]`, the five standard ignores): +| | ruff findings | +|---|---| +| Baseline, before any of this | 9,281 | +| After the format + safe-fix pass and excluding `TRAK/sightpy` | 3,686 | +| **Now** | **1,183** | -| Rule | Count | Nature | +Courses at zero: `WDWR`, `SPD`, `MOM`, `ECRYPT`, `ECRYPT_PROJECT`, `ECOTE`, +`NLP`, `EARIN`, `PORR`. What is left, by course: + +| Course | ruff | Can it be verified by running? | |---|---|---| -| `Q000` bad-quotes | 2050 | auto-fixable | -| `ANN001`/`ANN201` missing annotations | 1671 | **manual** | -| `E501` line-too-long | 628 | mostly manual | -| `T201` print | 316 | needs a decision per script | -| `D1xx` missing docstrings | ~600 | **manual** | -| `W291`/`W293` whitespace | 431 | auto-fixable | -| everything else | ~3600 | mixed | +| `TRAK` | 531 | **Yes** -- `run` ray-traces cornell_box in 37s. `sightpy/` is vendored and excluded; this is kuhy's own code | +| `twm_4` | 228 | No: `jupyter` on PATH is a dead pipx shim. `pip install jupyter` in a venv would fix that | +| `ERSMS-project` | 156 | No: docker-compose course | +| `PSD` | 152 | No: `run.sh` predates the harness and installs packages with sudo | +| `PBAD` | 115 | No: same dead jupyter shim | -Roughly 3,000 are auto-fixable and ~6,000 are hand work. The auto-fixable pass -should be its own commit so the manual diff stays reviewable. +**Before running the notebook courses, fix `scripts/course_run.sh` first.** +Its notebook branch is `jupyter nbconvert --execute --inplace`, and all four +notebooks are tracked WITH their outputs -- that would overwrite a committed +deliverable. This exact bug has already been found twice (TRAK's +`outputs/output.png`, SPD's `results_*.png`); execute to a temp path instead. -**Blocked on:** nothing. Start with `ruff check --fix` on the safe rules, then -add `pyproject.toml` and the ruff/mypy hooks in the same commit as the last fix -(a gate must not land before its subject is clean). +## mypy has NOT been run against any of this -### 2. Coverage — 19 tests for ~28k lines +`pyproject.toml` sets `strict = true` and `CLAUDE.md` says the repo runs mypy, +but mypy is not installed on this machine (pip is externally managed; use a +venv) and no commit so far has type-checked. **"Zero findings" in the commit +messages means zero RUFF findings.** Measured after installing it: -`gates/coverage.tsv` (per-directory floors, `target` always 100, floors may -only rise) is designed but not written. Nothing exists for the non-Python -languages yet. This is the largest remaining item by a wide margin and is -measured in weeks. +| Course | mypy --strict errors | +|---|---| +| `WDWR`, `SPD`, `MOM`, `ECRYPT_PROJECT` | 0 | +| `ECOTE` | 14 | +| `NLP` | 29 | +| `EARIN/lab1` alone | 46 | +| `TRAK` | 1 | +| `PSD/zin1` | 5 | + +Two structural notes for whoever runs it next: + +- It cannot be pointed at the whole repo. There are many `main.py` files and + mypy stops at `Duplicate module named "main"`. Run it per directory. +- Most errors are `Returning Any` from untyped third-party calls (openai, + pandas) and genuinely-wrong unions this pass introduced by hand, e.g. + `str | bool` returns in `gpt_chunks.reformat`. They are real. + +## Still open + +### 1. The 250-line cap: 51 files + +| Type | Count | Note | +|---|---|---| +| `.txt` | 27 | Blocked -- see `TODO-file-length-250.md` | +| `.py` | 11 | Several GREW during the lint pass, because docstrings and annotations are lines: `NLP/gpt_chunks.py` 911 -> 973, `EARIN/lab2/main.py` 581 -> 647, `PORR/.../linear_algebra_utils.py` 422 -> 553. Split them AFTER the lint work, not during | +| `.cpp` | 9 | | +| `.c` | 4 | | +| `.java` | 2 | | +| `.tex` | 1 | `ELAC/.../03-building-nfa-...tex`, 418 lines of consecutive tikz figures with no subsection to split on | + +### 2. Coverage + +`gates/coverage.tsv` is designed but not written, and there are 125 tests for +~28k lines. The largest remaining item by a wide margin. ### 3. Dependency freshness 9 `requirements.txt`, 5 `package.json`, 7 `pom.xml`, 1 `.csproj`, none pinned. -**The upgrade and the gate must land in the SAME commit** or the repo is -instantly and permanently red -- see the warning in -`~/utils/scripts/install_dependency_freshness_gate.sh`. +**The upgrade and the gate must land in the SAME commit.** -### 4. The 250-line cap +### 4. File-rewriting hooks -73 files still over. Breakdown: 27 `.txt` (blocked -- see -`TODO-file-length-250.md`), 19 `.tex`, 10 `.py`, 9 `.cpp`, 4 `.c`, 2 `.java`. -The `.tex` splits are `\input{}` per section, and the acceptance test is that -the rebuilt PDF is unchanged. **Do not split the source files before their -tests exist** -- splitting untested code is the highest-risk action here. +`trailing-whitespace` and `end-of-file-fixer` are deliberately absent; they +want their own reviewed commit. -### 5. File-rewriting hooks +### 5. Three checks that must be run on every future pass -`trailing-whitespace` and `end-of-file-fixer` are deliberately absent. They -will touch hundreds of submitted coursework files, so they want their own -reviewed commit rather than being buried in a config change. +`scripts/check_refactor_safety.py` exists because ruff and a smoke run BOTH +missed three regressions this campaign introduced. Run it before and after any +automated rewrite: -### 6. Two courses still fail `run` +```bash +cp -r /tmp/before +# ...do the pass... +./scripts/check_refactor_safety.py --signatures /tmp/before +./scripts/check_refactor_safety.py --logging --print-keywords +``` -- `Programming/PORR` -- its pytest suite solves systems up to 10000x10000 and - runs for many minutes at full CPU. Under the 2G/2-core cap it will be - OOM-killed rather than finish. It needs either a smaller default parameter - set for `run`, or to be marked as a deliberately heavy target. -- `Programming/ZPOB` -- latexmk exits 12 from bibtex even though `esej.pdf` is - produced. Probably a missing/!stale .bib entry; needs a clean-room rebuild - to see the first real error rather than the cached one. +And run mypy per directory afterwards: it found 12 latent TypeErrors in EARIN +that ruff could not see -- call sites still passing positionally to arguments +the pass had made keyword-only. + +### 6. Disk + +`Programming/PORR/code/` holds 3.8 GB of generated .npz caches (poli3 alone is +2.3 GB). All gitignored, all regenerable, none of it needed unless the full +benchmark is re-run. ## Standing rule -A gate lands only once the thing it gates is clean, so -`pre-commit run --all-files` is green at every commit. Every hook still missing -from `.pre-commit-config.yaml` is listed there with what blocks it. +A gate lands only once the thing it gates is clean, so `pre-commit run +--all-files` is green at every commit. Every hook still missing from +`.pre-commit-config.yaml` is listed there with what blocks it.