diff --git a/.file-length-exempt b/.file-length-exempt new file mode 100644 index 00000000..c81c04f2 --- /dev/null +++ b/.file-length-exempt @@ -0,0 +1,12 @@ +# Data files the 250-line cap reads as prose. In every one a line is a record, +# so splitting would corrupt it. Authored code beside them stays gated. +Programming/NLP/test_goldStandard/*/STSint.testinput.*.txt # official SemEval 2016 Task 2 iSTS test set, listed in test_goldStandard/00-README.txt +Programming/NLP/test_goldStandard/*/*-chunks-gpt-*.txt # recorded output of paid non-deterministic GPT runs (gpt_chunks.py); the experimental record +Programming/NLP/alignments_unformatted_*.txt # NLP pipeline stage data, one alignment per line, read by create_alignments.py and format_alignments.py +Programming/NLP/output.txt # NLP pipeline stage data, one record per line +Programming/NLP/reformated.txt # NLP pipeline stage data, one record per line +Programming/WDWR/projekt/old_code/data10000.txt # numeric matrix, 10000 scenario rows, solver input +Programming/WDWR/projekt/R/scenarios.txt # generated scenario vectors, one per line +Programming/WDWR/projekt/wdwr/solutions1.txt # CPLEX MILP solver log +Programming/WDWR/projekt/wdwr/KRZYSZTOF_RUDNICKI_307585_WDWR_PROJEKT/scenarios.txt # submitted copy of scenarios.txt, must stay byte-identical +Programming/WDWR/projekt/wdwr/KRZYSZTOF_RUDNICKI_307585_WDWR_PROJEKT/solutions1.txt # submitted copy of the CPLEX solver log, must stay byte-identical diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index d3ef5a04..70df871f 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -4,7 +4,7 @@ # Ordering rule for this repo: a hook lands only once the thing it gates is # already clean, so `pre-commit run --all-files` is green at every commit. # Hooks still to come, each blocked on its own cleanup: -# - file-length blocked on the .tex/code splits and TODO-file-length-250.md +# - file-length blocked on the .tex/code splits (27 files, TODO-refactor-remaining.md) # - mypy the six third-party-typing errors are GONE (ERSMS moved # to a typed DeclarativeBase; PSD has a scoped override). # Still blocked on ECOTE/NLP/EARIN, and on the fact that diff --git a/TODO-file-length-250.md b/TODO-file-length-250.md deleted file mode 100644 index 7ba02a68..00000000 --- a/TODO-file-length-250.md +++ /dev/null @@ -1,106 +0,0 @@ -# TODO: the 27 data `.txt` files the 250-line cap cannot classify - -REMOVE ME AFTER FINISH - -The 250-line cap applies to prose as well as code, and the shared gate treats -`.txt` as prose unless its mean line is under 25 characters (a wordlist). These -27 files are **data** with long lines, so the heuristic reads them as prose and -demands they be split. Splitting any of them corrupts it: in every one, a line -is a record. - -Nothing here blocks the refactor. It blocks exactly one step — installing the -`file-length` pre-commit hook and workflow — so it must be settled before that -lands, and not before. - -Run `bash ~/src/utils/scripts/check_file_length.sh --all` to see the current list. - -## Why the existing mechanism does not cover them - -`VENDORED_SUBPATHS` in `~/src/utils/file_length/_tables.py` matches **directory -prefixes** (`/{repo}/{sub}/`). That is enough for the 7 entries already in -`gates/vendored.txt`, and it is not enough here: - -- `Programming/NLP/test_goldStandard/images/` holds the official SemEval corpus - **and** kuhy's own GPT outputs, side by side. -- `Programming/WDWR/projekt/R/` holds `scenarios.txt` (data) **and** `model.R` - (authored, 464 lines, must stay gated and covered). - -A directory entry would exempt authored work in both cases. Per-file `# noqa` -style suppression is banned outright. So this needs a decision, not a workaround. - -## The four categories - -### 1. Official SemEval package — 12 files, genuinely third-party - -The `STSint.testinput.{headlines,images,answers-students}.{sent1,sent2}[.chunk].txt` -files. Verified against the package's own manifest in -`Programming/NLP/test_goldStandard/00-README.txt`, which enumerates exactly -these, plus `evalF1.pl` and `wellformed.pl`, as the contents of the -"STS 2016 Task 2 Interpretable Semantic Textual Similarity TEST DATASET" -(authors: Eneko Agirre, Montse Maritxalar et al.). - -Unambiguously vendored. The only question is *how* to express it. - -### 2. kuhy's GPT outputs living inside the corpus directory — 6 files - -``` -Programming/NLP/test_goldStandard/headlines/headlines-chunks-gpt-{one,two}.txt -Programming/NLP/test_goldStandard/images/images-chunks-gpt-{one,two}.txt -Programming/NLP/test_goldStandard/student/students-chunks-gpt-{one,two}.txt -``` - -**Not** in the package manifest — these are output from `gpt_chunks.py`, written -next to the corpus they were derived from. Regenerable in principle, but only by -re-running non-deterministic paid GPT calls, so untracking them destroys the -experimental record rather than saving space. - -### 3. NLP pipeline data — 4 files - -``` -Programming/NLP/alignments_unformatted_headlines.txt 375 lines -Programming/NLP/alignments_unformatted_student.txt 344 -Programming/NLP/output.txt 348 -Programming/NLP/reformated.txt 348 -``` - -Intermediate stage data. `alignments_unformatted_headlines.txt` is read as input -by `create_alignments.py:6` and `format_alignments.py:7`, so relocating these -costs exactly two path constants — the cheapest category to fix. - -### 4. WDWR solver data and output — 5 files - -``` -Programming/WDWR/projekt/old_code/data10000.txt 10000 lines (numeric matrix, solver input) -Programming/WDWR/projekt/R/scenarios.txt 999 (generated scenario vectors) -Programming/WDWR/projekt/wdwr/solutions1.txt 3038 (CPLEX MILP solver log) -Programming/WDWR/projekt/wdwr/KRZYSZTOF_.../scenarios.txt 999 (submission copy) -Programming/WDWR/projekt/wdwr/KRZYSZTOF_.../solutions1.txt 3038 (submission copy) -``` - -Two of these live under `KRZYSZTOF_RUDNICKI_307585_WDWR_PROJEKT/`, the submitted -deliverable. Decision 6 of the approved plan says leave semantic duplicates -alone, so **relocation is not available for those two** — moving a file inside a -submission alters what was submitted. - -## Options - -**A. Extend the shared manifest to support file paths.** Teach -`~/src/utils/file_length/exemptions.py:is_vendored` to read `gates/vendored.txt` at -the repo root, supporting exact files as well as directories. This is the design -originally chosen (one root manifest consumed by every gate) and is the only -option that handles all four categories. -Cost: `file_length` currently has **zero tests** and no coverage job in -`.github/workflows/shared-gates-tests.yml`. Adding code to a gate 16 repos -depend on means authoring its test suite first. - -**B. Relocate categories 2–4 into `data/` subdirectories** and add `data` to the -manifest per course dir. Cheap for categories 2 and 3. Cannot touch the two -WDWR submission-copy files. Does not address category 1. - -**C. Accept a documented residual** — install the file-length gate with these 27 -paths listed here, and treat the list as the record. Honest, but it is an -allowlist by another name, which the fleet rules reject. - -Recommendation: **A**, taken as its own `~/src/utils` change with its own tests and -CI job, sequenced before Milestone 2 step 4. B as a fallback for categories 2–3 -if A is judged too large, leaving categories 1 and 4 to a narrower version of A. diff --git a/TODO-refactor-remaining.md b/TODO-refactor-remaining.md index e17189f2..d61edc0a 100644 --- a/TODO-refactor-remaining.md +++ b/TODO-refactor-remaining.md @@ -3,8 +3,8 @@ REMOVE ME AFTER FINISH What is done, what is not, and the real numbers, re-measured 2026-08-28 after -the lint campaign started. Companion to `TODO-file-length-250.md`, which parks -one specific blocking decision. +the lint campaign started. The 27 data `.txt` files the cap misread as prose +are exempted per file in `.file-length-exempt` (reason per entry, 2026-11-01). ## Done @@ -150,12 +150,11 @@ fell out is a course that had never been run by the harness at all: ## Still open -### 1. The 250-line cap: 50 files +### 1. The 250-line cap: 27 files | Type | Count | Note | |---|---|---| -| `.txt` | 27 | Blocked -- see `TODO-file-length-250.md` | -| `.py` | 10 | Several GREW during the lint pass, because docstrings and annotations are lines: `NLP/gpt_chunks.py` 911 -> 973, `EARIN/lab2/main.py` 581 -> 647, `PORR/.../linear_algebra_utils.py` 422 -> 553. Split them AFTER the lint work, not during. TRAK is already clear: its three over-cap files were split in the same pass that linted them, each verified by a byte-identical render. `twm_4/TWM_KerasIntro.py` joined the list at 407 lines and should NOT be split: it is an nbconvert export, and a partial one | +| `.py` | 11 | Several GREW during the lint pass, because docstrings and annotations are lines: `NLP/gpt_chunks.py` 911 -> 973, `EARIN/lab2/main.py` 581 -> 647, `PORR/.../linear_algebra_utils.py` 422 -> 553. Split them AFTER the lint work, not during. TRAK is already clear: its three over-cap files were split in the same pass that linted them, each verified by a byte-identical render. `twm_4/TWM_KerasIntro.py` joined the list at 407 lines and should NOT be split: it is an nbconvert export, and a partial one | | `.cpp` | 9 | | | `.c` | 4 | | | `.java` | 2 | |