mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 20:34:02 +01:00
kuhyx
5ffd035c21
Cache read is charged per API round-trip on the whole conversation so far, so a long session's cost is driven by TURN COUNT, not message length. One measured session spent 267M cache-read against 856k output — a ratio of 290:1. The report priced tokens but never counted the turns spending them. Across the 263 stored transcripts this now shows 33,508 tool turns at an 8% batching rate, of which 4,437 (13%) decided nothing: polling a counter, running a linter, checking whether a push landed. At 606k average context re-read per turn that is ~2.7B cache-read tokens on turns a script could have taken. - turns.py classifies a command as poll / lint_test / vcs_check / substantive, deliberately conservative: state-changing commands (git commit, push) count as substantive even though they are scriptable, so the reported waste is a floor. A turn is only mechanical when EVERY command in it is. - Batching is grouped by requestId, not by transcript line: one API response can span several lines, and counting per line reports every turn as single-tool (0% batching where the real rate is 12%). - records.py gains TurnEfficiency (schema v3, additive — old records read as zeros) with derived rates written into to_dict so readers need not recompute. - report.py gains a "Turn efficiency" section pricing each mechanical kind. - _TurnTracker owns both the totals and the open turn, after pylint flagged _Accumulator over its attribute budget; _Accumulator.turns is now a property, so every existing caller is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18 lines
1.7 KiB
JSON
18 lines
1.7 KiB
JSON
{
|
|
"title": "Measure turn efficiency in session-autopsy",
|
|
"objective": "Session-autopsy priced tokens but never counted the API round-trips spending them. Cache read is charged per round-trip on the whole conversation so far, so a long session's cost is driven by turn count rather than message length — one measured session spent 267M cache-read against 856k output. Success is a report section that counts tool turns, their batching rate, and the subset that decided nothing (polling, lint/test, read-only VCS checks), priced in cache-read tokens.",
|
|
"acceptance_criteria": [
|
|
"A command classifier labels Bash commands poll / lint_test / vcs_check / substantive, and a turn is only mechanical when every command in it is",
|
|
"Batching is grouped by requestId, because one API response can span several transcript lines and per-line counting reports every turn as single-tool",
|
|
"SessionRecord carries TurnEfficiency; the schema bump is additive so pre-v3 records deserialize as zeros rather than failing",
|
|
"REPORT.md gains a Turn efficiency section pricing each mechanical kind at the measured average context re-read",
|
|
"100% branch coverage across the package, ruff + pylint + mypy clean, full pre-commit gate passes with no --no-verify"
|
|
],
|
|
"out_of_scope": [
|
|
"Changing how any existing detector or candidate scores",
|
|
"Acting on the findings (scripting the polling loops) — this contract only measures",
|
|
"Classifying non-Bash tools; a turn that only ran Edit/Read counts as substantive"
|
|
],
|
|
"verifier": "pre-commit run --files <changed>; pytest --cov=python_pkg.session_autopsy (100% gate); python -m python_pkg.session_autopsy scan --force && report, checked against the real 263-transcript store"
|
|
}
|