mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-11 14:07:49 +01:00
kuhyx
5ffd035c21
Cache read is charged per API round-trip on the whole conversation so far, so a long session's cost is driven by TURN COUNT, not message length. One measured session spent 267M cache-read against 856k output — a ratio of 290:1. The report priced tokens but never counted the turns spending them. Across the 263 stored transcripts this now shows 33,508 tool turns at an 8% batching rate, of which 4,437 (13%) decided nothing: polling a counter, running a linter, checking whether a push landed. At 606k average context re-read per turn that is ~2.7B cache-read tokens on turns a script could have taken. - turns.py classifies a command as poll / lint_test / vcs_check / substantive, deliberately conservative: state-changing commands (git commit, push) count as substantive even though they are scriptable, so the reported waste is a floor. A turn is only mechanical when EVERY command in it is. - Batching is grouped by requestId, not by transcript line: one API response can span several lines, and counting per line reports every turn as single-tool (0% batching where the real rate is 12%). - records.py gains TurnEfficiency (schema v3, additive — old records read as zeros) with derived rates written into to_dict so readers need not recompute. - report.py gains a "Turn efficiency" section pricing each mechanical kind. - _TurnTracker owns both the totals and the open turn, after pylint flagged _Accumulator over its attribute budget; _Accumulator.turns is now a property, so every existing caller is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
43 lines
3.4 KiB
JSON
43 lines
3.4 KiB
JSON
{
|
|
"intent": "Show, per session, how many API round-trips were spent on work that decided nothing, so the waste can be scripted away instead of guessed at.",
|
|
"scope": [
|
|
"python_pkg/session_autopsy/turns.py (new classifier)",
|
|
"python_pkg/session_autopsy/{parse,records,report}.py",
|
|
"python_pkg/session_autopsy/tests/{test_turns,test_parse,test_report}.py",
|
|
"Non-goal: acting on the findings, or changing existing detector scoring"
|
|
],
|
|
"changes": [
|
|
"turns.py classifies a Bash command as poll / lint_test / vcs_check / substantive. Deliberately conservative: state-changing commands (git commit, git push) are substantive even though they are scriptable, and a turn is mechanical only when every command in it is \u2014 so the reported waste is a floor, not a guess.",
|
|
"parse.py groups tool calls by requestId rather than by transcript line. One API response can span several lines, and per-line counting reported 0% batching where the true rate is 12%.",
|
|
"records.py adds TurnEfficiency (schema v3, additive: pre-v3 records read as zeros) with derived batching/mechanical rates written into to_dict so consumers need not recompute them.",
|
|
"report.py adds a Turn efficiency section pricing each mechanical kind at the measured average context re-read per turn.",
|
|
"Turn state and totals were folded into a _TurnTracker class after pylint flagged _Accumulator at 15/12 instance attributes. The budget allowed exactly one new attribute, which forced the right shape: the open turn exists only to be folded into the totals, so they belong in one object. _Accumulator.turns is now a property, keeping every existing caller and test unchanged."
|
|
],
|
|
"verification": [
|
|
{
|
|
"command": "pytest python_pkg/session_autopsy/tests --cov=python_pkg.session_autopsy",
|
|
"result": "pass",
|
|
"evidence": "112 passed; every module in the package at 100% statement and branch coverage, including the new turns.py (25 stmts / 12 branches)."
|
|
},
|
|
{
|
|
"command": "ruff check && ruff format --check && pylint && mypy",
|
|
"result": "pass",
|
|
"evidence": "ruff: All checks passed, formatted. pylint 10.00/10. mypy clean. 112 tests, every module at 100% statement and branch coverage."
|
|
},
|
|
{
|
|
"command": "python -m python_pkg.session_autopsy scan --force && report",
|
|
"result": "pass",
|
|
"evidence": "263 transcripts analyzed in 1.8s. Report: 33,508 tool turns carrying 37,411 calls, batched 2,595 (8%), avg context re-read 606k/turn; mechanical 4,437 turns (13%) ~= 2.7B cache-read tokens (poll 891, lint/test 2,044, vcs 1,502)."
|
|
}
|
|
],
|
|
"risks": [
|
|
"The classifier is regex-based, so an unusual command shape can be misfiled. Mitigated by the conservative rule (mixed turns count as substantive), which biases the number down rather than up.",
|
|
"Grouping by requestId depends on that field being present; lines without one fall back to the message id, and turns with neither collapse together. Only affects batching precision, not the mechanical counts.",
|
|
"Schema v3 records are read by older code as unknown keys, which record_from_dict already ignores; no migration is needed."
|
|
],
|
|
"rollback": [
|
|
"git revert the commit; TurnEfficiency is additive so stored v3 records remain readable by the reverted code, which ignores the extra key.",
|
|
"Re-run scan --force && report and confirm REPORT.md renders without the Turn efficiency section."
|
|
]
|
|
}
|