mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 19:07:04 +01:00
Krzysztof kuhy Rudnicki
c5cd63fb80
Some checks are pending
Counting transcript records reports 1.00 tool calls per message no matter what the model did: the harness writes one JSONL record per tool_use block, all sharing message.id. A hand-rolled per-record count read the real 1.9% batching rate as 0.0%, which nearly shipped as a headline finding. Group by message.id instead, and report it, so the number comes from a tested code path with the window handled by find_transcripts rather than an ad-hoc timestamp. Three snapshot fields give next week's run a baseline. Axes was at the 12-attribute cap, so the counters live in their own Batching dataclass with the two ratios as properties. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NiSSDu4gKbp9GGxTTtXvYW
17 lines
1.2 KiB
JSON
17 lines
1.2 KiB
JSON
{
|
|
"title": "Measure tool-call batching per API message in token_audit",
|
|
"objective": "A per-record count of tool calls reports 1.00 calls/message regardless of actual behaviour, because the harness splits one API message into one record per tool_use block. Grouping by message.id makes the real rate (1.9%) observable, and persisting it in the snapshot lets the next weekly run adjudicate whether batching improved without any hand-rolled analysis.",
|
|
"acceptance_criteria": [
|
|
"Batching is grouped by message.id, not by transcript record",
|
|
"The markdown report shows batched share and calls per message",
|
|
"weekly.json carries tool_messages, batched_messages and batched_calls",
|
|
"A window with no tool calls does not divide by zero",
|
|
"100% branch coverage retained and no lint suppressions introduced"
|
|
],
|
|
"out_of_scope": [
|
|
"Changing reconciled totals, cost weights, or the mtime-based window selection",
|
|
"Acting on the measurement (hooks/nudges live in ~/.claude, not this repo)"
|
|
],
|
|
"verifier": "python3 -m pytest python_pkg/token_audit/tests; python3 -m coverage report (100%); pre-commit run --files <changed>; python3 -m python_pkg.token_audit --days 7 --no-write"
|
|
}
|