mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-10 20:27:02 +01:00
docs/agent-harness-evaluation-2026-08-18.md: fix prettier formatting
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011w2Pj4Ek3ZNMQVYcFHgXDY
This commit is contained in:
parent
5b5fb23a19
commit
912fbe4f1c
@ -9,12 +9,12 @@ harness internals + self-hosted infra. Every claim below cites the specific memo
|
||||
|
||||
## Baseline, stated plainly
|
||||
|
||||
You are not a harness *shopper* — you are a harness *builder*. The existing `~/.claude/` setup has 17 hooks across 6
|
||||
You are not a harness _shopper_ — you are a harness _builder_. The existing `~/.claude/` setup has 17 hooks across 6
|
||||
lifecycle events, 21 support scripts, 27 project skills + 13 process-methodology skills (`~/.agents/skills`), a
|
||||
background daemon, and a session-autopsy pipeline that mines your own transcripts (445 sessions, 153,946 messages)
|
||||
to auto-detect automation candidates and compile them into deterministic scripts — two already done (`finish`:
|
||||
-57% tokens/call, `phone-deploy`: -40%). You independently arrived at the exact thesis of Adam Jacob's token-spend
|
||||
piece you've already encoded as a standing rule (`rules/token-spend.instructions.md`) *before* evaluating this list.
|
||||
piece you've already encoded as a standing rule (`rules/token-spend.instructions.md`) _before_ evaluating this list.
|
||||
That reframes the right question from "which harness should I adopt" to "does anything on this list do something my
|
||||
current setup doesn't, better than I could bolt on myself."
|
||||
|
||||
@ -25,29 +25,33 @@ current setup doesn't, better than I could bolt on myself."
|
||||
Backed by repeated, concrete evidence across multiple sessions/repos.
|
||||
|
||||
### MCP Inspector (mcp, typescript, 10.6k★)
|
||||
|
||||
You author and maintain **7+ MCP servers** (`i3wm-mcp`, `yay-mcp`, `aseprite-mcp`, `reaper-mcp`, `opengameart-mcp`,
|
||||
`freesound-mcp`, plus 23 more under `~/mcp-servers/servers/`) per `project-mcp-server-fleet.md` and the deployment
|
||||
skill `project-mcp-per-repo-pattern.md`. Nothing in your memory or tooling survey shows a dedicated tool/resource/prompt
|
||||
debugger for the servers you *build* — you've been testing them indirectly through Claude Code sessions themselves.
|
||||
debugger for the servers you _build_ — you've been testing them indirectly through Claude Code sessions themselves.
|
||||
MCP Inspector is a GUI built exactly for this gap: exercising tools/resources/prompts in isolation before wiring a new
|
||||
server into a real session. Direct, unambiguous fit for your own MCP-authorship workflow.
|
||||
|
||||
### cocoindex-code (mcp, cli, tree-sitter/AST, 2.6k★)
|
||||
|
||||
Your token-spend discipline is measured, not aspirational: `err-549ef470` ("ALREADY IN CONTEXT" read-hook error) fired
|
||||
**870 times across 190 sessions** — the single most common error in your corpus — and 16% of all tool turns (11,532 of
|
||||
73,904) are mechanical, non-decision reads/greps. cocoindex-code replaces grep-and-re-read-whole-file with AST-aware
|
||||
semantic lookup, attacking exactly this measured cost center. This is a closer match to your actual bottleneck than
|
||||
generic "compression" tools (see Headroom/context-mode rejection below) because it targets the *read* side specifically,
|
||||
generic "compression" tools (see Headroom/context-mode rejection below) because it targets the _read_ side specifically,
|
||||
which your own hook (`track_reads_pretool.sh`, since disabled) was already trying to police mechanically.
|
||||
|
||||
### Docker MCP Gateway (mcp, sandbox, cli, 1.5k★)
|
||||
|
||||
You run 23+ Dockerfiles for self-hosted MCP server containers (`~/mcp-servers/servers/*`) per the repo survey, on top
|
||||
of an already-Docker-heavy self-hosted stack (Gitea, SearXNG, Syncyomi, Joplin Server, all `docker-compose.yml`-based).
|
||||
A container-aware MCP gateway is a direct fit for consolidating discovery/auth across a fleet this size, rather than
|
||||
each server being wired in ad hoc. Speculative on the *specific* tool, high-confidence on the *need* — you already
|
||||
each server being wired in ad hoc. Speculative on the _specific_ tool, high-confidence on the _need_ — you already
|
||||
have the infrastructure pattern this tool assumes.
|
||||
|
||||
### agents.md (ide, typescript, 23.7k★) — as a *format*, not a new tool
|
||||
### agents.md (ide, typescript, 23.7k★) — as a _format_, not a new tool
|
||||
|
||||
You've independently built the same thing at larger scale: `CLAUDE.md` + recursively-loaded `rules/` + `memories/` +
|
||||
skills, already proven to load automatically and tested with your own canary methodology
|
||||
(`reference-claude-context-loading.md`). `agents.md`'s pitch (nested per-directory briefings, load-what's-relevant) is
|
||||
@ -64,34 +68,38 @@ Not "adopt this tool" — "the format you're missing is this one, for the repo-s
|
||||
Plausible fit, but the evidence is thinner or the tool may just be a worse version of what you have.
|
||||
|
||||
### Composio (sandbox, tool-discovery, python/typescript, 29.7k★)
|
||||
|
||||
You've hand-rolled the "toolkit + OAuth" pattern per-app already: `reference-oauth-connect-github-pattern.md`
|
||||
(device-flow buttons registered per-repo under your own GitHub account) and Firebase RTDB credentials at
|
||||
`~/.config/crdt-sync/`. Composio centralizes 1,000+ toolkit integrations with auth handled once. Thin evidence because
|
||||
your current per-app pattern is deliberate (`project-oauth-connect-github-pattern.md` reads as a chosen design, not a
|
||||
pain point) — no session shows friction from *lacking* a centralized toolkit layer, only evidence that you'd need one
|
||||
pain point) — no session shows friction from _lacking_ a centralized toolkit layer, only evidence that you'd need one
|
||||
if you started integrating third-party SaaS tools at volume, which you haven't been doing.
|
||||
|
||||
### planning-with-files / get-shit-done (memory / cli-python, skill packs)
|
||||
|
||||
Your `~/.agents/skills` tree already has `writing-plans`, `executing-plans`, and `subagent-driven-development` — a
|
||||
self-built process-methodology layer covering similar ground (crash-proof planning, session recovery). Also
|
||||
`feedback-split-big-plans-into-session-prompts.md` documents you independently arrived at "one self-contained prompt
|
||||
per session, not one big plan" for the same reason these tools exist (context rot avoidance in `get-shit-done`'s
|
||||
pitch). Speculative because I can't confirm your existing skills' *content* matches these tools' specific mechanics
|
||||
pitch). Speculative because I can't confirm your existing skills' _content_ matches these tools' specific mechanics
|
||||
line-for-line — worth a side-by-side read, not a blind swap.
|
||||
|
||||
### Vibe-kanban / Symphony / AgentBox (multi-agent fleet managers)
|
||||
|
||||
You already run background forks/subagents heavily (session-autopsy shows this as a standard pattern) and have
|
||||
`~/.agents/skills/dispatching-parallel-agents`. A fleet-queue UI for running many coding-agent tasks at once is
|
||||
plausible if your parallel-agent volume grows, but no memory file documents friction from *managing* concurrent agent
|
||||
plausible if your parallel-agent volume grows, but no memory file documents friction from _managing_ concurrent agent
|
||||
runs — the one documented multi-agent friction (`feedback-check-for-concurrent-agent-work.md`, a duplicate-build
|
||||
incident) was about **coordination visibility** (not knowing another session had already built something), which a
|
||||
fleet manager doesn't obviously solve — that's a `git status`-before-building discipline problem, already fixed as a
|
||||
standing rule, not a tooling gap.
|
||||
|
||||
### Terminal-Bench / SWE-bench (eval harnesses)
|
||||
|
||||
You hold a genuinely rigorous testing bar (100% branch coverage standing rule in `code-quality.md`, CI-mirror gates on
|
||||
5 repos, `pre-commit run --all-files` before every rerun) — the *rigor* profile these benchmarks assume. But these are
|
||||
benchmarks for evaluating *which model/harness* performs best on held-out tasks, not for evaluating *your own repos'*
|
||||
5 repos, `pre-commit run --all-files` before every rerun) — the _rigor_ profile these benchmarks assume. But these are
|
||||
benchmarks for evaluating _which model/harness_ performs best on held-out tasks, not for evaluating _your own repos'_
|
||||
code — you don't train or fine-tune agents, you use Claude Code directly. Speculative only as a way to sanity-check a
|
||||
harness choice in the abstract, not as something you'd integrate into your workflow.
|
||||
|
||||
@ -102,16 +110,18 @@ harness choice in the abstract, not as something you'd integrate into your workf
|
||||
Looked relevant on the surface; concrete evidence says otherwise.
|
||||
|
||||
### Headroom, context-mode, MCP-Zero (progressive-disclosure / token-compression harnesses)
|
||||
|
||||
**Rejected, not "no evidence" — actively contradicted by evidence.** These pitch generic tool-output/context
|
||||
compression as a bolt-on layer. You've already built two purpose-specific, *measured* compilations of exactly this
|
||||
compression as a bolt-on layer. You've already built two purpose-specific, _measured_ compilations of exactly this
|
||||
idea for your two highest-frequency workflows: `finish` (123 invocations/93 sessions → compiled script, -57% tokens)
|
||||
and `phone-deploy` (30 invocations/29 sessions → compiled script, -40% tokens), governed by a written standing
|
||||
principle (`rules/token-spend.instructions.md`) that explicitly says "convert to deterministic coordination only once
|
||||
the steps are predictable" — i.e., generic compression proxies are the wrong layer; you compile the *specific*
|
||||
the steps are predictable" — i.e., generic compression proxies are the wrong layer; you compile the _specific_
|
||||
recurring loop into code instead. A general-purpose compression proxy is a regression from what you're already doing:
|
||||
narrower, deterministic, and measured per-workflow beats a blanket lossy compressor in front of everything.
|
||||
|
||||
### Mem0, cognee, Graphiti (memory layers)
|
||||
|
||||
Rejected on direct substitution grounds: you run **vestige**, a self-hosted memory MCP with local embedding models
|
||||
(`~/.cache/vestige/fastembed`, 669MB, `nomic-embed-text-v1.5` + reranker) and a 27-table SQLite schema
|
||||
(`knowledge_nodes`, `sessions`, `domains`, `agent_traces`, `insights`, `fsrs_cards`), actively written to as of today's
|
||||
@ -122,7 +132,8 @@ create exactly the kind of redundant-parallel-system friction `feedback-check-fo
|
||||
about.
|
||||
|
||||
### Langfuse, MLflow, Opik, Arize Phoenix (observability/eval-ops platforms)
|
||||
Rejected — these are built for teams operating LLM apps *in production for other users*, with tracing/scoring/prompt-
|
||||
|
||||
Rejected — these are built for teams operating LLM apps _in production for other users_, with tracing/scoring/prompt-
|
||||
versioning over live traffic. Your friction pattern is the opposite shape: `feedback-verify-real-deployment-path.md`,
|
||||
`feedback-migrated-means-installed-on-phone.md`, and `feedback-verify-apk-contents-before-install.md` all show the
|
||||
recurring failure is **claiming "done" without running the real deployment path** (dev venv vs. systemd interpreter,
|
||||
@ -131,6 +142,7 @@ workflow rule ("testing is always the LAST step") plus the phone-deploy skill's
|
||||
platforms address "did you actually run it on the target device" — that's not what LLM-tracing tools check.
|
||||
|
||||
### n8n, Dify, langflow (low-code workflow builders)
|
||||
|
||||
Rejected. You have zero evidence of wanting a visual/low-code layer anywhere — every automation in your stack (39
|
||||
systemd user units, 17 Claude Code hooks, CI workflows across ~15 repos) is code-first, version-controlled, and
|
||||
reviewed via pre-commit/CI. Your entire `shell.instructions.md` + `token-spend.instructions.md` posture is explicitly
|
||||
@ -139,12 +151,14 @@ that code can express" — code, not a drag-and-drop canvas). A low-code workflo
|
||||
demonstrated preference for deterministic scripts you can read, diff, and put under CI.
|
||||
|
||||
### browser-use, Stagehand, Steel (standalone browser-agent harnesses)
|
||||
Rejected as *redundant*, not irrelevant — you already have `claude-in-chrome` MCP tools wired in and actively used this
|
||||
|
||||
Rejected as _redundant_, not irrelevant — you already have `claude-in-chrome` MCP tools wired in and actively used this
|
||||
session, plus `servo-fetch` (Servo-engine fetch/crawl/screenshot MCP) under `~/mcp-servers/servers/`. Both solve the
|
||||
"drive a real browser from an agent" problem you'd otherwise reach for these tools to solve. No session shows friction
|
||||
with the browser tooling you already have that a swap would fix.
|
||||
|
||||
### OpenClaw, Khoj, nanobot, CowAgent (self-hosted "personal agent" runtimes)
|
||||
|
||||
Surface-level strong fit — you unambiguously self-host (Gitea, SearXNG, Syncyomi, Joplin Server + DuckDNS, Ollama on
|
||||
an RTX 3090) and run 39 systemd user units as personal automation. But these products are chat/webhook-fronted
|
||||
always-on daemons (Telegram/Discord/WhatsApp bots, "second brain" Q&A) — a different shape of automation than what you
|
||||
@ -154,9 +168,10 @@ rewriting working, well-tested purpose-built daemons into a chatbot-shaped frame
|
||||
memory file shows you wanting conversational/chat-triggered automation, only scheduled/event-triggered guard logic.
|
||||
|
||||
### SWE-agent, AutoGen/AG2, CrewAI, LangGraph, MetaGPT, and the rest of the general-purpose multi-agent frameworks
|
||||
|
||||
Rejected as a category. You don't build third-party-facing agent products — you use Claude Code as your harness and
|
||||
extend it via skills/hooks/MCP servers. None of these frameworks solve a problem you have; they solve "how do I build
|
||||
my own multi-agent LLM application," which isn't your work. If you ever start building an agent *product* (not
|
||||
my own multi-agent LLM application," which isn't your work. If you ever start building an agent _product_ (not
|
||||
personal tooling), this verdict would flip — but nothing in ~140 memory files or 51 session-history directories shows
|
||||
that intent.
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user