utils/flutter_web_drive/README.md

flutter_web_drive

Headless, LLM-friendly driver for Flutter web apps. An agent can navigate and test a desktop app (a Flutter web build in a Chrome --app window, the way kartka and todo ship) without touching the real screen, mouse or keyboard.

• A long-lived headless chromium runs as a systemd-run --user unit with a persistent profile. It has to be a unit because setsid children die with the calling Bash scope.
• Each CLI verb attaches over CDP, does one thing and exits. The exit code is the verdict. On failure it prints one stderr line naming the step that failed.
• Widgets are addressed through Flutter's semantics tree. Flutter paints to a canvas, so the accessibility mirror ([id^=flt-semantic-node-]) is the only addressable view. The driver switches it on by clicking flt-semantics-placeholder. The app's lib/ needs no changes.

Spike result (2026-10-24, chromium 155, Flutter 3.47.8 CanvasKit)

--headless=new + CDP Page.captureScreenshot renders CanvasKit frames correctly. (The old --headless --screenshot path was blank.) No Xvfb is needed, and none is used.

Other findings from the spike:

• Semantics enabled once stay enabled across later CDP connections.
• Console messages and exceptions are replayed on Runtime.enable / Log.enable, so console works from a fresh attach.
• The viewport is smaller than --window-size: 1400x1000 gives 1400x879. start reports the real viewport.

Safety

• Every attach reads /json/version. It refuses any browser that is not HeadlessChrome (exit 4). For example, Steam's CEF listens on 9222 on this PC. Never point the driver at 9222.
• The unit unsets DISPLAY/WAYLAND_DISPLAY. --password-store=basic keeps chromium away from the desktop keyring, which could prompt on :0. --disable-extensions drops the Claude extension that chromium-flags.conf loads.
• start refuses a CDP port that some other browser already answers on.
• Secrets go through text --stdin / text --file PATH (MCP: from_file), never argv. tree shows password fields as *.

CLI

export PYTHONPATH=~/src/utils
D="python3 -m flutter_web_drive --app myapp"   # --app namespaces state/artefacts
$D start --url http://127.0.0.1:47392/ --profile ~/.local/share/myapp-agent/profile \
         --cdp-port 9342 [--unit fwd-myapp] [--size 1400x1000]
$D tree [--filter TEXT] [--row LABEL] [--all]
$D tap "Browse" [--contains]          # exact label or one line of it
$D tap-xy 40 100
$D text "hello" | $D text --stdin | $D text --file ~/.config/x/password
$D key Enter                          # Escape Tab Backspace Delete Space Arrow* Home End PageUp PageDown
$D scroll 0 600 [--at X Y]
$D wait "Sync now" [--timeout S<=50] [--contains] [--gone] [--flag current] [--row LABEL]
$D shot [OUT.png] [--min-colours 16]  # prints "path WxH colours=N stddev=S"
$D eval "location.href"
$D console [--limit 20] [--warnings] [--fail-on-error]
$D stop

tree prints one line per node:

#5 button "Browse | Tab 2 of 4" [tap,current] @0,75 80x67
#73 textbox "Sync account email" = "me@example.com" [focused] @16,203 1368x48

• [tags]: tap (tappable), current, selected, checked, mixed, pressed, expanded, disabled, focused.
• = value is the field value, aria-valuenow or aria-description. For example, an ExpansionTile reads "Collapsed".
• --row LABEL keeps the nodes inside the row of that label, e.g. a chapter row's "Mark unread" button: wait "Mark unread" --row "Chapter 5".
• tap prefers tappable matches and says when there were several. When nothing matches, it lists what is tappable.

Exit codes: 0 ok, 1 the step's check failed (not found, timeout, page exception, blank shot), 2 usage, 3 no driver browser (not started or crashed), 4 the CDP port belongs to a non-headless browser.

Artefacts (screenshots, state.json) go to ~/data/claude-scratch/e2e/<app>/ (override with $FWD_ARTEFACTS), never to /tmp. A wait timeout saves wait-timeout-*.png and names it in the error.

MCP server

mcp_server.py is a thin stdio server (mcp 2.x MCPServer). Its tools match the verbs: start stop shot tree tap tap_xy text key scroll wait eval_js console, each taking app. Every tool is one Driver call, so no logic is duplicated. Register it (user scope):

claude mcp add flutter-web-drive -s user -e PYTHONPATH=$HOME/src/utils \
  -- python3 -m flutter_web_drive.mcp_server

Layout

file	role
cdp.py	websocket session, page discovery, headless check
semantics.py	the JS snippets, Node, matching, row scoping, tree format
actions.py / keys.py	verbs on one attached page
browser.py / state.py	chromium argv, systemd-run unit, per-app state
api.py	Driver: the verbs the CLI and the MCP server share
cli.py / mcp_server.py	the two front ends

Tests (tests/, 100 % branch, no network or browser) use a fake CDP page. The real-browser check is a consumer flow, e.g. ~/src/kartka/tools/agent/flows/f0_smoke.sh.

PYTHONPATH=. python -m pytest -q flutter_web_drive/tests \
  --cov=flutter_web_drive --cov-branch --cov-fail-under=100
ruff check flutter_web_drive && ruff format --check flutter_web_drive
pip install --user -r flutter_web_drive/requirements.txt