testsAndMisc/python_pkg/android_ui

android_ui — drive Android apps by element, never by coordinates

Shared across every repo with an Android surface (~/src/screen-locker's workout_app, ~/src/todo, ~/src/dufs-cloud/app). Works against a plain release APK on a physical device over adb — no root, no emulator, no Flutter debug connection, no change to app source.

Why

Driving an app by tapping pixel positions read off a screenshot fails in ways that look like success. All of these were observed in one session (2026-08-10), verifying the workout app's Firebase restore:

Failure	What it looks like
Tap lands on an unfocused field	Typed text goes nowhere; nothing errors; the next screenshot shows an empty box
Soft keyboard opens	A field moves 1572 → 1319; every coordinate captured a step earlier is wrong
Widget sits behind the keyboard	The tree reports its laid-out position (1549) not its real one (1939) — a 390px error that taps a letter key
Screenshot scaling	The harness renders 1080×2400 at 900×2000, so every read coordinate needs a manual ×1.2
Typing into a filled field	input text inserts at the cursor: old@example.comnew@example.com

Two device quirks are handled rather than documented, because both produced false results before they were understood:

• uiautomator dump returns only the EditText nodes while a Flutter text field holds focus. The button you are about to tap is simply missing, and it stays missing across retries — a naive retry loop never converges.
• KEYCODE_BACK is a route pop in Flutter, not a keyboard dismiss. It navigates out of the screen and discards typed input.

Use

export PYTHONPATH=~/src/testsAndMisc
python3 -m python_pkg.android_ui dump
python3 -m python_pkg.android_ui find "Connect Firebase"
python3 -m python_pkg.android_ui --exact tap "Back"
python3 -m python_pkg.android_ui wait "Connected." --timeout 30
python3 -m python_pkg.android_ui focus

Every subcommand exits non-zero and names the query on failure, so a script can tell a real failure from a successful no-op.

from python_pkg.android_ui import AndroidUi

ui = AndroidUi()                      # or AndroidUi(serial="23181JEGR08034")
ui.tap("Connect Firebase", exact=True)
ui.type_into_field(0, "kuhy@example.com")   # verifies the field changed
ui.wait_for("Connected.", timeout=30)

Guarantees

• Ambiguity is an error. A query matching 0 or >1 elements raises; it never silently acts on the first match. (tap "Back" correctly refuses when "OFFLINE BACKUP" is also on screen — use --exact.)
• Coordinates are never reused. Every action re-reads the tree immediately before acting, and tap closes the keyboard first when one is up.
• Typing is verified. type_into / type_into_field re-read the field and raise if the contents did not change. They clear before typing, so a filled field is replaced, not appended to.
• Empty fields are findable. An empty EditText has no text, content-desc or resource-id; it is kept anyway, addressable by position via editable_fields() / type_into_field(index, …).
• Partial trees are retried, and keyboard_is_up() distinguishes "element absent" from "element behind the keyboard".

Known gaps

• A widget with no accessibility label cannot be targeted at all. The workout app's Settings gear is one — that is an a11y bug in the app, and this tool surfacing it is the point. Add a Semantics(label: …) and it becomes addressable.
  • But check hint first. Material/Flutter text fields publish their label as hint and leave text/content-desc empty; those ARE addressable (since 2026-08-27) and need no app change. Adding a Semantics(label:) wrapper to such a field is actively harmful — it emits a SECOND labelled node at the same rect, and wrapping in MergeSemantics to collapse them removes the hint from Android's tree entirely, making the whole form anonymous. Verified on the manual-workout form; reverted.
  • Verify any a11y change with adb exec-out uiautomator dump, never with flutter test: the widget test asserts on Flutter's internal SemanticsNode, while this tool reads Android's AccessibilityNodeInfo. Nine widget tests passed green on the build whose on-device labels had just gone empty.
• A widget outside the viewport is not in the tree. android-ui scroll-to QUERY (AndroidUi.scroll_to) swipes up until it appears, and stops with an error at the end of the list or after 8 swipes. It only scrolls downward.

One phone, many sessions

Sessions share the phone by working on separate displays. By default every command drives the real screen (display 0) and takes this session's pkg:__screen__ lease (python_pkg.phone_lease: owner = the nearest ancestor claude process, 180 s past the last call). A call from another session waits up to 60 s, then exits 3 naming the holder. Do not retry in a loop.

--display self drives this session's virtual display instead (~/.claude/scripts/phone_vd.sh start <package> creates it) and takes only that app's lease, so several sessions can act at once:

android-ui --display self dump
android-ui --display self tap "Save"

uiautomator dump sees display 0 only, so on a virtual display the tree comes from the com.kuhy.a11ydump instrumentation helper (A11yDump.java, _a11y_helper.py): built once into ~/.cache/android_ui/ and installed with adb install -r -t on first use. Its package must stay in phone-focus-mode's whitelist, or the focus-owner sweep hides it. Android runs one instrumentation per package, so a second am instrument kills the first: dumps of one phone queue behind a host-side flock (~/.cache/android_ui/a11ydump-<serial>.lock). Every input call carries -d (0 for the real screen), because input without it goes to whichever display last took focus. A numeric --display that is another session's virtual display is refused.

Raw adb from a Claude Bash call is covered too: the ~/.claude/hooks/phone_guard_pretool.sh hook (python_pkg.phone_guard) takes the matching lease or blocks the call. See the phone-deploy skill, section 3, for the full rules.