testsAndMisc/python_pkg/android_ui

android_ui — drive Android apps by element, never by coordinates

Shared across every repo with an Android surface (~/src/screen-locker's workout_app, ~/src/todo, ~/src/dufs-cloud/app). Works against a plain release APK on a physical device over adb — no root, no emulator, no Flutter debug connection, no change to app source.

Why

Driving an app by tapping pixel positions read off a screenshot fails in ways that look like success. All of these were observed in one session (2026-08-10), verifying the workout app's Firebase restore:

Failure	What it looks like
Tap lands on an unfocused field	Typed text goes nowhere; nothing errors; the next screenshot shows an empty box
Soft keyboard opens	A field moves 1572 → 1319; every coordinate captured a step earlier is wrong
Widget sits behind the keyboard	The tree reports its laid-out position (1549) not its real one (1939) — a 390px error that taps a letter key
Screenshot scaling	The harness renders 1080×2400 at 900×2000, so every read coordinate needs a manual ×1.2
Typing into a filled field	input text inserts at the cursor: old@example.comnew@example.com

Two device quirks are handled rather than documented, because both produced false results before they were understood:

• uiautomator dump returns only the EditText nodes while a Flutter text field holds focus. The button you are about to tap is simply missing, and it stays missing across retries — a naive retry loop never converges.
• KEYCODE_BACK is a route pop in Flutter, not a keyboard dismiss. It navigates out of the screen and discards typed input.

Use

export PYTHONPATH=~/src/testsAndMisc
python3 -m python_pkg.android_ui dump
python3 -m python_pkg.android_ui find "Connect Firebase"
python3 -m python_pkg.android_ui --exact tap "Back"
python3 -m python_pkg.android_ui wait "Connected." --timeout 30
python3 -m python_pkg.android_ui focus

Every subcommand exits non-zero and names the query on failure, so a script can tell a real failure from a successful no-op.

from python_pkg.android_ui import AndroidUi

ui = AndroidUi()                      # or AndroidUi(serial="23181JEGR08034")
ui.tap("Connect Firebase", exact=True)
ui.type_into_field(0, "kuhy@example.com")   # verifies the field changed
ui.wait_for("Connected.", timeout=30)

Guarantees

• Ambiguity is an error. A query matching 0 or >1 elements raises; it never silently acts on the first match. (tap "Back" correctly refuses when "OFFLINE BACKUP" is also on screen — use --exact.)
• Coordinates are never reused. Every action re-reads the tree immediately before acting, and tap closes the keyboard first when one is up.
• Typing is verified. type_into / type_into_field re-read the field and raise if the contents did not change. They clear before typing, so a filled field is replaced, not appended to.
• Empty fields are findable. An empty EditText has no text, content-desc or resource-id; it is kept anyway, addressable by position via editable_fields() / type_into_field(index, …).
• Partial trees are retried, and keyboard_is_up() distinguishes "element absent" from "element behind the keyboard".

Known gaps

• A widget with no accessibility label cannot be targeted at all. The workout app's Settings gear is one — that is an a11y bug in the app, and this tool surfacing it is the point. Add a Semantics(label: …) and it becomes addressable.
  • But check hint first. Material/Flutter text fields publish their label as hint and leave text/content-desc empty; those ARE addressable (since 2026-08-27) and need no app change. Adding a Semantics(label:) wrapper to such a field is actively harmful — it emits a SECOND labelled node at the same rect, and wrapping in MergeSemantics to collapse them removes the hint from Android's tree entirely, making the whole form anonymous. Verified on the manual-workout form; reverted.
  • Verify any a11y change with adb exec-out uiautomator dump, never with flutter test: the widget test asserts on Flutter's internal SemanticsNode, while this tool reads Android's AccessibilityNodeInfo. Nine widget tests passed green on the build whose on-device labels had just gone empty.
• Scrolling is not yet modelled: a widget outside the viewport is not in the tree. Scroll with adb shell input swipe first, or add scroll_to().

One phone, many sessions

Sessions share the phone by working on separate displays. By default every command drives the real screen (display 0) and takes this session's pkg:__screen__ lease (python_pkg.phone_lease: owner = the nearest ancestor claude process, 180 s past the last call). A call from another session waits up to 60 s, then exits 3 naming the holder. Do not retry in a loop.

--display self drives this session's virtual display instead (~/.claude/scripts/phone_vd.sh start <package> creates it) and takes only that app's lease, so several sessions can act at once:

android-ui --display self dump
android-ui --display self tap "Save"

uiautomator dump sees display 0 only, so on a virtual display the tree comes from the com.kuhy.a11ydump instrumentation helper (A11yDump.java, _a11y_helper.py): built once into ~/.cache/android_ui/ and installed with adb install -r -t on first use. Its package must stay in phone-focus-mode's whitelist, or the focus-owner sweep hides it. Android runs one instrumentation per package, so a second am instrument kills the first: dumps of one phone queue behind a host-side flock (~/.cache/android_ui/a11ydump-<serial>.lock). Every input call carries -d (0 for the real screen), because input without it goes to whichever display last took focus. A numeric --display that is another session's virtual display is refused.

Raw adb from a Claude Bash call is covered too: the ~/.claude/hooks/phone_guard_pretool.sh hook (python_pkg.phone_guard) takes the matching lease or blocks the call. See the phone-deploy skill, section 3, for the full rules.