testsAndMisc/python_pkg/android_ui/tests
Krzysztof kuhy Rudnicki b73811c233
Some checks are pending
Pre-commit checks / pre-commit (push) Waiting to run
Python tests / test (push) Waiting to run
Shell tests / shell-tests (push) Waiting to run
Shell tests / arch-tests (push) Waiting to run
Add android_ui: drive Android apps by element, never by coordinates
Shared across every repo with an Android surface (screen-locker's workout_app,
todo, dufs-cloud/app). Works against a plain release APK on a physical device
over adb — no root, no emulator, no Flutter debug connection, no app changes.

Driving an app by tapping pixel positions read off a screenshot fails in ways
that look like success. All observed in one session (2026-08-10) verifying
workout_app's Firebase restore:

* A tap aimed at a password field landed with the field unfocused. The typed
  password went nowhere and NOTHING errored — the next screenshot just showed
  an empty box. type_into/type_into_field now re-read the field and raise.
* Opening the soft keyboard moved a field from y=1572 to y=1319, so a
  coordinate captured one step earlier was already wrong. Every action
  re-resolves the element immediately before acting.
* A widget behind the keyboard reports its LAID-OUT position (1549) not its
  real one (1939) — a 390px error that taps a letter key and reports success.
  keyboard_is_up() distinguishes "absent" from "behind the keyboard", and tap
  closes the keyboard first.
* KEYCODE_BACK is a route pop in Flutter, not a keyboard dismiss: it navigated
  out of Settings and discarded typed credentials, twice. dismiss_keyboard
  tries ESCAPE then BACK and verifies via dumpsys input_method.
* `input text` inserts at the cursor, so typing over a filled field produced
  "canary@example.com321krzychu@gmail.com". Fields are cleared first.
* uiautomator returns ONLY the focused EditText nodes while a Flutter text
  field has focus, and stays that way across retries — so a naive retry loop
  never converges. Partial trees are retried and the best result kept.
* An EMPTY EditText has no text, content-desc or resource-id, so a
  "must be labelled" filter drops the very field you need to fill in. Editable
  nodes are kept regardless and addressable by position.

Ambiguity is an error rather than a silent first-match: `tap "Back"` correctly
refuses when "OFFLINE BACKUP" is also on screen.

Its own test suite caught a real bug in _escape: backslash sat inside the
escape loop, so "&" became "\\&" and a password containing shell
metacharacters would have been typed wrong.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JupKvZCNKRmEeRQY3G1unG
2026-08-10 17:01:28 +02:00
..
__init__.py Add android_ui: drive Android apps by element, never by coordinates 2026-08-10 17:01:28 +02:00
test_cli.py Add android_ui: drive Android apps by element, never by coordinates 2026-08-10 17:01:28 +02:00
test_driver.py Add android_ui: drive Android apps by element, never by coordinates 2026-08-10 17:01:28 +02:00
test_main_entry.py Add android_ui: drive Android apps by element, never by coordinates 2026-08-10 17:01:28 +02:00