mirror of
https://github.com/kuhyx/testsAndMisc.git
synced 2026-11-12 10:31:02 +01:00
Krzysztof kuhy Rudnicki
aa37852296
A raw `adb shell input tap` took no lease, so one session's tap landed in another session's app (2026-09-25). python_pkg.phone_guard classifies every adb call in a Bash command and, as the Claude Code PreToolUse hook, takes the matching phone_lease (one app / real screen / whole phone) or exits 2: key/text input without -d, input on another session's virtual display, or a lease still held elsewhere after 40 s. android_ui gains --display self|<id>: it drives only this session's phone_vd display under that app's lease, and every input carries -d. uiautomator sees display 0 only, so virtual-display trees come from a tiny instrumentation APK (A11yDump.java) built from source without Gradle. Prunes the oldest evidence artifact to stay under the 60-file cap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WpA6wW2DKiovQADTdjWJoH
120 lines
6.3 KiB
Markdown
120 lines
6.3 KiB
Markdown
# android_ui — drive Android apps by element, never by coordinates
|
||
|
||
Shared across every repo with an Android surface (`~/src/screen-locker`'s
|
||
workout_app, `~/src/todo`, `~/src/dufs-cloud/app`). Works against a plain release APK
|
||
on a physical device over `adb` — no root, no emulator, no Flutter debug
|
||
connection, no change to app source.
|
||
|
||
## Why
|
||
|
||
Driving an app by tapping pixel positions read off a screenshot fails in ways
|
||
that look like success. All of these were observed in one session
|
||
(2026-08-10), verifying the workout app's Firebase restore:
|
||
|
||
| Failure | What it looks like |
|
||
| ------------------------------- | -------------------------------------------------------------------------------------------------------------- |
|
||
| Tap lands on an unfocused field | Typed text goes nowhere; nothing errors; the next screenshot shows an empty box |
|
||
| Soft keyboard opens | A field moves 1572 → 1319; every coordinate captured a step earlier is wrong |
|
||
| Widget sits behind the keyboard | The tree reports its _laid-out_ position (1549) not its real one (1939) — a 390px error that taps a letter key |
|
||
| Screenshot scaling | The harness renders 1080×2400 at 900×2000, so every read coordinate needs a manual ×1.2 |
|
||
| Typing into a filled field | `input text` inserts at the cursor: `old@example.comnew@example.com` |
|
||
|
||
Two device quirks are handled rather than documented, because both produced
|
||
false results before they were understood:
|
||
|
||
- `uiautomator dump` returns **only the `EditText` nodes** while a Flutter text
|
||
field holds focus. The button you are about to tap is simply missing, and it
|
||
stays missing across retries — a naive retry loop never converges.
|
||
- `KEYCODE_BACK` is a **route pop** in Flutter, not a keyboard dismiss. It
|
||
navigates out of the screen and discards typed input.
|
||
|
||
## Use
|
||
|
||
```bash
|
||
export PYTHONPATH=~/src/testsAndMisc
|
||
python3 -m python_pkg.android_ui dump
|
||
python3 -m python_pkg.android_ui find "Connect Firebase"
|
||
python3 -m python_pkg.android_ui --exact tap "Back"
|
||
python3 -m python_pkg.android_ui wait "Connected." --timeout 30
|
||
python3 -m python_pkg.android_ui focus
|
||
```
|
||
|
||
Every subcommand exits non-zero and names the query on failure, so a script can
|
||
tell a real failure from a successful no-op.
|
||
|
||
```python
|
||
from python_pkg.android_ui import AndroidUi
|
||
|
||
ui = AndroidUi() # or AndroidUi(serial="23181JEGR08034")
|
||
ui.tap("Connect Firebase", exact=True)
|
||
ui.type_into_field(0, "kuhy@example.com") # verifies the field changed
|
||
ui.wait_for("Connected.", timeout=30)
|
||
```
|
||
|
||
## Guarantees
|
||
|
||
- **Ambiguity is an error.** A query matching 0 or >1 elements raises; it never
|
||
silently acts on the first match. (`tap "Back"` correctly refuses when
|
||
"OFFLINE BACKUP" is also on screen — use `--exact`.)
|
||
- **Coordinates are never reused.** Every action re-reads the tree immediately
|
||
before acting, and `tap` closes the keyboard first when one is up.
|
||
- **Typing is verified.** `type_into` / `type_into_field` re-read the field and
|
||
raise if the contents did not change. They clear before typing, so a filled
|
||
field is replaced, not appended to.
|
||
- **Empty fields are findable.** An empty `EditText` has no text, content-desc
|
||
or resource-id; it is kept anyway, addressable by position via
|
||
`editable_fields()` / `type_into_field(index, …)`.
|
||
- **Partial trees are retried,** and `keyboard_is_up()` distinguishes "element
|
||
absent" from "element behind the keyboard".
|
||
|
||
## Known gaps
|
||
|
||
- A widget with no accessibility label cannot be targeted at all. The workout
|
||
app's Settings gear is one — that is an a11y bug in the app, and this tool
|
||
surfacing it is the point. Add a `Semantics(label: …)` and it becomes
|
||
addressable.
|
||
- **But check `hint` first.** Material/Flutter text fields publish their
|
||
label as `hint` and leave text/content-desc empty; those ARE addressable
|
||
(since 2026-08-27) and need no app change. Adding a `Semantics(label:)`
|
||
wrapper to such a field is actively harmful — it emits a SECOND labelled
|
||
node at the same rect, and wrapping in `MergeSemantics` to collapse them
|
||
removes the `hint` from Android's tree entirely, making the whole form
|
||
anonymous. Verified on the manual-workout form; reverted.
|
||
- Verify any a11y change with `adb exec-out uiautomator dump`, never with
|
||
`flutter test`: the widget test asserts on Flutter's internal
|
||
`SemanticsNode`, while this tool reads Android's `AccessibilityNodeInfo`.
|
||
Nine widget tests passed green on the build whose on-device labels had
|
||
just gone empty.
|
||
- Scrolling is not yet modelled: a widget outside the viewport is not in the
|
||
tree. Scroll with `adb shell input swipe` first, or add `scroll_to()`.
|
||
|
||
## One phone, many sessions
|
||
|
||
Sessions share the phone by working on separate displays. By default every
|
||
command drives the real screen (display 0) and takes this session's
|
||
`pkg:__screen__` lease (`python_pkg.phone_lease`: owner = the nearest
|
||
ancestor `claude` process, 180 s past the last call). A call from another
|
||
session waits up to 60 s, then exits 3 naming the holder. Do not retry in a
|
||
loop.
|
||
|
||
`--display self` drives this session's virtual display instead
|
||
(`~/.claude/scripts/phone_vd.sh start <package>` creates it) and takes only
|
||
that app's lease, so several sessions can act at once:
|
||
|
||
android-ui --display self dump
|
||
android-ui --display self tap "Save"
|
||
|
||
`uiautomator dump` sees display 0 only, so on a virtual display the tree
|
||
comes from the `com.kuhy.a11ydump` instrumentation helper (`A11yDump.java`,
|
||
`_a11y_helper.py`): built once into `~/.cache/android_ui/` and installed
|
||
with `adb install -r -t` on first use. Its package must stay in
|
||
phone-focus-mode's whitelist, or the focus-owner sweep hides it. Every
|
||
`input` call carries `-d` (0 for the real screen), because input without it
|
||
goes to whichever display last took focus. A numeric `--display` that is
|
||
another session's virtual display is refused.
|
||
|
||
Raw `adb` from a Claude Bash call is covered too: the
|
||
`~/.claude/hooks/phone_guard_pretool.sh` hook (`python_pkg.phone_guard`)
|
||
takes the matching lease or blocks the call. See the `phone-deploy` skill,
|
||
section 3, for the full rules.
|