Find Material text fields by their hint label

A Flutter/Material TextField publishes its label as `hint` and leaves text
and content-desc empty, so every field on such a form arrived as an
anonymous EditText. The workout app's manual-workout form is nine of them:
the only way to fill it was by screen position, which is precisely the
mis-tap-writes-the-wrong-field failure this tool exists to prevent.

`hint` ranks below text and content-desc but above resource-id. Material
clears the hint once content arrives, so a filled field still answers to its
real text and a caller can tell an empty field from a completed one.

The app-side alternative does not work and is documented as a trap: wrapping
the field in Semantics(label:) emits a second labelled node at the same rect,
and MergeSemantics collapses them by dropping the hint from Android's tree,
which leaves the form MORE anonymous than before. Widget tests cannot catch
that -- they assert on Flutter's SemanticsNode while this tool reads
Android's AccessibilityNodeInfo, so nine of them passed green on the build
whose on-device labels had just gone empty.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AWq9MRemX9ZWvNmnhNWUP2

docs/superpowers/evidence/android-ui-hint-labels-2026-08-27.json

@@ -0,0 +1,40 @@
+{
+  "intent": "Let android_ui address Flutter/Material text fields by their visible label, so a form can be filled by name instead of by screen position.",
+  "scope": [
+    "python_pkg/android_ui/_elements.py",
+    "python_pkg/android_ui/tests/test_driver.py",
+    "python_pkg/android_ui/README.md",
+    "Non-goal: no change to any app under test — the gap was in this tool, not in the app"
+  ],
+  "changes": [
+    "Parse the node's `hint` attribute and carry it on UiElement (default \"\", so existing constructor calls are unaffected).",
+    "Rank hint below text and content-desc but above resource-id in `label`, and add it to the `matches` haystack: Material clears the hint once content arrives, so a filled field still answers to its real text.",
+    "Keep a node that has only a hint, which the previous 'must be labelled' filter dropped.",
+    "Document that wrapping such a field in Semantics(label:)/MergeSemantics is a trap that empties Android's tree, and that a11y changes must be verified with uiautomator dump rather than flutter test."
+  ],
+  "verification": [
+    {
+      "command": "python3 -m pytest python_pkg/android_ui/tests/ -q",
+      "result": "pass",
+      "evidence": "59 passed (57 before, +2 new: hint names an empty field; real text outranks the hint once typed)"
+    },
+    {
+      "command": "pre-commit run --files python_pkg/android_ui/_elements.py python_pkg/android_ui/tests/test_driver.py python_pkg/android_ui/README.md",
+      "result": "pass",
+      "evidence": "no failures reported"
+    },
+    {
+      "command": "python3 -m python_pkg.android_ui dump  # workout_app manual-workout form, real device 23181JEGR08034",
+      "result": "pass",
+      "evidence": "Fields previously dumped as '' now read 'Start time (HH:MM)', 'End time (HH:MM)', 'Location name', 'How did you get there?', 'Cost (e.g. 40 PLN)', 'What sport/activity', 'What was done'. Nine fields then filled by name, each reporting 'typed into <label> and verified'."
+    }
+  ],
+  "risks": [
+    "A screen whose hint duplicates another element's text could become ambiguous; matches() already raises AmbiguousElementError rather than acting on a coin flip.",
+    "hint is only present while a Material field is empty, so resolve every target from one dump pass before typing rather than re-dumping between fields."
+  ],
+  "rollback": [
+    "git revert the commit; UiElement.hint defaults to \"\" so nothing else depends on it.",
+    "After rollback, re-run the android_ui suite and expect Material fields to dump as anonymous EditTexts again."
+  ]
+}

python_pkg/android_ui/README.md

@@ -73,5 +73,17 @@ ui.wait_for("Connected.", timeout=30)
   app's Settings gear is one — that is an a11y bug in the app, and this tool
   surfacing it is the point. Add a `Semantics(label: …)` and it becomes
   addressable.
+  - **But check `hint` first.** Material/Flutter text fields publish their
+    label as `hint` and leave text/content-desc empty; those ARE addressable
+    (since 2026-08-27) and need no app change. Adding a `Semantics(label:)`
+    wrapper to such a field is actively harmful — it emits a SECOND labelled
+    node at the same rect, and wrapping in `MergeSemantics` to collapse them
+    removes the `hint` from Android's tree entirely, making the whole form
+    anonymous. Verified on the manual-workout form; reverted.
+  - Verify any a11y change with `adb exec-out uiautomator dump`, never with
+    `flutter test`: the widget test asserts on Flutter's internal
+    `SemanticsNode`, while this tool reads Android's `AccessibilityNodeInfo`.
+    Nine widget tests passed green on the build whose on-device labels had
+    just gone empty.
 - Scrolling is not yet modelled: a widget outside the viewport is not in the
   tree. Scroll with `adb shell input swipe` first, or add `scroll_to()`.

python_pkg/android_ui/_elements.py

@@ -42,11 +42,20 @@ class UiElement:
     bounds: tuple[int, int, int, int]
     enabled: bool
     focused: bool
+    # Material/Flutter text fields publish their label as `hint` and leave
+    # text/content-desc empty, so without this an entire form is anonymous
+    # edit boxes and the only way to fill it is by screen position.
+    hint: str = ""
 
     @property
     def label(self) -> str:
-        """The best human-facing name for this element."""
-        return self.text or self.content_desc or self.resource_id
+        """The best human-facing name for this element.
+
+        ``hint`` ranks below the others but above nothing: it is a field's
+        placeholder, which Material clears once the field has content, so it
+        names an EMPTY field and defers to real text as soon as one is typed.
+        """
+        return self.text or self.content_desc or self.hint or self.resource_id
 
     @property
     def center(self) -> tuple[int, int]:
@@ -56,7 +65,7 @@ class UiElement:
 
     def matches(self, query: str, *, exact: bool = False) -> bool:
         """Return True if ``query`` names this element."""
-        haystacks = (self.text, self.content_desc, self.resource_id)
+        haystacks = (self.text, self.content_desc, self.hint, self.resource_id)
         if exact:
             return any(h == query for h in haystacks)
         return any(query.lower() in h.lower() for h in haystacks if h)
@@ -101,18 +110,20 @@ def _parse_tree(xml: str) -> list[UiElement]:
             continue
         text = node.get("text", "")
         desc = node.get("content-desc", "")
+        hint = node.get("hint", "")
         res = node.get("resource-id", "")
         cls = node.get("class", "")
         # An EMPTY text field has no text, content-desc or resource-id, so a
         # "must be labelled" filter drops it -- and an empty field is exactly
         # the thing a caller needs to find in order to type into it. Keep every
         # editable node regardless of label.
-        if not (text or desc or res or cls.endswith("EditText")):
+        if not (text or desc or hint or res or cls.endswith("EditText")):
             continue
         elements.append(
             UiElement(
                 text=text,
                 content_desc=desc,
+                hint=hint,
                 resource_id=res,
                 class_name=cls,
                 bounds=(

python_pkg/android_ui/tests/test_driver.py

@@ -24,11 +24,12 @@ MOD = "python_pkg.android_ui.driver"
 def _node(text: str = "", **attrs: str) -> str:
     """Render one ``<node>`` element for a fake dump.
 
-    ``attrs`` accepts ``desc``, ``res``, ``cls``, ``bounds``, ``enabled`` and
-    ``focused``; each falls back to a representative default.
+    ``attrs`` accepts ``desc``, ``hint``, ``res``, ``cls``, ``bounds``,
+    ``enabled`` and ``focused``; each falls back to a representative default.
     """
     return (
         f'<node text="{text}" content-desc="{attrs.get("desc", "")}" '
+        f'hint="{attrs.get("hint", "")}" '
         f'resource-id="{attrs.get("res", "")}" '
         f'class="{attrs.get("cls", "android.widget.TextView")}" '
         f'bounds="{attrs.get("bounds", "[0,0][100,50]")}" '
@@ -209,3 +210,29 @@ class TestTap:
             c for c in ui.device.calls if c[:3] == ("shell", "input", "keyevent")
         ]
         assert keyevents, "expected the keyboard to be dismissed first"
+
+
+def test_hint_names_an_empty_material_field() -> None:
+    """A Flutter/Material field publishes its label only as ``hint``.
+
+    Such a field carries no text and no content-desc, so without hint it is an
+    anonymous edit box and the only way to fill it is by screen position --
+    which is how a mis-tap silently writes a value into the wrong field.
+    """
+    dump = _node(cls="android.widget.EditText", hint="Location name")
+    (element,) = drv._parse_tree(_tree(dump))
+
+    assert element.label == "Location name"
+    assert element.matches("location")
+
+
+def test_real_text_outranks_the_hint_once_typed() -> None:
+    """Material clears the hint when content arrives, so text must win.
+
+    Otherwise a filled field would still answer to its placeholder and a
+    caller could not tell an empty field from a completed one.
+    """
+    dump = _node("17:30", cls="android.widget.EditText", hint="Start time (HH:MM)")
+    (element,) = drv._parse_tree(_tree(dump))
+
+    assert element.label == "17:30"