Add android_ui MCP server and Firebase flow
Some checks are pending
dependency freshness / dependencies on newest stable (push) Waiting to run
md-naming / markdown naming convention (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (mypy, mypy) (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (pylint, pylint) (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (rest, ) (push) Waiting to run
Pre-commit checks / pre-commit (${{ matrix.group }}) (shellcheck, shellcheck) (push) Waiting to run
Python tests / test (push) Waiting to run
repo-contract / repo contract (push) Waiting to run
Shell tests / shell-tests (push) Waiting to run
Shell tests / lib-tests (push) Waiting to run
Shell tests / arch-tests (push) Waiting to run

Expose dump/find/tap/type_into/type_into_field/wait_for/scroll_to/focus
over FastMCP stdio. Every driver failure comes back as isError naming
the query and count; typed text is never echoed. Add the workout_app
Firebase flow script and register the server in .mcp.json.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Krzysztof Rudnicki 2026-11-16 08:45:30 +01:00
parent 637141a97c
commit a4738c6fdf
16 changed files with 1090 additions and 42 deletions

9
.mcp.json Normal file
View File

@ -0,0 +1,9 @@
{
"mcpServers": {
"android-ui": {
"type": "stdio",
"command": "${HOME}/src/testsAndMisc/.venv/bin/python",
"args": ["-m", "python_pkg.android_ui.mcp_server"]
}
}
}

View File

@ -267,6 +267,8 @@ repos:
# android_ui parses `uiautomator dump` output. Same reason as
# beautifulsoup4 above: requirements.txt is not on pylint's path.
- defusedxml
# android_ui's MCP server; same isolated-venv reason.
- fastmcp==4.2.0
# artgate computes image gates over pixel arrays. Same reason again:
# it is the first package here to import numpy.
- numpy

View File

@ -0,0 +1,17 @@
{
"title": "android_ui MCP server",
"objective": "Expose AndroidUi's eight element-targeted methods as FastMCP stdio tools with the CLI's error guarantees, plus a label-only workout_app Firebase flow script.",
"acceptance_criteria": [
"The in-process client lists exactly dump, find, tap, type_into, type_into_field, wait_for, scroll_to, focus (test_mcp_server.py)",
"tap with 0 and >1 matches returns isError naming the query and count (test_mcp_server.py)",
"type_into / type_into_field on an unchanged field return isError with before/after length and never the typed text (test_mcp_server.py, test_mcp_target.py)",
"No serial with 2 devices returns isError listing both; display=self with no virtual display returns isError and creates none (test_mcp_target.py)",
"100% branch coverage of python_pkg/android_ui"
],
"out_of_scope": [
"Screenshots",
"Rollout to ~/src/todo and ~/src/dufs-cloud",
"Any screen-locker source change"
],
"verifier": "python_pkg/android_ui/tests under pytest --cov-branch, plus a stdio smoke client"
}

View File

@ -0,0 +1,67 @@
{
"intent": "An agent drives Android apps by element through MCP tools (dump, find, tap, type_into, type_into_field, wait_for, scroll_to, focus) instead of shelling out to adb or tapping coordinates, with the CLI's guarantees: 0 or >1 matches and unchanged fields are errors.",
"scope": [
"python_pkg/android_ui/mcp_server.py, _mcp_target.py, workout_firebase_flow.py and their tests",
"driver.py/_text_entry.py: unchanged-field messages give before/after length, never the typed text",
"fastmcp==4.2.0 pinned (bumped from 4.1.6 by the freshness gate) in meta/requirements.txt, requirements-test.txt and the pylint hook deps",
"Non-goals: screenshots, emulator support, app source changes, ~/.claude.json"
],
"changes": [
"FastMCP stdio server, one tool per AndroidUi method; every UiAutomationError and device/display/lease refusal is a ToolError (isError)",
"Target resolution reuses cli._target and phone_lease; default display 'self' never creates a virtual display; several devices without serial errors listing them",
"Typed text is redacted from every error, including adb 'input text' command echoes",
"workout_firebase_flow.py: label-only Home -> Settings -> Firebase sync -> Sign in with Google flow, hands over on secret screens, --negative step",
".mcp.json registers the server (project scope, stdio, ${HOME}/src/testsAndMisc/.venv/bin/python -m python_pkg.android_ui.mcp_server)"
],
"verification": [
{
"command": "capped.sh python -m pytest -q --cov=python_pkg.android_ui --cov-branch python_pkg/android_ui/tests",
"result": "pass",
"evidence": "137 passed; 100.00% branch coverage (rerun 2026-11-08 after the chooser fix)"
},
{
"command": "stdio smoke client: python -m python_pkg.android_ui.mcp_server, list_tools + call dump/focus",
"result": "pass",
"evidence": "8 tools listed, no screenshot tool; no device: isError 'no phone serial ...'; one device, no virtual display: isError 'no virtual display for this session: run phone_vd.sh start <package>'"
},
{
"command": "clean venv: capped.sh pip install --dry-run -r meta/requirements.txt (baseline vs with fastmcp)",
"result": "pass",
"evidence": "both resolve, exit 0; fastmcp 4.2.0 rechecked after the bump"
},
{
"command": "pre-commit hooks on changed files (SKIP=pylint) + standalone pylint",
"result": "pass",
"evidence": "ruff/mypy/prettier hooks pass; pylint 10.00/10 from the test venv; pylint hook env with fastmcp not installed under the 60 s cap; pytest-coverage hook needs defusedxml/sql_drills deps on the hook interpreter"
},
{
"command": "stdio MCP client on Pixel 23181JEGR08034 (slot com.kuhy.slot5, own virtual display): find/tap/dump",
"result": "pass",
"evidence": "no serial with Pixel+emulator attached: isError '2 devices attached (23181JEGR08034, emulator-5554) and no serial given'; find 'Settings': one Button bounds [954,11,1080,137]; tap 'No Such Button': isError naming the query + on-screen list; tap 'Workout': isError '5 elements match', nothing tapped"
},
{
"command": "python -m python_pkg.android_ui.workout_firebase_flow --serial 23181JEGR08034 --display self --account <kuhy-syncs account>",
"result": "pass",
"evidence": "exit 0 'workout-firebase-flow: connected'; screenshot shows 'Connected to Firebase.' and Disconnect. First attempt exposed a bug: the account chooser title is localised ('Wybierz konto'), so the chooser is now found by its GMS container id (red test first, then green); the app opens into the workout screen on launch, so one KEYCODE_BACK (phone_vd.sh key) put it on Home before the script ran"
},
{
"command": "python -m python_pkg.android_ui.workout_firebase_flow --serial 23181JEGR08034 --display self --negative",
"result": "pass",
"evidence": "exit 1; stderr: waited 3.0s for 'No Such Button': no element matches 'No Such Button'"
},
{
"command": "reorder run (SYNC/OFFLINE BACKUP swapped in a scratch screen-locker clone)",
"result": "fail",
"evidence": "NOT RUN: phone_deploy --slot prep exits 47 (shared-storage guard added 18:43 the same day): workout_app hard-codes /sdcard paths; swap reverted, scratch clone clean"
}
],
"risks": [
"Phone run on the Pixel 6a (C8-C12) not done yet: the device was not attached",
"Not registered yet (.mcp.json waits for kuhy's confirmation); the launching venv needs pip install -r meta/requirements.txt",
"display=self needs the server to share the caller's phone_lease identity: a stdio client that strips the env (MCP SDK default) got 'no virtual display'; under Claude Code the claude ancestor matches, but PHONE_LEASE_OWNER (board sessions) reaching the server is unverified"
],
"rollback": [
"git revert the commit",
"Validate: android-ui CLI tests still pass and the CLI dump works"
]
}

View File

@ -1,40 +0,0 @@
{
"intent": "The i3 shutdown block shows which earners can still push shutdown later, like the gaming-budget block does: '⏻ 22:00 +🧩1h +📖1h +🗂30m +⚙30m →max 23:00'.",
"scope": [
"linux_configuration/i3blocks/shutdown_countdown.sh",
"linux_configuration/tests/test_i3blocks_shutdown_hint.sh",
"linux_configuration/tests/test_i3blocks_efficiency.sh",
".github/workflows/shell-tests.yml",
"Non-goal: no hint in the override or SHUTDOWN states; no change to how shutdown is computed (screen-locker owns that)"
],
"changes": [
"earner_hint reads the enforcer's /api/budget .rules.earners and lists those with earned_seconds == 0, using bonus_seconds as the shutdown amount (equal for every earned_time earner today)",
"'→max 23:00' is appended when the listed bonuses exceed the headroom to SHUTDOWN_CEILING_MINUTES (default 1380, earned_time's ceiling); no hint at all at the ceiling",
"A dead or erroring server shows the bare time, which still comes from /etc/shutdown-schedule.conf",
"Efficiency test stubs curl as a dead server and allows 2 execs (bash + curl); new behaviour test runs in the shell-tests CI job"
],
"verification": [
{
"command": "bash linux_configuration/i3blocks/shutdown_countdown.sh (live enforcer, conf at 22:00)",
"result": "pass",
"evidence": "'⏻ 22:00 +🧩1h +📖1h +🗂30m +⚙30m →max 23:00'; after the LeetCode solve was applied: '⏻ 23:00'"
},
{
"command": "bash linux_configuration/tests/test_i3blocks_shutdown_hint.sh",
"result": "pass",
"evidence": "All shutdown hint tests passed (listing, ceiling note, all-earned, 1h30 padding, unknown earner label, server down / ok:false / non-JSON)"
},
{
"command": "bash linux_configuration/tests/test_i3blocks_efficiency.sh && bash linux_configuration/tests/test_i3blocks_gaming_budget.sh",
"result": "pass",
"evidence": "All i3blocks efficiency regression tests passed; All gaming_budget checks passed"
}
],
"risks": [
"If an earner's shutdown_minutes ever differs from its gaming_minutes, the hint shows the gaming amount; /api/budget does not expose shutdown amounts.",
"The 23:00 ceiling is duplicated from earned_time as a default; a registry change needs SHUTDOWN_CEILING_MINUTES or this default updated."
],
"rollback": [
"git revert this commit; the block shows the bare '⏻ HH:MM' again"
]
}

View File

@ -25,6 +25,7 @@ anthropic==2.8.0
beautifulsoup4==4.15.0
coverage==7.17.3
defusedxml==0.7.1
fastmcp==4.2.0
genanki==0.13.1
markdownify==1.2.3
numpy==2.5.5

View File

@ -21,6 +21,7 @@ darglint==1.8.1
dead==2.1.0
defusedxml==0.7.1
docformatter==1.7.8
fastmcp==4.2.0
fixit==2.2.1
flake8==7.5.0
flake8-annotations==3.3.1

View File

@ -67,6 +67,22 @@ ui.wait_for("Connected.", timeout=30)
- **Partial trees are retried,** and `keyboard_is_up()` distinguishes "element
absent" from "element behind the keyboard".
## MCP server
```bash
.venv/bin/python -m python_pkg.android_ui.mcp_server # stdio; registered in .mcp.json
```
Tools, 1:1 with `AndroidUi`: `dump`, `find`, `tap`, `type_into`,
`type_into_field`, `wait_for`, `scroll_to`, `focus`. No screenshot tool. Each
takes optional `serial` (default: the one attached device; with several it
errors and lists them) and `display` (default `self`, this session's virtual
display, never created by the server; `0` for the real screen).
Error contract: zero or several matches, an unchanged field, an adb failure,
a busy phone or a bad display all return `isError` naming the query and the
count (or before/after length). Typed text is never echoed.
## Known gaps
- A widget with no accessibility label cannot be targeted at all. The workout

View File

@ -0,0 +1,109 @@
"""Device, display and lease resolution for :mod:`python_pkg.android_ui.mcp_server`.
The same rules as the CLI (:func:`python_pkg.android_ui.cli._target` is reused
as-is), with two MCP-specific differences:
* the default display is ``"self"`` rather than the real screen, so an agent
drives its own ``phone_vd.sh`` display unless it asks for display 0; and
* with several devices and no serial the error names every serial, because
an agent can only pick one if it is told what there is to pick from.
Every failure is a :class:`~fastmcp.exceptions.ToolError`, which FastMCP
returns as an ``isError`` result with this message unmasked.
"""
from __future__ import annotations
import re
import shutil
import subprocess
from fastmcp.exceptions import ToolError
from python_pkg.android_ui._elements import _escape
from python_pkg.android_ui.cli import _target, _TargetError
from python_pkg.android_ui.driver import AndroidUi
from python_pkg.phone_lease import (
NoPhoneError,
PhoneBusyError,
acquire,
resolve_serial,
)
_ADB_DEVICES_TIMEOUT_S = 10.0
_REDACTED = "<typed text redacted>"
# The driver's adb error names the failed command, and for typing that command
# is `input -d N text <the text>`: cut everything after `text` on that line.
_INPUT_TEXT = re.compile(r"(input -d \S+ text ).*")
def attached_serials() -> list[str]:
"""Return the serials ``adb devices`` reports as ready, in its order."""
adb = shutil.which("adb")
if adb is None:
return []
try:
done = subprocess.run(
[adb, "devices"],
capture_output=True,
text=True,
timeout=_ADB_DEVICES_TIMEOUT_S,
check=False,
)
except OSError, subprocess.TimeoutExpired:
return []
serials = []
for line in done.stdout.splitlines()[1:]:
serial, _, state = line.partition("\t")
if state == "device":
serials.append(serial)
return serials
def _resolve(serial: str | None) -> str:
"""Return the device to drive; never guess between several."""
try:
return resolve_serial(serial)
except NoPhoneError as exc:
found = attached_serials()
if len(found) > 1:
msg = (
f"{len(found)} devices attached ({', '.join(found)}) and no "
"serial given: pass serial=<one of them>; refusing to guess"
)
raise ToolError(msg) from exc
raise ToolError(str(exc)) from exc
def connect(serial: str | None, display: str | int, command: str) -> AndroidUi:
"""Resolve device and display, take the lease, and return a driver.
``display`` is ``"self"`` (this session's virtual display), ``0`` / ``"0"``
(the real screen), or a virtual display id this session owns. No virtual
display is ever created here: a missing one is an error.
"""
resolved = _resolve(serial)
try:
target, scope = _target(resolved, str(display))
except _TargetError as exc:
raise ToolError(str(exc)) from exc
try:
acquire(resolved, f"android_ui mcp {command}", scope=scope)
except PhoneBusyError as exc:
raise ToolError(str(exc)) from exc
return AndroidUi(serial=resolved, display=target)
def redact(message: str, text: str) -> str:
"""Remove typed ``text`` from an error message, however it got there.
The driver's own "unchanged" messages give only lengths, but an adb
failure while typing quotes the command line. Never echo typed text: a
"does this look like a password?" heuristic is one that can fail open.
"""
if not text:
return message
cleaned = _INPUT_TEXT.sub(rf"\g<1>{_REDACTED}", message)
for form in (_escape(text), text):
cleaned = cleaned.replace(form, _REDACTED)
return cleaned

View File

@ -101,7 +101,8 @@ class TextEntryMixin(ABC):
if not changed:
msg = (
f"typed {len(text)} character(s) into editable field #{index} "
f"but its contents did not change — the tap did not focus it"
f"but its contents did not change (length {len(before)} before "
f"and after) — the tap did not focus it"
)
raise UiAutomationError(msg)

View File

@ -223,6 +223,7 @@ class AndroidUi(ScrollMixin, TextEntryMixin):
# A password field renders as bullets, so compare length, not content.
msg = (
f"typed {len(text)} character(s) into {query!r} but its contents "
f"did not change — the tap probably did not focus the field"
f"did not change (length {len(before)} before and after) — the tap "
f"probably did not focus the field"
)
raise UiAutomationError(msg)

View File

@ -0,0 +1,201 @@
"""MCP server over :class:`~python_pkg.android_ui.driver.AndroidUi`, on stdio.
python3 -m python_pkg.android_ui.mcp_server
Eight tools, one per driver method, adding no behaviour: ``dump``, ``find``,
``tap``, ``type_into``, ``type_into_field``, ``wait_for``, ``scroll_to`` and
``focus``. No screenshot tool, on purpose: an element list is cheaper and can
be asserted on.
Error contract: every :class:`~python_pkg.android_ui.driver.UiAutomationError`
(zero or several matches, a field that did not change, an adb failure) and
every device/display/lease refusal is an ``isError`` result whose message
names the query. Nothing failed is ever returned success-shaped. Typed text is
never echoed back in an error.
Every tool takes optional ``serial`` (default: the one attached device) and
``display`` (default ``"self"``, this session's ``phone_vd.sh`` display; pass
``0`` for the real screen). See :mod:`python_pkg.android_ui._mcp_target`.
Nothing here may write to stdout: it carries the MCP protocol.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from fastmcp import FastMCP
from fastmcp.exceptions import ToolError
from python_pkg.android_ui._mcp_target import connect, redact
from python_pkg.android_ui.driver import UiAutomationError
if TYPE_CHECKING:
from collections.abc import Callable
from python_pkg.android_ui.driver import AndroidUi, UiElement
mcp = FastMCP(
"android-ui",
instructions=(
"Drive an Android app by element (text, content-desc, hint, "
"resource-id), never by coordinates. A query matching 0 or >1 "
"elements is an error: refine it or pass exact=true. Default display "
"is this session's virtual display ('self'); pass display=0 for the "
"real screen."
),
)
def _element(element: UiElement) -> dict[str, Any]:
"""Return one element as plain data, bounds and tap centre included."""
return {
"label": element.label,
"text": element.text,
"content_desc": element.content_desc,
"hint": element.hint,
"resource_id": element.resource_id,
"class_name": element.class_name,
"bounds": list(element.bounds),
"center": list(element.center),
"enabled": element.enabled,
"focused": element.focused,
}
def _drive[T](
command: str,
serial: str | None,
display: str | int,
action: Callable[[AndroidUi], T],
typed: str = "",
) -> T:
"""Run ``action`` on a resolved driver, turning failures into ToolError."""
ui = connect(serial, display, command)
try:
return action(ui)
except UiAutomationError as exc:
raise ToolError(redact(str(exc), typed)) from exc
@mcp.tool
def dump(serial: str | None = None, display: str | int = "self") -> list[str]:
"""List every labelled element on screen, one line each."""
return _drive("dump", serial, display, lambda ui: [str(e) for e in ui.dump()])
@mcp.tool
def find(
query: str,
*,
exact: bool = False,
serial: str | None = None,
display: str | int = "self",
) -> dict[str, Any]:
"""Return the single element matching ``query`` without touching it."""
return _drive(
"find", serial, display, lambda ui: _element(ui.find(query, exact=exact))
)
@mcp.tool
def tap(
query: str,
*,
exact: bool = False,
timeout: float = 15.0,
serial: str | None = None,
display: str | int = "self",
) -> dict[str, Any]:
"""Tap the one element matching ``query``; error on 0 or >1 matches."""
return _drive(
"tap",
serial,
display,
lambda ui: _element(ui.tap(query, exact=exact, timeout=timeout)),
)
@mcp.tool
def type_into(
query: str,
text: str,
*,
exact: bool = False,
timeout: float = 15.0,
serial: str | None = None,
display: str | int = "self",
) -> str:
"""Type ``text`` into the field matching ``query``; error if it did not change."""
_drive(
"type",
serial,
display,
lambda ui: ui.type_into(query, text, exact=exact, timeout=timeout),
typed=text,
)
return f"typed {len(text)} character(s) into {query!r} and verified"
@mcp.tool
def type_into_field(
index: int,
text: str,
serial: str | None = None,
display: str | int = "self",
) -> str:
"""Type into the ``index``-th editable field (top to bottom); verified."""
_drive(
"type",
serial,
display,
lambda ui: ui.type_into_field(index, text),
typed=text,
)
return f"typed {len(text)} character(s) into editable field #{index} and verified"
@mcp.tool
def wait_for(
query: str,
*,
timeout: float = 15.0,
exact: bool = False,
serial: str | None = None,
display: str | int = "self",
) -> dict[str, Any]:
"""Poll until ``query`` matches exactly one element, or error on timeout."""
return _drive(
"wait",
serial,
display,
lambda ui: _element(ui.wait_for(query, timeout=timeout, exact=exact)),
)
@mcp.tool
def scroll_to(
query: str,
*,
exact: bool = False,
max_swipes: int = 8,
serial: str | None = None,
display: str | int = "self",
) -> dict[str, Any]:
"""Swipe up until ``query`` is on screen, then return it."""
return _drive(
"scroll-to",
serial,
display,
lambda ui: _element(ui.scroll_to(query, exact=exact, max_swipes=max_swipes)),
)
@mcp.tool
def focus(serial: str | None = None, display: str | int = "self") -> str:
"""Return the focused window (package/activity)."""
return _drive("focus", serial, display, lambda ui: ui.current_focus())
if __name__ == "__main__":
mcp.run(transport="stdio", show_banner=False)

View File

@ -0,0 +1,244 @@
"""Tests for :mod:`python_pkg.android_ui.mcp_server`.
The server runs in-process (FastMCP's in-memory client) over the REAL
:class:`AndroidUi`, wired to the scripted adb stand-in, so every ``isError``
asserted here comes from the driver's own verification -- not from a mock
that was told to fail. No real adb runs.
"""
from __future__ import annotations
import asyncio
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from unittest.mock import MagicMock, patch
from fastmcp import Client
import pytest
from python_pkg.android_ui import mcp_server
from python_pkg.android_ui.driver import AndroidUi, UiAutomationError
from python_pkg.android_ui.tests.conftest import FakeDevice, node, tree
from python_pkg.phone_guard.vd_state import SessionDisplay
from python_pkg.phone_lease import PhoneBusyError
if TYPE_CHECKING:
from collections.abc import Iterator
from mcp.types import CallToolResult
TARGET = "python_pkg.android_ui._mcp_target"
CLI = "python_pkg.android_ui.cli"
TOOLS = {
"dump",
"find",
"tap",
"type_into",
"type_into_field",
"wait_for",
"scroll_to",
"focus",
}
# Real screen, so FakeDevice's display-0 bookkeeping applies.
REAL = {"display": 0}
FIELD = "android.widget.EditText"
@dataclass
class Rig:
"""The scripted adb plus the lease mock every tool call goes through."""
adb: FakeDevice
acquire: MagicMock
@pytest.fixture
def rig(monkeypatch: pytest.MonkeyPatch) -> Iterator[Rig]:
"""One phone (SER1), no virtual display, a free lease, a fake adb."""
fake = FakeDevice()
def build(serial: str | None = None, display: int | None = None) -> AndroidUi:
ui = AndroidUi(serial=serial, display=display, settle_seconds=0.0)
monkeypatch.setattr(ui, "_run", fake.run)
return ui
with (
patch(f"{TARGET}.resolve_serial", side_effect=lambda s=None: s or "SER1"),
patch(f"{TARGET}.acquire") as acquire,
patch(f"{TARGET}.AndroidUi", side_effect=build),
patch(f"{CLI}.owner_id", return_value="claude:1"),
patch(f"{CLI}.own_display", return_value=None),
patch(f"{CLI}.display_owner", return_value=None),
):
yield Rig(fake, acquire)
def call(name: str, args: dict[str, Any] | None = None) -> CallToolResult:
"""Call one tool through a real MCP client session."""
async def _go() -> CallToolResult:
async with Client(mcp_server.mcp) as client:
return await client.call_tool(name, args or {}, raise_on_error=False)
return asyncio.run(_go())
def error_text(result: CallToolResult) -> str:
"""Return the error message of a failed call, asserting it failed."""
assert result.is_error, f"expected isError, got {result}"
return " ".join(getattr(c, "text", "") for c in result.content)
def _two_backs() -> str:
return tree(
node("Back", bounds="[0,0][100,50]"),
node("OFFLINE BACKUP", bounds="[0,100][100,150]"),
)
class TestToolList:
def test_exactly_the_eight_driver_tools(self) -> None:
async def _go() -> set[str]:
async with Client(mcp_server.mcp) as client:
return {t.name for t in await client.list_tools()}
assert asyncio.run(_go()) == TOOLS
def test_every_tool_takes_serial_and_display(self) -> None:
async def _go() -> list[dict[str, Any]]:
async with Client(mcp_server.mcp) as client:
return [t.inputSchema for t in await client.list_tools()]
for schema in asyncio.run(_go()):
assert {"serial", "display"} <= set(schema["properties"])
class TestHappyPaths:
def test_dump_lists_elements(self, rig: Rig) -> None:
rig.adb.trees = [tree(node("Settings"), node("Home"))]
result = call("dump", REAL)
assert not result.is_error
assert any("Settings" in line for line in result.data)
def test_find_returns_bounds_without_tapping(self, rig: Rig) -> None:
rig.adb.trees = [tree(node("Settings", bounds="[10,20][110,70]"))]
result = call("find", {"query": "Settings", **REAL})
assert result.data["bounds"] == [10, 20, 110, 70]
assert result.data["center"] == [60, 45]
assert rig.adb.taps() == []
def test_tap_taps_the_one_match(self, rig: Rig) -> None:
rig.adb.trees = [tree(node("Connect", bounds="[0,0][100,50]"))]
result = call("tap", {"query": "Connect", **REAL})
assert not result.is_error
assert rig.adb.taps() == [(50, 25)]
def test_exact_disambiguates(self, rig: Rig) -> None:
rig.adb.trees = [_two_backs()]
result = call("tap", {"query": "Back", "exact": True, **REAL})
assert not result.is_error
assert rig.adb.taps() == [(50, 25)]
def test_wait_for_and_scroll_to(self, rig: Rig) -> None:
rig.adb.trees = [tree(node("Connected."))]
assert call("wait_for", {"query": "Connected.", **REAL}).data["label"] == (
"Connected."
)
assert not call("scroll_to", {"query": "Connected.", **REAL}).is_error
def test_focus_names_the_window(self, rig: Rig) -> None:
assert call("focus", REAL).data == "com.example/com.example.Main"
def test_type_into_verified(self, rig: Rig) -> None:
before = node(desc="Email", cls=FIELD)
after = node("a@b.c", desc="Email", cls=FIELD)
rig.adb.trees = [tree(before), tree(before), tree(after)]
result = call("type_into", {"query": "Email", "text": "a@b.c", **REAL})
assert not result.is_error
def test_type_into_field_verified(self, rig: Rig) -> None:
rig.adb.trees = [tree(node(cls=FIELD)), tree(node("a@b.c", cls=FIELD))]
result = call("type_into_field", {"index": 0, "text": "a@b.c", **REAL})
assert not result.is_error
class TestErrorContract:
def test_tap_with_no_match_is_an_error(self, rig: Rig) -> None:
rig.adb.trees = [tree(node("Settings"))]
result = call("tap", {"query": "No Such Button", "timeout": 0.2, **REAL})
message = error_text(result)
assert "No Such Button" in message
assert "Settings" in message # what IS on screen
assert rig.adb.taps() == []
def test_tap_with_two_matches_is_an_error(self, rig: Rig) -> None:
rig.adb.trees = [_two_backs()]
message = error_text(call("tap", {"query": "Back", "timeout": 0.2, **REAL}))
assert "2 elements match 'Back'" in message
assert rig.adb.taps() == []
def test_unchanged_type_into_is_an_error(self, rig: Rig) -> None:
rig.adb.trees = [tree(node("old", desc="Email", cls=FIELD))]
result = call("type_into", {"query": "Email", "text": "hunter2!", **REAL})
message = error_text(result)
assert "'Email'" in message
assert "length 3 before and after" in message
assert "hunter2" not in message
def test_unchanged_type_into_field_is_an_error(self, rig: Rig) -> None:
rig.adb.trees = [tree(node(cls=FIELD))]
result = call("type_into_field", {"index": 0, "text": "hunter2!", **REAL})
message = error_text(result)
assert "field #0" in message
assert "length 0 before and after" in message
assert "hunter2" not in message
def test_adb_failure_while_typing_never_echoes_the_text(
self, rig: Rig, monkeypatch: pytest.MonkeyPatch
) -> None:
rig.adb.trees = [tree(node(cls=FIELD))]
real_run = rig.adb.run
def failing(*args: str, timeout: float = 30.0) -> str:
if "text" in args:
msg = f"adb failed (1): {' '.join(args)}\nerror: hunter2secret"
raise UiAutomationError(msg)
return real_run(*args, timeout=timeout)
monkeypatch.setattr(rig.adb, "run", failing)
result = call("type_into_field", {"index": 0, "text": "hunter2secret", **REAL})
message = error_text(result)
assert "hunter2" not in message
assert "redacted" in message
def test_lease_held_elsewhere_is_an_error(self, rig: Rig) -> None:
rig.acquire.side_effect = PhoneBusyError("held by claude:9")
message = error_text(call("dump", REAL))
assert "held by claude:9" in message
assert rig.adb.calls == []
class TestDisplay:
def test_default_is_self_and_errors_without_a_virtual_display(
self, rig: Rig
) -> None:
message = error_text(call("dump"))
assert "no virtual display" in message
assert rig.adb.calls == [] # nothing created, nothing driven
def test_self_drives_the_sessions_own_display(self, rig: Rig) -> None:
mine = SessionDisplay(owner_dir="d", display=7, sf="sf", package="com.example")
with (
patch(f"{CLI}.own_display", return_value=mine),
patch("python_pkg.android_ui._a11y_helper.dump_display") as dump_vd,
):
dump_vd.return_value = tree(node("Settings"))
result = call("find", {"query": "Settings"})
assert not result.is_error
assert dump_vd.call_args.args[1] == 7
assert rig.acquire.call_args.kwargs["scope"] == "pkg:com.example"
def test_display_zero_is_the_real_screen(self, rig: Rig) -> None:
call("dump", {"display": "0"})
assert rig.acquire.call_args.kwargs["scope"] == "pkg:__screen__"
assert any(c[:2] == ("shell", "uiautomator") for c in rig.adb.calls)

View File

@ -0,0 +1,118 @@
"""Tests for :mod:`python_pkg.android_ui._mcp_target` and the server entry point.
Device selection must never guess: with several devices and no serial the
call fails, naming every serial, before any driver exists.
"""
from __future__ import annotations
import runpy
import subprocess
import sys
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
from fastmcp.exceptions import ToolError
import pytest
from python_pkg.android_ui import _mcp_target
from python_pkg.phone_lease import NoPhoneError
if TYPE_CHECKING:
from collections.abc import Iterator
TARGET = "python_pkg.android_ui._mcp_target"
DEVICES = (
"List of devices attached\n"
"23181JEGR08034\tdevice\n"
"emulator-5554\tdevice\n"
"OFFLINE1\toffline\n"
"\n"
)
def _adb_devices(stdout: str) -> MagicMock:
return MagicMock(return_value=subprocess.CompletedProcess([], 0, stdout, ""))
@pytest.fixture(autouse=True)
def _adb_on_path() -> Iterator[None]:
with patch(f"{TARGET}.shutil.which", return_value="/usr/bin/adb"):
yield
class TestAttachedSerials:
def test_lists_only_ready_devices(self) -> None:
with patch(f"{TARGET}.subprocess.run", _adb_devices(DEVICES)):
assert _mcp_target.attached_serials() == [
"23181JEGR08034",
"emulator-5554",
]
def test_adb_that_cannot_run_means_no_devices(self) -> None:
with patch(f"{TARGET}.subprocess.run", side_effect=OSError("no adb")):
assert _mcp_target.attached_serials() == []
def test_adb_not_installed_means_no_devices(self) -> None:
with (
patch(f"{TARGET}.shutil.which", return_value=None),
patch(f"{TARGET}.subprocess.run") as run,
):
assert _mcp_target.attached_serials() == []
run.assert_not_called()
class TestResolve:
def test_two_devices_and_no_serial_lists_both(self) -> None:
with (
patch(f"{TARGET}.resolve_serial", side_effect=NoPhoneError("several")),
patch(f"{TARGET}.subprocess.run", _adb_devices(DEVICES)),
patch(f"{TARGET}.AndroidUi") as driver,
pytest.raises(ToolError) as caught,
):
_mcp_target.connect(None, "0", "tap")
assert "23181JEGR08034" in str(caught.value)
assert "emulator-5554" in str(caught.value)
driver.assert_not_called()
def test_no_device_keeps_the_lease_message(self) -> None:
with (
patch(f"{TARGET}.resolve_serial", side_effect=NoPhoneError("no phone")),
patch(f"{TARGET}.subprocess.run", _adb_devices("List\n")),
pytest.raises(ToolError, match="no phone"),
):
_mcp_target.connect(None, "0", "tap")
def test_explicit_serial_drives_that_device(self) -> None:
with (
patch(f"{TARGET}.acquire") as acquire,
patch(f"{TARGET}.AndroidUi") as driver,
):
_mcp_target.connect("emulator-5554", 0, "dump")
assert acquire.call_args.args[0] == "emulator-5554"
driver.assert_called_once_with(serial="emulator-5554", display=None)
def test_bad_display_is_a_tool_error(self) -> None:
with (
patch(f"{TARGET}.resolve_serial", return_value="SER1"),
patch("python_pkg.android_ui.cli.own_display", return_value=None),
patch("python_pkg.android_ui.cli.owner_id", return_value="claude:1"),
pytest.raises(ToolError, match="display id or 'self'"),
):
_mcp_target.connect(None, "main", "dump")
class TestRedact:
def test_empty_text_leaves_the_message_alone(self) -> None:
assert _mcp_target.redact("no element", "") == "no element"
def test_text_outside_the_adb_command_is_still_removed(self) -> None:
assert "s3cr3t" not in _mcp_target.redact("stderr: s3cr3t", "s3cr3t")
class TestEntryPoint:
def test_main_runs_stdio(self) -> None:
sys.modules.pop("python_pkg.android_ui.mcp_server", None)
with patch("fastmcp.FastMCP.run") as run:
runpy.run_module("python_pkg.android_ui.mcp_server", run_name="__main__")
run.assert_called_once_with(transport="stdio", show_banner=False)

View File

@ -0,0 +1,161 @@
"""Tests for :mod:`python_pkg.android_ui.workout_firebase_flow`.
The driver is a mock: what is under test is the flow's decisions (already
connected, account chooser, credential screen, timeout) and its exit codes,
not adb. No real device is touched.
"""
from __future__ import annotations
import runpy
import sys
from typing import TYPE_CHECKING
from unittest.mock import MagicMock, patch
from fastmcp.exceptions import ToolError
import pytest
from python_pkg.android_ui import workout_firebase_flow as flow
from python_pkg.android_ui.driver import ElementNotFoundError, UiElement
if TYPE_CHECKING:
from collections.abc import Iterator
MOD = "python_pkg.android_ui.workout_firebase_flow"
TARGET = "python_pkg.android_ui._mcp_target"
_CHOOSER_ID = "com.google.android.gms:id/google_sign_in_container"
@pytest.fixture(autouse=True)
def _no_sleep() -> Iterator[None]:
with patch(f"{MOD}.time.sleep"):
yield
def _el(
text: str = "", *, cls: str = "android.widget.TextView", **kw: str
) -> UiElement:
"""Build one element; ``hint``/``res``/``desc`` override the empty defaults."""
return UiElement(
text=text,
content_desc=kw.get("desc", ""),
resource_id=kw.get("res", ""),
class_name=cls,
bounds=(0, 0, 100, 50),
enabled=True,
focused=False,
hint=kw.get("hint", ""),
)
def _ui(*screens: list[UiElement]) -> MagicMock:
"""A driver whose successive dumps return ``screens`` (last one repeats)."""
ui = MagicMock()
seq = list(screens)
ui.dump.side_effect = lambda: seq.pop(0) if len(seq) > 1 else seq[0]
return ui
CONNECTED = [_el("Disconnect"), _el("Connected to Firebase.")]
SIGN_IN = [_el("Sign in with Google")]
class TestFlow:
def test_navigates_by_label_and_connects(self) -> None:
ui = _ui(SIGN_IN, SIGN_IN, CONNECTED)
assert flow.run_flow(ui, account=None, timeout=5) == "connected"
taps = [c.args[0] for c in ui.tap.call_args_list]
assert taps == [flow.SETTINGS, flow.SYNC_ROW, flow.SIGN_IN]
ui.scroll_to.assert_called_once_with(flow.SYNC_ROW)
def test_already_connected_taps_nothing_on_the_sync_screen(self) -> None:
ui = _ui(CONNECTED)
assert flow.run_flow(ui, account=None, timeout=5) == "already connected"
assert flow.SIGN_IN not in [c.args[0] for c in ui.tap.call_args_list]
def test_account_chooser_picks_the_given_account(self) -> None:
# As the Pixel shows it: a Polish title, so only the GMS id is stable.
chooser = [_el("Wybierz konto", res=_CHOOSER_ID), _el("me@example.com")]
ui = _ui(SIGN_IN, chooser, CONNECTED)
assert flow.run_flow(ui, account="me@example.com", timeout=5) == "connected"
ui.tap.assert_called_with("me@example.com", exact=True)
def test_account_chooser_without_account_hands_over(self) -> None:
ui = _ui(SIGN_IN, [_el("Wybierz konto", res=_CHOOSER_ID)])
with pytest.raises(flow.HandOverError, match="--account"):
flow.run_flow(ui, account=None, timeout=5)
def test_password_field_hands_over_and_types_nothing(self) -> None:
pw = [_el(cls="android.widget.EditText", hint="Enter your password")]
ui = _ui(SIGN_IN, pw)
with pytest.raises(flow.HandOverError, match="secret"):
flow.run_flow(ui, account=None, timeout=5)
ui.type_into.assert_not_called()
ui.type_into_field.assert_not_called()
def test_non_secret_text_field_does_not_hand_over(self) -> None:
email = [_el(cls="android.widget.EditText", hint="Email or phone")]
ui = _ui(SIGN_IN, email, CONNECTED)
assert flow.run_flow(ui, account=None, timeout=5) == "connected"
def test_never_connecting_times_out_naming_the_query(self) -> None:
ui = _ui(SIGN_IN)
with (
patch(f"{MOD}.time.monotonic", side_effect=[0.0, 0.0, 99.0]),
pytest.raises(ElementNotFoundError, match="Disconnect"),
):
flow.run_flow(ui, account=None, timeout=5)
class TestMain:
def test_success_exits_zero(self, capsys: pytest.CaptureFixture[str]) -> None:
with (
patch(f"{MOD}.connect") as connect,
patch(f"{MOD}.run_flow", return_value="connected"),
):
assert flow.main(["--serial", "PIX", "--display", "0"]) == 0
connect.assert_called_once_with("PIX", "0", "workout-firebase-flow")
assert "connected" in capsys.readouterr().out
def test_negative_step_exits_nonzero_naming_the_query(
self, capsys: pytest.CaptureFixture[str]
) -> None:
ui = MagicMock()
ui.tap.side_effect = ElementNotFoundError("no element matches 'No Such Button'")
with patch(f"{MOD}.connect", return_value=ui):
assert flow.main(["--negative"]) == 1
ui.tap.assert_called_once_with(flow.NEGATIVE, exact=True, timeout=3.0)
assert "No Such Button" in capsys.readouterr().err
def test_bad_target_exits_four(self, capsys: pytest.CaptureFixture[str]) -> None:
with patch(f"{MOD}.connect", side_effect=ToolError("2 devices attached")):
assert flow.main([]) == flow.EXIT_TARGET
assert "2 devices attached" in capsys.readouterr().err
def test_hand_over_exits_six(self, capsys: pytest.CaptureFixture[str]) -> None:
with (
patch(f"{MOD}.connect"),
patch(f"{MOD}.run_flow", side_effect=flow.HandOverError("secret")),
):
assert flow.main([]) == flow.EXIT_HAND_OVER
assert "take over" in capsys.readouterr().err
def test_negative_button_that_exists_still_fails(
self, capsys: pytest.CaptureFixture[str]
) -> None:
with patch(f"{MOD}.connect", return_value=MagicMock()):
assert flow.main(["--negative"]) == flow.EXIT_ELEMENT
assert "No Such Button" in capsys.readouterr().err
def test_module_entry_point_exits_with_main(self) -> None:
# patch.dict restores the original module afterwards, so the ``flow``
# imported above stays the one later tests patch.
with (
patch.dict(sys.modules),
patch.object(sys, "argv", ["workout-firebase-flow"]),
patch(f"{TARGET}.connect", side_effect=ToolError("no phone")),
):
sys.modules.pop(MOD)
with pytest.raises(SystemExit) as caught:
runpy.run_module(MOD, run_name="__main__")
assert caught.value.code == flow.EXIT_TARGET

View File

@ -0,0 +1,140 @@
"""Drive workout_app's "connect Firebase" flow by label, with no coordinates.
python3 -m python_pkg.android_ui.workout_firebase_flow --serial 23181JEGR08034 \
[--display self] [--account EMAIL]
python3 -m python_pkg.android_ui.workout_firebase_flow --serial ... --negative
Home -> Settings (gear tooltip) -> scroll to the "Firebase sync" row -> Sync
settings -> "Sign in with Google" -> wait for "Disconnect". Every workout_app
label lives in this file and nowhere else, so a layout change (reordering the
SYNC and OFFLINE BACKUP sections) needs no edit here -- only a renamed label
does, and then the run fails naming it.
Exit codes: 0 connected (or already was), 1 an element was missing or
ambiguous (stderr names the query), 4 bad device/display/lease, 6 a screen
wants a password, 2FA code or an account choice: a human takes over. It never
types anything; ``--negative`` taps a button that does not exist, to prove a
missing element exits non-zero.
"""
from __future__ import annotations
import argparse
import sys
import time
from typing import TYPE_CHECKING
from fastmcp.exceptions import ToolError
from python_pkg.android_ui._mcp_target import connect
from python_pkg.android_ui.driver import ElementNotFoundError, UiAutomationError
if TYPE_CHECKING:
from python_pkg.android_ui.driver import AndroidUi, UiElement
# workout_app / sync_settings_ui labels (sync_settings_ui-v0.5.2).
SETTINGS = "Settings" # home_screen.dart: the gear's tooltip
SYNC_ROW = "Firebase sync" # settings_screen_sections.dart: the row's subtitle
SIGN_IN = "Sign in with Google"
CONNECTED = "Disconnect" # only rendered while Firebase is connected
# Google's account chooser: its title follows the phone's locale ("Wybierz
# konto" on the Pixel), so it is found by its GMS container id instead.
CHOOSER = "com.google.android.gms:id/google_sign_in_container"
NEGATIVE = "No Such Button"
# Text that means a screen wants a secret. Matched on editable fields only, so
# a label such as "Use the account password instead" does not trip it.
_SECRET_MARKERS = ("password", "passcode", "verification code", "2-step", "pin")
EXIT_ELEMENT = 1
EXIT_TARGET = 4
EXIT_HAND_OVER = 6
_POLL_S = 1.0
_NEGATIVE_TIMEOUT_S = 3.0
class HandOverError(RuntimeError):
"""A screen needs a human: a secret, a 2FA code, or an account choice."""
def _connected(screen: list[UiElement]) -> bool:
return any(e.matches(CONNECTED, exact=True) for e in screen)
def _asks_for_secret(screen: list[UiElement]) -> bool:
for element in screen:
if not element.class_name.endswith("EditText"):
continue
names = f"{element.hint} {element.content_desc} {element.resource_id}".lower()
if any(marker in names for marker in _SECRET_MARKERS):
return True
return False
def run_flow(ui: AndroidUi, *, account: str | None, timeout: float) -> str:
"""Run the flow from the home screen; return how it ended."""
ui.tap(SETTINGS, exact=True)
# Below the fold on a phone; scrolling to it is what makes the run survive
# a section reorder without a coordinate.
ui.scroll_to(SYNC_ROW)
ui.tap(SYNC_ROW)
if _connected(ui.dump()):
return "already connected"
ui.tap(SIGN_IN, exact=True)
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
screen = ui.dump()
if _connected(screen):
return "connected"
if _asks_for_secret(screen):
msg = "the sign-in asks for a password or 2FA code (a secret)"
raise HandOverError(msg)
if any(e.matches(CHOOSER, exact=True) for e in screen):
if account is None:
msg = "the Google account chooser is up: pass --account <email>"
raise HandOverError(msg)
ui.tap(account, exact=True)
time.sleep(_POLL_S)
msg = f"waited {timeout}s for {CONNECTED!r} after tapping {SIGN_IN!r}"
raise ElementNotFoundError(msg)
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(prog="workout-firebase-flow")
parser.add_argument("--serial", help="device serial (required with 2+ devices)")
parser.add_argument(
"--display", default="self", help="'self' (default) or 0 for the real screen"
)
parser.add_argument("--account", help="Google account to pick in the chooser")
parser.add_argument("--timeout", type=float, default=60.0)
parser.add_argument(
"--negative", action="store_true", help=f"tap {NEGATIVE!r} and fail"
)
return parser
def main(argv: list[str] | None = None) -> int:
"""Run the flow (or the negative step); return a process exit code."""
args = _parser().parse_args(argv)
try:
ui = connect(args.serial, args.display, "workout-firebase-flow")
except ToolError as exc:
sys.stderr.write(f"workout-firebase-flow: {exc}\n")
return EXIT_TARGET
try:
if args.negative:
ui.tap(NEGATIVE, exact=True, timeout=_NEGATIVE_TIMEOUT_S)
msg = f"negative step: {NEGATIVE!r} exists on screen, the check is void"
raise ElementNotFoundError(msg)
outcome = run_flow(ui, account=args.account, timeout=args.timeout)
except UiAutomationError as exc:
sys.stderr.write(f"workout-firebase-flow: {exc}\n")
return EXIT_ELEMENT
except HandOverError as exc:
sys.stderr.write(f"workout-firebase-flow: {exc}; take over on the phone\n")
return EXIT_HAND_OVER
sys.stdout.write(f"workout-firebase-flow: {outcome}\n")
return 0
if __name__ == "__main__":
sys.exit(main())