Skip to content

feat: wait until a window, dialog, element, text or app condition holds - #70

Merged
iamnbutler merged 1 commit into
mainfrom
feat/wait
Oct 17, 2026
Merged

iamnbutler merged 1 commit into
mainfrom
feat/wait

Conversation

@iamnbutler

Copy link
Copy Markdown
Contributor

Adds desktop_wait, the 42nd tool. It is a bounded, read-only wait that returns as soon as one condition holds (matched, with what matched), or timed_out with the last observation.

Conditions (until, one per call)

  • Window: window_appears: an app's window, by bundle_id or pid, optionally with an exact title or a title_contains substring. Only windows that are on screen, minimized or on a Space count, so Finder's off-screen menu-bar strips don't match. window_gone: window pid + window is no longer listed, or its app quit.
  • Dialog: dialog_appears / dialog_gone, in app pid or its window window, as desktop_dialog reads it.
  • Element: element_appears / element_gone in window pid + window. element is an exact role (AX role or the listed role) plus at least one of an exact title (label), value or identifier. An optional scope restricts the search to one container.
  • Text: text_changes / text_equals / text_contains. The text is the window's or a scope's text (as desktop_read reads it), or the value of the one element element matches.
  • App: app_running / app_frontmost / app_quit, by bundle_id or pid.

How it works

  • Polling. It lives client-side in src/wait.ts and polls existing reads with backoff: 200 ms, ×1.5, up to 2 s between polls. Each poll is a separate client process, so no lane lock is held between polls. MCP runs it like the other reads, outside the action queue.
  • Timeout. timeout_ms defaults to 10 s, with a cap of 45 s. The final inspection after an element appears gets up to 10 s more, so the worst case (about 55 s) stays under the 60 s default request timeout of the MCP TypeScript SDK. The wait doesn't use the client's 30 s per-request deadline; each poll still has its own.
  • No input. Nothing activates an app.
  • Element and text polls take no screenshot. I first used the same observation call as desktop_read, then found that it captures a screenshot every time: it fails with "Capture failed" while the session is locked. Polling it would put ScreenCaptureKit traffic on every other agent (the fix: wait out a quarantined ScreenCaptureKit gate and explain the refusal #63 problem).
    • The new client subcommand elements (native/sources/client/elements.swift) uses upstream's existing inspectAccessibilityTree instead. That is a detached, read-only AX traversal that needs only Accessibility, stores no snapshot and takes a traversal budget; the poll uses desktop_read's large budget.
    • The subcommand returns only the matching elements (up to 20, with a count) or the text, plus truncation.
    • The runtime now allows .inspectAccessibilityTree. It is an existing upstream operation, so there is no new Bridge operation or patch.
    • The scope is resolved with the same scopeSource / locateScope as scoped inspect and read.
  • Absence is reported only when certain:
    • An element is never gone when the traversal was cut short (depth, count, children, deadline or incomplete AX read).
    • A window or app is never gone when the inventory is partial.
    • A dialog is gone only when the dialog read says "No dialog matched…" or "No active dialog…" and the app's window listing shows Accessibility reached every window. The dialog read skips windows it can't reach and then reports "no dialog". I saw that happen live while locked, and this guard now prevents the false positive.
    • "Dialog target is ambiguous" (several dialogs) counts as present.
  • Results use the read tools' shapes. Windows and apps are desktop_windows / desktop_apps rows with their targets. A dialog is the desktop_dialog listing.
    • When an element appears, the wait runs one desktop_inspect (with the scope, if given) and returns inspection.snapshot_id, the matching elements with their IDs from that snapshot, and the screenshot, ready for desktop_click and the other actions.
    • When the inspection doesn't list the element (an unscoped inspect stops at 200 elements), snapshot_id is null, and the note says to inspect, with a scope, before acting.
  • Validation refuses arguments a condition doesn't use.

The docs table and a section are in docs/tools.md; the tool count is now 42.

Checks

  • npm run check passes.
  • Native client and signed dev app build (Developer ID).
  • Live, against the dev app (screen locked):
    • app_running matched on the first poll, in 170 ms.
    • app_frontmost and app_quit time out with the last observation.
    • window_appears for Finder returns the real windows with targets, and no longer the off-screen strips.
    • dialog_appears for TextEdit times out with dialog: null.
    • dialog_gone stays unmatched while Accessibility enumeration is incomplete.
    • element_* and text_* polls take no screenshot. While locked they read ax_incomplete_read with 0 elements, and element_gone correctly doesn't match.
  • pending: screen locked. Live checks still to run:
    1. window_appears with title_contains, started before desktop_open of a file in TextEdit, matches the new window.
    2. window_gone matches after desktop_close of that window.
    3. element_appears for TextEdit's close button (AXButton, identifier _XCUI:CloseWindow) returns a snapshot, and desktop_click's argument check accepts that element ID and snapshot (no click needed).
    4. text_contains on the window text and on an element matcher (AXTextArea, …) after desktop_type adds text. text_changes reports before.
    5. dialog_appears after Command-S on an untitled TextEdit document (Save sheet), then dialog_gone after answering it with Cancel.
    6. app_quit after desktop_quit of TextEdit.
    7. A scope search on a long web page in Safari, for an element deeper than 200 elements.
    8. While a wait polls, another agent's action interleaves; confirm with lsof on ~/.peekaboo locks that none is held between polls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant