Playwright for desktop apps, built for coding agents.

Available on macOS · Windows · Linux.

---

A fast CLI/MCP control layer that lets agents like Claude, Codex, opencode, and Gemini inspect and operate desktop apps through structured UI data, then save successful flows as replayable tests.

Status

Available on macOS, Windows, and Linux. macOS is the primary target and runs the full stack — Accessibility tree first, screenshots as fallback.

Windows ships a standalone binary (bundled Swift runtime, no toolchain needed) with input/screenshot/apps (Win32 SendInput, GDI BitBlt/PrintWindow → real PNG, EnumWindows), find-text / click-text via WinRT OCR, and observe / tree / find / click by selector via UI Automation. Roles are mapped onto the same names macOS uses, so guiport click 'button[name="Send"]' means the same thing on both — a script written against a Mac runs unchanged on Windows. record is still pending; track it under the windows label.

Linux ships a standalone binary (static Swift stdlib, glibc 2.35+) with the same shape: shell-out to xdotool/wmctrl/scrot on X11 and ydotool/grim on Wayland for input + screenshot, /proc + wmctrl for app enumeration. AT-SPI2-backed tree/observe/find and tesseract OCR throw atspi_pending / ocr_pending until those bindings land. Track under linux.

Prebuilt binaries for all three platforms are attached to every release.

Support matrix

Which verbs work on each platform today. Pending verbs don't crash — they exit non-zero with a labeled, actionable error (the code is in the table), so agents can branch on it. CI builds all three platforms and runs the platform-agnostic unit tests on each.

Verb / capability	macOS	Windows	Linux
apps (list / resolve)	✅	✅	✅¹
click-at	✅	✅	✅
type	✅	✅	✅
hotkey	✅	✅	✅
screenshot (full screen)	✅	✅²	✅
screenshot (single window)	✅	✅²	✅ X11 · ⏳ Wayland
observe / tree / find / click (by selector)	✅	✅ ⁴	⏳ atspi_pending
find-text / click-text (OCR)	✅	✅³	⏳ ocr_pending
record	✅	⏳ recorder_pending	⏳ recorder_pending
lifecycle (launch / quit / kill / restart)	✅	⏳ unsupported	⏳ unsupported

✅ implemented · ⏳ pending (throws a clear, labeled error)

¹ X11 gives full window counts via wmctrl; on Wayland apps falls back to a /proc walk with no per-window data. ² Windows currently writes the capture as a 32-bpp BMP (a .png path is rewritten to .bmp); PNG encoding via WIC is the follow-up. ³ Windows find-text / click-text are backed by WinRT OCR (Windows.Media.Ocr).

⁴ Windows observe / tree / find / click are backed by UI Automation. Control types are mapped onto the macOS role names selectors normalise to (Edit → AXTextField, Button → AXButton, …), so one selector works on both platforms; the raw UIA type stays available as subrole for the cases where the platforms genuinely differ. click --press invokes through UIA rather than clicking pixels, which works even when the element is scrolled out of view. Apps that draw their own UI (some Electron/WebView2 surfaces) can expose little or nothing to UIA — find-text reads the same screen with OCR and does not depend on the tree.

Why

Agents shouldn't drive desktop apps by guessing pixels. guiport exposes the desktop as structured data: app/window list, focused app/window, accessibility tree, element role/name/value/state/bounds/actions, screenshots only when needed, deterministic replay scripts after exploration.

Install

macOS 13+. See INSTALL.md for full options + platform status.

# Homebrew (recommended) — universal binary, auto-updates with `brew upgrade`
brew install edihasaj/guiport/guiport

# Or install script
curl -fsSL https://raw.githubusercontent.com/edihasaj/guiport/main/scripts/install.sh | sh

# Or from source
swift build -c release && sudo cp .build/release/guiport /usr/local/bin/guiport

After install, grant the two macOS permissions and verify:

guiport doctor --fix   # opens the right Privacy panes
guiport doctor         # all green when ready

Homebrew ships (and runs the CLI from inside) a signed guiport.app, so macOS shows guiport's real logo in Accessibility and Screen Recording and the grant survives upgrades. For source builds, run scripts/install-macos-app.sh before granting permissions so macOS sees one stable app identity instead of a new bare binary per build.

Windows (input/screenshot/apps/UIA tree) — grab the prebuilt guiport-<ver>-windows-x64.zip from releases, or install from source:

iwr -useb https://raw.githubusercontent.com/edihasaj/guiport/main/scripts/install.ps1 | iex

Linux (same shape; AT-SPI2 tree pending) — grab the prebuilt guiport-<ver>-linux-x86_64.tar.gz from releases. Install xdotool+wmctrl+scrot (X11) or ydotool+grim (Wayland), then:

curl -fsSL https://raw.githubusercontent.com/edihasaj/guiport/main/scripts/install.sh | sh

See INSTALL.md for full per-platform notes.

Quick start

guiport doctor                                       # check permissions
guiport apps --json                                  # list running apps with windows
guiport observe --app "Safari"                       # focused window summary
guiport tree --app "Safari" --json                   # full accessibility tree
guiport find --app "Safari" 'button[name="Save"]'    # match selector
guiport click --app "Safari" 'button[name="Save"]'
guiport type "hello"
guiport screenshot --app "Safari" -o safari.png
guiport stream                                      # live NDJSON frame feed; Ctrl-C stops
guiport stream --app "Safari" --fps 2               # stream one window
guiport stream --frames 1 -o /tmp/latest.png        # pull one fresh frame

# Foreground an app without a click, guard keystrokes, and verify state:
guiport activate --app "Safari"                      # raise it (no relaunch, no click)
guiport type --into "Safari" "hello"                 # activate + verify, then type
guiport type --require-frontmost "Safari" "hello"    # refuse if Safari isn't frontmost
guiport assert --app "Safari" --frontmost \
  --front-title-contains "Start Page"                # nonzero exit when unmet

# Vision fallback for canvas / sparse-AX apps:
guiport find-text --app "Figma" "Save"               # OCR via Apple Vision
guiport click-text --app "Figma" "Save"              # OCR + click center
guiport click-at 420 180                             # raw coordinates
guiport record smoke.yaml                            # WIP
guiport run smoke.yaml
guiport serve --mcp                                  # MCP server over stdio

# Reusable, app-specific flows live in user plugins (never in core):
guiport plugin list                                  # discover ~/.guiport/plugins
guiport plugin run focus-and-type type-into text="hi" --app TextEdit

Computer-use stream

guiport stream keeps one screen session alive instead of repeatedly starting and stopping guiport. It writes newline-delimited JSON frame events to stdout and atomically refreshes one PNG path. An agent can let the stream run, read only the frame events it needs, inspect the latest image, act, and continue. The amber control indicator stays active for the same lifetime and follows the real cursor. Press Ctrl-C when finished, or use --seconds / --frames for a bounded session.

guiport stream --fps 2 -o artifacts/live.png
guiport stream --app "Safari" --window "Start Page" -o artifacts/safari-live.png
guiport stream --frames 1 -o /tmp/guiport-frame.png

Plugins

A plugin is a named, reusable flow built from the public primitives — "launch → foreground → navigate → verify → type" captured once instead of hand-rolled by every caller. Core ships zero app-specific knowledge; plugins live in a user directory so personal/private automations never touch the repo.

• Location: ~/.guiport/plugins/*.{yaml,yml} (override with GUIPORT_PLUGINS_DIR or --dir).
• Shape: a plugin targets one app and declares named actions; each action is a list of flow steps (same grammar as guiport run) plus {{param}} placeholders. Steps compose primitives (activate, click, type, hotkey, screenshot) with state predicates (assert: { frontmost, front_title_contains, focused, find }) so a flow verifies it's where it thinks it is before typing.

# ~/.guiport/plugins/focus-and-type.yaml
name: focus-and-type
app: TextEdit
actions:
  - name: type-into
    params: [text]
    steps:
      - activate: true                 # foreground, no relaunch, no stray click
      - wait: 250
      - assert: { frontmost: true }    # refuse to type into the wrong app
      - click: 'AXTextArea'
      - assert: { focused: 'AXTextArea' }
      - type: '{{text}}'

guiport plugin list                                  # human table (--json for machines)
guiport plugin run focus-and-type type-into text="hello world"
guiport plugin run <plugin> <action> key=value ...   # override app with --app

Exposed over MCP too, as the plugin_list and plugin_run tools. A runnable example ships in examples/plugins/ — copy it into ~/.guiport/plugins/ (or pass --dir examples/plugins).

Selector syntax

role[attr=value][attr~=substring][index]

Examples:

button[name="Save"]
textfield[identifier="search"]
AXButton[name~="Open"][index=0]

Supported attributes: role, name (title), value, identifier, description, text (matches name or value), index.

Vision fallback (canvas / Electron apps)

For apps with sparse or absent accessibility (Figma, custom-rendered editors, hardened Electron), guiport falls back through three layers:

1. click-at X Y — raw screen coordinates. The agent reads coords off a screenshot.
2. find-text "Save" / click-text "Save" — Apple Vision (VNRecognizeTextRequest) OCRs the window and returns bounds + center for matched text. On-device, free, no extra deps.
3. LLM vision — out of scope for MVP; agents can call screenshot + their own model to get coords, then click-at.

OCR-found bounds drift across font/scale changes, so prefer AX selectors for replay and OCR for exploration.

Permissions

guiport needs:

• Accessibility — required for AX tree + input events.
• Screen Recording — required for screenshot and screenshot-on-failure artifacts.

Run guiport doctor to check status and get System Settings deep links. Run guiport doctor --fix to trigger the missing permission prompts. For source builds, install the app wrapper with scripts/install-macos-app.sh first.

Driving input from a background agent

macOS posts synthetic events into the caller's security session. A coding agent (or SSH shell, or CI runner) usually runs in a Background launchd session, so it can read AX trees from anywhere but its clicks/keystrokes never reach the on-screen app — they go to the wrong session.

guiport agent solves this with a tiny daemon that runs in the logged-in Aqua (GUI) session via a LaunchAgent. The CLI does all the AX work locally (resolve the element, compute the point — session-agnostic) and forwards only the final low-level event over a Unix socket; the daemon posts it where it lands. A guiport invoked from a real Terminal in the GUI session skips the bridge entirely (zero overhead).

guiport agent install     # run once; writes a LaunchAgent + starts the daemon
guiport agent status      # installed / running / socket

The daemon needs its own Accessibility grant — launchd is its parent, so it can't inherit the grant from a granted terminal the way a foreground CLI does. After agent install, enable guiport under System Settings → Privacy & Security → Accessibility. A Developer-ID-signed build keeps that grant stable across brew upgrade. Then guiport click/type/hotkey from any background process lands on screen.

"Controlling your screen" indicator + Stop

Whenever guiport acts on screen, the Aqua daemon shows a live indicator so you always know an agent is driving — and can take over instantly. In the GuiPort amber theme:

• a pulsing glow around every screen edge,
• a soft halo that tracks the cursor,
• a small chip just below the menu bar: ● Stop guiport esc.

You keep full control of the screen the whole time. To halt guiport, click the Stop chip or press ESC. Either raises a shared Stop signal that every input op checks first, so the next click/type/hotkey/run step aborts with a cancelled error — which the driving agent (Claude, Codex, a YAML replay) sees as a failed tool call and stops on. The Stop is sticky for a short cool-down (GUIPORT_CANCEL_WINDOW_MS, default 10000) so a whole burst of queued actions halts, then auto-heals.

guiport stop            # raise the Stop signal from the CLI (same as the pill / ESC)
guiport resume          # clear it and re-enable
guiport overlay status  # is a daemon running right now? is guiport stopped?
guiport overlay demo    # animate the halo around the screen; Ctrl-C stops it
guiport overlay demo --seconds 10  # timed mode for scripts and smoke tests

No login item. When you run guiport from a Terminal in your GUI session, the overlay host is spawned on demand the first time guiport touches the screen and idle-exits after a few minutes (tune with GUIPORT_DAEMON_IDLE_MS) — it only runs while guiport is actually driving, never as a background startup process. It's macOS-only; the Stop signal itself is cross-platform, so guiport stop halts input on every platform.

(The persistent guiport agent install LaunchAgent is a separate, opt-in thing — only needed to let a background-session agent, e.g. over SSH, deliver clicks at all. GUI-session use needs neither it nor any install.)

Architecture

• Pure Swift, single binary.
• GuiportCore library: AX bridge, selector engine, input, screenshots, replay runner, MCP server.
• guiport CLI: thin wrapper using swift-argument-parser.

Non-goals (MVP)

• No accessibility tree (AT-SPI2) on Linux yet, and no record on Windows or Linux — see the support matrix.
• No vision-first automation.
• No autonomous Manus clone.
• No background/session-0 automation.

Related

• shotport — token-cheap screenshot capture for agents. Calls guiport for macOS app/desktop pixels and accessibility text, then extracts text first and budgets the image to a token limit.
• vmlab — cross-OS verify orchestrator; drives guiport through its guiport transport.

License

MIT — see LICENSE.

Author

Edi Hasaj