Skip to content

feat: add native desktop review harness - #5958

Open
jedwards27 wants to merge 4 commits into
block:mainfrom
jedwards27:feat/native-review-harness
Open

feat: add native desktop review harness#5958
jedwards27 wants to merge 4 commits into
block:mainfrom
jedwards27:feat/native-review-harness

Conversation

@jedwards27

Copy link
Copy Markdown

Summary

  • add a declarative macOS native-review journey runner for the real Tauri/WKWebView app
  • drive pointer and keyboard input through CGEvent while using a debug-only semantic probe for reliable perception
  • record window-only MP4s and write exact-SHA receipts, screenshots, AX diagnostics, and cleanup evidence
  • isolate bundle ID, keyring, HOME, app/WebKit state, fixture identity, and relay access; reject non-loopback relays
  • add a seeded tooltip-dwell journey and deliberate fail-loud mutation fixture

Why this is separate from Playwright

This exercises native launch/packaging, WKWebView behavior, macOS input delivery, window capture, timing-sensitive hover behavior, and cleanup boundaries that browser E2E does not establish.

Validation

At clean HEAD 8b3450af0ef952f5a0b5b8e593f29deb523e5d87:

  • pre-push hooks passed: branch skew, Desktop check/typecheck/test, Rust tests, and Tauri checks
  • native harness Python suite: 9/9
  • doctor: Accessibility and Screen Recording granted; Swift driver builds
  • happy native journey passed three consecutive times with finalized MP4s and cleanup
  • deliberate broken locator failed loudly with screenshot and AX diagnostics
  • journey schemas and git diff --check passed

Representative local receipts:

  • test-results/native-review/8b3450af0ef9/tooltip_fresh_dwell/tooltip_fresh_dwell-20260815T074527-b4904c/receipt.json
  • test-results/native-review/8b3450af0ef9/broken_tooltip/broken_tooltip-20260815T074613-529821/receipt.json

Scope

This is the macOS Desktop MVP. Follow-up stacks will add broader journeys, base/head performance comparison and budgets, iOS Simulator driving/recording, and teammate-grade setup documentation.

Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
wesbillman and others added 3 commits August 15, 2026 15:21
Fail closed on non-standard fixture ports, use fixed loopback database
coordinates, scrub repository-controlled subprocess environments, and
remove generated review keys when fixture seeding fails.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Default harness commands to a credential-free environment and isolate cleanup HOME without reconstructing the host environment.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>

@themiguelamador themiguelamador left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found several blocking correctness/security issues in the native-review harness:

  • The semantic probe accepted unauthenticated cross-origin writes on a wildcard-CORS loopback endpoint and had no request-size cap. A local webpage/process could overwrite evidence or force oversized writes.
  • The Swift recorder advanced its frame clock only when AVAssetWriter was ready. Backpressure left the target instant in the past and turned the capture task into a busy-spin.
  • The launcher depended on PyYAML without declaring or pinning it, so a clean Hermit checkout could not reliably run the advertised command.
  • Journey validation accepted unknown action fields, unsupported keys, booleans as integer durations/timeouts, invalid sustained durations, and duplicate/invalid metric names. The sample measure field was silently ignored and absent from the receipt contract.
  • Relay/bootstrap validation accepted malformed local authorities or invalid pubkeys.
  • All four commits include agent authorship or Co-authored-by: Carl attribution, contrary to this repository’s AGENTS.md attribution rule. This requires replacing the commit history, not an additive fix.

I prepared an attribution-clean, signed replacement at Complear/buzz branch review/pr-5958-fix, commit e278be7ddd9b388d0569c65b26a0fa55916f91bd. It is based on the PR’s true merge base (51beba603) and includes the full PR plus fixes: authenticated/capped probe writes and frontend URL validation, strict journey/relay/pubkey validation, durable measurement receipts and schemas, a backpressure-safe capture clock, and a uv/PyYAML lock.

Verification on that exact tree:

  • native-review Python contracts: 23 passed
  • semantic probe TypeScript test: passed
  • Swift driver release build: passed
  • Desktop pnpm check: passed (only pre-existing warnings/info outside this PR)
  • Desktop pnpm typecheck: passed
  • complete Desktop pnpm test: passed
  • JSON/schema syntax, launcher shell syntax, uv lock check, and git diff --check: passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants