Skip to content

feat: complete, harden, and package native review workflows - #5972

Open
jedwards27 wants to merge 19 commits into
block:mainfrom
jedwards27:feat/native-review-docs
Open

feat: complete, harden, and package native review workflows#5972
jedwards27 wants to merge 19 commits into
block:mainfrom
jedwards27:feat/native-review-docs

Conversation

@jedwards27

@jedwards27 jedwards27 commented Aug 15, 2026

Copy link
Copy Markdown

Summary

  • finalize every recorded Desktop/iOS native-review run as a Buzz-compatible H.264 MP4
  • add a finding-bundle command for optional clipping, focused/redacted logs, minimal receipt provenance, and hashed manifest
  • fail the receipt if shareable evidence finalization fails instead of silently omitting the artifact

Stacked on #5972 because the native-review harness has not landed on main yet.

Validation

At exact clean head 2b74739e0f7869c559612dcedb4b61b7094c7aab:

  • python3 -m unittest discover -s tools/native-review/tests -p 'test_*.py' — 29/29 pass
  • pre-push hooks — branch skew, Desktop check/typecheck/4,954 tests, mobile tests, Rust tests, and Tauri checks all pass
  • just native-review-ios 'iPhone 17 Pro' — passes on iOS 26.5; receipt clean and cleanup passes
  • generated video-share.mp4 is H.264/yuv420p, 994×2160, and accepted by the live Buzz upload endpoint
  • the finding-bundle command converted the PR Polish mobile message actions #5873 failing simulator receipt into a 1.2 MB shareable MP4 plus focused disposed-image log excerpt, minimal receipt, and SHA-256 manifest; live upload accepted it

Receipt: test-results/native-review/2b74739e0f78/ios_pairing/ios-20260815T120507-595304/receipt.json

Carl added 7 commits August 14, 2026 20:54
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
…urneys

* origin/main:
  fix(desktop): hide the offcanvas-collapsed sidebar so it stops painting over the community rail (block#5947)
  Polish mobile message threads and composer (block#5645)
  chore(release): release Buzz Desktop version 0.5.14 (block#5917)
  ci(release): remove desktop smoke gate (block#5914)

Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
@jedwards27
jedwards27 requested a review from a team as a code owner August 15, 2026 17:15
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Jude Edwards <judeedwards@squareup.com>
@jedwards27 jedwards27 changed the title docs: explain native review workflows docs: complete and harden native review workflows Aug 15, 2026
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
@jedwards27 jedwards27 changed the title docs: complete and harden native review workflows feat: complete, harden, and package native review workflows Aug 15, 2026
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
@jedwards27
jedwards27 force-pushed the feat/native-review-docs branch from d856366 to 682a6f0 Compare August 15, 2026 22:00
wesbillman and others added 6 commits August 15, 2026 15:27
Fail closed on non-standard fixture ports, use fixed loopback database
coordinates, scrub repository-controlled subprocess environments, and
remove generated review keys when fixture seeding fails.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Apply absolute maximum budgets to the worst candidate sample while retaining
median-based relative regression comparisons. Cover the catastrophic-outlier
case directly.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Create a unique run-owned simulator, scrub credentials from iOS review subprocesses, and delete only the owned device during cleanup.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Default harness commands to a credential-free environment and isolate cleanup HOME without reconstructing the host environment.

Co-authored-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>

Signed-off-by: Wes <wesbillman@users.noreply.github.com>
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>
(cherry picked from commit 3635687)
Signed-off-by: Carl <5f365698229751c0461f57bb03a4e93134e6e936bd7039ebe7b737282a43c754@buzz.block.builderlab.xyz>

@themiguelamador themiguelamador left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking findings:

  1. The documented native-review command/test path imports PyYAML, but the dependency is neither installed nor pinned; it fails from a clean checkout even after Hermit activation.
  2. The evidence redactor leaks common secret forms (Authorization: Bearer …, JSON keys such as "token": "…") and copies receipt failure/cleanup strings without recursive redaction, so a bundle described as privacy-safe can disclose credentials.
  3. The AVAssetWriter capture loop does not advance its frame clock while the writer is backpressured. Its sleep target then remains in the past and the driver busy-spins, perturbing the performance run it is measuring.
  4. scroll accepts a target locator but posts at the mouse's previous position, so journeys can scroll the wrong view.
  5. iOS receipts omit the common schema fields (flow, timestamps, isolation, steps, measurements, performance) and add an undeclared device, making them invalid against the checked-in receipt schema and producing incomplete publication provenance.
  6. Journey/performance/media inputs accept malformed durations, duplicate samples, booleans, and non-finite numbers; some fail late, while duplicate receipts can satisfy the minimum cohort size without independent runs.

Fixed in Complear/buzz@b3562d00e (branch review/pr-5972-fix): pinned uv/PyYAML environment, comprehensive evidence redaction, backpressure-safe capture scheduling, locator-targeted scrolls, schema-compatible iOS receipts, and fail-closed input validation with regression coverage. Verification: native-review 50/50, Swift release build, desktop 4,954/4,954 plus lint/typecheck, mobile 1,417/1,417 plus analysis, shell/JSON/Python syntax checks.

Princess Donut added 3 commits August 18, 2026 15:46
* origin/main: (43 commits)
  perf(desktop): parallelize relay agent directory rebuild (block#6258)
  Refine the mobile emoji picker (block#5853)
  fix(desktop): exclude archived agents from nest, order regeneration (block#5905)
  Add font size and conversation density preferences (block#5644)
  fix(desktop): emit camelCase config-write payload fields (block#6062)
  fix(desktop): downscale large avatars for agent-share PNG body (block#6260)
  fix(desktop): preserve early relay auth challenges (block#3320)
  Polish mobile message actions (block#5873)
  Refine mobile pairing confirmation (block#6018)
  chore(scripts): add buzz-adopt-prod-agents.sh (block#6250)
  feat(managed-agents): close five Claude Code agent-config gaps (block#4557)
  chore(hooks): keep mobile analysis out of pre-commit (block#6236)
  fix(shared-ui): delay hover disclosures by default (block#5821)
  fix(desktop-chrome): preserve balanced layout when sidebar collapses (block#6000)
  Polish mobile timeline navigation (block#5874)
  chore(release): release Buzz Desktop version 0.5.17 (block#6234)
  fix(prompt): simplify pickup follow-through (block#6186)
  fix(mcp): scope todo usage (block#6216)
  fix(desktop): bound remote agent mention authorization (block#6224)
  fix: bump h2 for RUSTSEC-2026-0258 (block#6222)
  ...

Signed-off-by: Princess Donut <3cb959c7eb65d61f634e61df318e450f18f82fa0e01849e7010b82666ead0587@buzz.block.builderlab.xyz>

# Conflicts:
#	desktop/src/main.tsx
#	mobile/ios/Podfile.lock
Signed-off-by: Princess Donut <3cb959c7eb65d61f634e61df318e450f18f82fa0e01849e7010b82666ead0587@buzz.block.builderlab.xyz>
Signed-off-by: Princess Donut <3cb959c7eb65d61f634e61df318e450f18f82fa0e01849e7010b82666ead0587@buzz.block.builderlab.xyz>
@jedwards27

Copy link
Copy Markdown
Author

Implemented the requested hardening in b854c00ea29afedf7634c86ac5febdbf36ad3a85 (including the base merge). The six blocking points are addressed as follows:

  1. Clean PyYAML provisioning: both launchers now use explicit uv run --with pyyaml==6.0.3; fresh empty-cache desktop and iOS --help runs pass. The inaccessible internal-Artifactory project/lock was removed.
  2. Recursive secret redaction: nested receipt/failure/cleanup content is redacted, including JSON, single-quoted, environment-style, and Authorization: Bearer forms. The original JSON leak probe now emits {"authorization": "[REDACTED]"}.
  3. Capture backpressure: the frame schedule advances while AVAssetWriter is not ready, preventing a stale deadline and busy-spin.
  4. Locator-targeted scroll: scroll resolves the target locator and posts at the target element rather than the previous mouse position.
  5. Schema-compatible iOS receipts: iOS output now conforms to the common receipt schema and carries the required flow/timestamp/isolation/step/measurement/performance provenance without the undeclared device field.
  6. Fail-closed validation: malformed/non-finite/bool numeric values, missing or zero waits, zero sustained durations, repeated run IDs, and cross-cohort reused artifact paths are rejected; regression coverage was added.

Exact-head validation was performed independently twice on b854c00e:

  • native-review Python suite: 50/50 passed
  • all three desktop journey schema validations: passed
  • Swift release build: passed
  • Python compile and launcher shell syntax: passed
  • desktop and iOS launchers with fresh empty uv caches: passed
  • targeted redaction, duplicate-run/path, and duration/wait probes: passed
  • both validating checkouts were clean and matched the PR head OID

GitHub currently reports the PR as mergeable at this SHA. DCO passes; other repository checks/review requirements remain pending. Any head change invalidates the exact-head evidence above.

Please re-review b854c00ea29afedf7634c86ac5febdbf36ad3a85.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants