UI testing strategy: adopt Playwright + axe-core for spec-driven browser tests in CI

[smunini]

Summary

crates/ui (helios-ui) has no browser-level test coverage. We should add one, wire it into CI, and structure it the way this repo already structures every other conformance concern: an external normative spec, an executable conformance harness, and a CI gate.

This issue captures a tool evaluation (Playwright vs. a Rust-native CDP driver — both actually built and run against a live hfs), a recommendation, and a phased plan.

A placeholder workflow, .github/workflows/ui-tests.yml, is being added to main so there is a CI seat to grow into. It is a no-op-friendly skeleton: it runs cargo test -p helios-ui, then boots the UI and runs the browser suite only once crates/ui/e2e/package.json exists. Until then the browser steps skip and the job stays green, so it gates nobody.

---

The gap

Today crates/ui is covered by tests/router_http.rs and tests/i18n_http.rs — tower::oneshot calls that assert over the response string. They're fast and valuable and should stay. But they never construct a DOM and never execute assets/theme.js, so an entire class of behavior is structurally unreachable:

Behavior	Where it lives	Reachable by oneshot?
data-theme applied before first paint (the FOUC guard)	assets/theme.js, loaded without defer	❌
localStorage cache + OS prefers-color-scheme precedence	assets/theme.js	❌
Toggle issues PATCH /_user/settings with an RFC 7386 merge-patch	assets/theme.js	❌
Graceful degradation when /_user/settings is unavailable	assets/theme.js	❌
Progressive enhancement — the README's core promise that the UI works with JS off	layouts/base.html, handlers	❌
htmx fragment swap actually replacing the target	hx-* attributes	❌
WCAG 2.2 AA conformance	rendered DOM + CSS	❌

That last row is not hypothetical.

This evaluation already found two real accessibility defects on /ui

Both reproduce on the current main, in both light and dark themes, and are rated serious by axe-core:

1. aria-prohibited-attr — crates/ui/templates/pages/index.html:42

<span class="pill pill--square" aria-label="{{ i18n.t("chart-expand") }}">

aria-label is prohibited on a generic <span> (no role). Assistive tech discards the name, so the "Expand chart" affordance is unlabelled. It also isn't focusable or keyboard-operable. Should be a <button> (or carry an explicit role).

2. target-size — theme toggle buttons, crates/ui/assets/app.css

button[data-set-theme="light"], button[data-set-theme="dark"]

Render at 20×20 px; WCAG 2.2 AA (SC 2.5.8, Target Size (Minimum)) requires ≥24×24 px, and there is insufficient spacing to neighbors to earn the exception.

Neither is caught by any existing test, and neither would be caught by any test we could write with tower::oneshot. That is the argument for this issue in one line.

---

Tool evaluation

I built and ran both candidates against a live hfs --features ui on 127.0.0.1:8080, implementing the same behaviors in each.

A. Playwright + @axe-core/playwright (recommended)

B. chromiumoxide (pure-Rust Chrome DevTools Protocol driver)

Also considered and dropped early: Cypress (weaker multi-browser story, no true JS-disabled context), fantoccini/thirtyfour (WebDriver needs a separate geckodriver/chromedriver daemon — the snap geckodriver on this box already emits mount-namespace errors, which is exactly the CI fragility we don't want).

Results — both work. It's ergonomics and breadth, not capability.

	Playwright	chromiumoxide
Found both a11y defects	✅	✅ (identical output)
JS-disabled rendering	✅ javaScriptEnabled: false (config)	✅ Emulation.setScriptExecutionDisabled
Assert PATCH body	✅ 13 lines	⚠️ 57 lines — manual Network.requestWillBeSent listener, base64-decode postDataEntries, hand-rolled polling loop
axe-core integration	✅ new AxeBuilder({page}).withTags(WCAG).analyze()	⚠️ hand-vendor axe.min.js, inject, JSON.stringify round-trip
LOC for equivalent coverage	126 LOC → 11 tests	243 LOC → 5 tests
Auto-waiting / retry	✅ built in	❌ manual sleep loops
Cross-browser	✅ Chromium, Firefox, WebKit	❌ Chrome only
Trace viewer, HTML report, sharding	✅	❌
Dependency friction hit during eval	none	chromiumoxide@0.9.1 has a broken feature combo — rustls pulls chromiumoxide_fetcher, which won't compile without zip8; tokio-runtime was silently removed in 0.9

Recommendation: Playwright, with @axe-core/playwright.

The honest counter-argument is that Playwright introduces a Node toolchain into a repo that has zero Node today (no package.json, no setup-node in any workflow). Three things defuse it:

1. Precedent. We already run non-Rust conformance harnesses in CI — Inferno is Ruby-in-Docker (inferno-us-core.yml, inferno-bulk-data.yml), and hts-ig-conformance.yml shells out to the Java IG Publisher. A Node harness for the browser layer is the same trade we've already made twice.
2. The Rust path doesn't actually avoid npm. axe-core ships on npm. The chromiumoxide experiment still had to pull axe.min.js out of node_modules. "Pure Rust" buys us a worse harness and still needs the npm artifact.
3. Containment. Everything lives under crates/ui/e2e/ — package.json, package-lock.json, npm ci, pinned browser. Nothing leaks into the cargo workspace; cargo build/cargo test are untouched.

The chromiumoxide prototype is a genuinely viable fallback if we later decide the Node dependency is unacceptable. It is not a strawman — it passed every test I ported to it.

---

Spec-driven design

Mirrors how HFS already treats FHIR: normative spec → conformance harness → CI gate. Three tiers, in dependency order.

Tier 1 — Normative external specs (nothing to author locally)

• WCAG 2.2 AA is the spec; axe-core is the harness. Tags: wcag2a, wcag2aa, wcag21a, wcag21aa, wcag22aa. Run across the light × dark theme matrix — contrast and target-size violations differ per theme.
• Zero-maintenance: the spec is versioned and owned upstream, exactly like the US Core IG.

Tier 2 — Local behavioral specs as first-class artifacts

The UI's own behavior needs a spec that is readable, reviewable, and executable. Proposal: Gherkin .feature files under crates/ui/e2e/features/, bound to steps via playwright-bdd, so the .feature file — not the test code — is the source of truth, and Playwright stays the runner.

Feature: Theme selection
  Scenario: A returning dark-mode user sees no flash of light
    Given a stored theme preference of "dark"
    When I open "/ui"
    Then "data-theme" is "dark" before first paint

  Scenario: Toggling roams the choice across devices
    When I open "/ui"
    And I click the "dark" theme toggle
    Then a PATCH to "/_user/settings" carries {"theme": "dark"}
    And the local cache holds "dark"

Spec-first is the workflow: write the scenario, watch it fail, then build the UI. New UI work lands a .feature in the same PR.

Tier 3 — Design and invariant specs

• Visual regression. README.md already names the Figma file (CcLtq79cH2aHv4Ii9aNQTP, frames "Dashboard V1.1" / "… - Dark" / "… - Tenant Selector") as the normative design source. Playwright's toHaveScreenshot() pins rendered output to committed baselines derived from those frames, in both themes.
• Executable "rules of the road." crates/ui/README.md states rules that are currently enforced only by review. Several are mechanically checkable and belong in this suite (or a lint step next to it):
  • every interactive control works with JS disabled (enumerate hx-* / data-set-theme controls; assert each has a real <a href>/<form> fallback exercised by the nojs project);
  • no inline <script> blobs in rendered HTML;
  • no external-origin requests at runtime (no CDN) — assert every request the page makes is same-origin;
  • no uncaught pageerror on any route.

The test pyramid stays right-side-up: the Rust oneshot tests remain the fast inner ring for routing/i18n/headers; the browser suite is the thin outer ring for behavior only a real engine can observe.

---

Phased plan

• Phase 0 — CI seat (this issue's PR). Land .github/workflows/ui-tests.yml as a green, skipping placeholder. Runs cargo test -p helios-ui today; auto-activates when the suite appears.
• Phase 1 — Harness + first specs. Create crates/ui/e2e/ (Playwright + @axe-core/playwright, npm ci, pinned Chromium). Port the ~11 tests from this evaluation (theme, progressive enhancement, a11y). Fix the two defects above. Flip ui-tests.yml to a required check on crates/ui/**.
• Phase 2 — Spec-driven layer. Introduce playwright-bdd + features/*.feature. Establish "new UI work ships a .feature" as the rule.
• Phase 3 — Visual + theme matrix. toHaveScreenshot() baselines from the Figma frames; light × dark. Decide baseline storage (repo vs. artifact) and a masking policy for dynamic values.
• Phase 4 — Breadth. Cross-browser (WebKit/Firefox), sharding, real read paths (metric cards, tenant selector, CodeSystem/ValueSet lookup, htmx swaps) as they land.

Open questions for @angela-helios

1. Baselines in-repo? Screenshot baselines are binary and churn. Commit them, or publish as artifacts and compare against last-known-good?
2. Runner browsers. Self-hosted runners: cache ~/.cache/ms-playwright, or run the suite in the official mcr.microsoft.com/playwright container?
3. Gate scope. Should ui-tests.yml also run on push to main, or PR-only (as drafted)?
4. playwright-bdd vs. plain *.spec.ts. BDD buys a reviewable spec artifact at the cost of a layer of indirection. Worth it here?
5. Blocking vs. advisory a11y. Start target-size/aria-prohibited-attr as hard failures, or land the suite advisory-only for one sprint?

---

Reproducing the evaluation

cargo run -p helios-hfs --features ui        # serves http://127.0.0.1:8080/ui
# then, in the e2e harness:
npx playwright test                          # 11 tests; the 2 a11y specs fail on current main

/cc @smunini