Skip to content

ci: add browser tests on the GitHub runner - #308

Open
YasharF wants to merge 1 commit into
sinonjs:mainfrom
YasharF:ci/browser-test-alongside-saucelabs
Open

YasharF wants to merge 1 commit into
sinonjs:mainfrom
YasharF:ci/browser-test-alongside-saucelabs

Conversation

@YasharF

@YasharF YasharF commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

The current Sauce Labs setup tests the code in real browsers. However:

  1. The Sauce job only runs on main, because SAUCE_USERNAME and SAUCE_ACCESS_KEY can't safely be exposed to PRs. So PRs get no browser test results, and dependabot-automerge, which needs saucelabs-test, is skipped on every Dependabot PR.
  2. Running the browser tests needs Sauce credentials, so contributors without them can't check their changes in a real browser.

Background

The Sauce Labs setup is there to make sure the code works in real browsers. Anything that replaces it should provide the same level of testing and verification. GitHub Actions runners can run the same real browsers. A benefit of running them on the runners is that there are no separate credentials to keep as secrets or rotate.

Solution

  • Adds a browser-test job that runs the suite in Chrome, Edge and Firefox, on both ubuntu-latest and windows-latest, as six parallel jobs.
  • Each job installs the vendor's current stable browser, in line with the Sauce config's browserVersion: "latest", and fails if the browser under test isn't the version it installed.
  • It uses the same @mochify/driver-webdriver and the same mochify bundle config as the Sauce run. The difference is where the browser runs: a WebDriver server started on the runner instead of ondemand.saucelabs.com, with the tests served by mochify's local server instead of the external page. Its config is a new file, mochify.webdriver-local.cjs, excluded from coverage like mochify.webdriver.cjs.
  • No secrets and no new dependencies, so it runs on every push and PR, including forks and Dependabot.
  • dependabot-automerge now needs browser-test instead of saucelabs-test, so it can run on Dependabot PRs.
  • saucelabs-test, npm run test-cloud and prepublishOnly are unchanged. Both browser jobs can run side by side for bake time, and the Sauce job can be removed in a future PR.

How to verify

See the Actions run at https://github.com/YasharF/samsam/actions/runs/34800321341. Each browser-test job prints the browser name, version and OS in its "Check browser version" step, and reports 602 passing and 3 pending tests.

The 3 pending tests skip themselves when they don't apply, the same as in the Sauce runs:

  • deep-equal.test.js, "returns true for buffers with identical content": browsers have no Buffer.
  • create-matcher.test.js, "when Symbol is not defined › should call assertType and return a matcher": it replaces modules with proxyquire, which only works in Node.
  • match.test.js, "when engine cannot compare Map instances › throws an error": it only applies to old engines that can't compare Maps, so it also skips in Node.

npm test in Node reports 604 passing and 1 pending, the Map test, so both cover the same 605 tests.

@codecov

codecov Bot commented Sep 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (5f88086) to head (d9d6d81).
⚠️ Report is 64 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##              main      #308     +/-   ##
===========================================
  Coverage   100.00%   100.00%             
===========================================
  Files           28        29      +1     
  Lines          532      1619   +1087     
===========================================
+ Hits           532      1619   +1087     
Flag Coverage Δ
unit ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant