Improve macOS integration test performance #727
Conversation
|
marcins
force-pushed
the
mszczepanski/improve-mac-integration-test-perf
branch
2 times, most recently
from
2f32582 to
edb0e79
Compare
|
Just doing some more benchmarking, looks like |
* Use brute-force watcher on darwin only
…edup Don't use fast-glob, it doesn't work with Atlaspack fs :P Improve some comments
marcins
force-pushed
the
mszczepanski/improve-mac-integration-test-perf
branch
from
c575009 to
5ac273b
Compare
| "posthtml": "^0.16.5", | ||
| "posthtml-parser": "^0.10.1", | ||
| "resolve": "^1.12.0", | ||
| "rimraf": "^5.0.5", |
There was a problem hiding this comment.
I still don't get why I have to add these as half the time running yarn add in a workspace will throw an error about rimraf being required but not being present in the workspace.
| "license": "(MIT OR Apache-2.0)", | ||
| "private": true, | ||
| "main": "src/index.js", | ||
| "main": "./lib/index.js", |
There was a problem hiding this comment.
Fixes running the Mocha profiler (via ATLASPACK_PROFILE_MOCHA=true) with TypeScript.
| { | ||
| inputFS: overlayFS, | ||
| shouldDisableCache: false, | ||
| watchBackend: os.platform() === 'darwin' ? 'fs-events' : undefined, |
There was a problem hiding this comment.
Isn't fs-events the default on mac anyway?
There was a problem hiding this comment.
If watchman is present it uses watchman, which I believe is the case in our CI.
| // For debugging purposes, log all instances of the projectRoot in the cache. | ||
| // Otherwise, fail the test if one is found. | ||
| if (process.env.ATLASPACK_DEBUG_CACHE_FILEPATH != null) | ||
| return assertNoFilePathInCacheDebug(fs, dir, projectRoot); |
There was a problem hiding this comment.
No brace hey? How fancy of you
There was a problem hiding this comment.
It's possibly Cursor's fault and I didn't notice.. if you don't like it, lint for it 🧌
Motivation
macOS integration tests seem to be getting slower again. Profiling revealed two large hot spots both originating from the Cache tests:
Changes
This PR changes the watcher backend to
fs-eventson macOS. This reduces the CI time by approx 4-5 min (from ~30min to ~26min) based on two test runs.I did also try and optimise theassertNoFilePathInCachebut the few things I tried weren't any faster (using a chunked faster string search algo, scanning in a promise queue, using a Rust based function for searching for the string in the file).assertNoFilePathInCachewas optimised to use a promise queue to check files - testing with Hyperfine locally showed the best value for concurrency was 50, this reduces macOS test time by another ~5 min and Linux by ~2 min.Checklist
docs/folder[no-changeset]: private packages only