Raise the Android pool to 8 and add a tiered parallel runner - #166
Merged
Conversation
Two changes that only make sense together: eight emulators are no use without a way to spend them, and the runner has nothing to fill above four. MAX_EMULATORS goes 4 -> 8. `capabilities_android` now derives its udid list from BASE_PORT and that count instead of spelling four out, so growing the pool is one edit rather than two files with nothing to catch a half-done change. The old ceiling was set at 6 off a ~7.5 GB per-emulator estimate. Measured on the 64 GB Linux runner, four emulators are 5.4-7.4 GB PSS each and 24 GB together, so eight lands near 50 GB with ~11 GB spare. Under load it will be more, which is why the tier table says to check `free -g` first. `run_android_parallel.ts` mirrors the iOS runner — one pass per device class, because DEVICES_PER_TEST_COUNT is a single global per invocation — with one difference that matters: it provisions nothing. Appium will not boot an emulator, so it checks the pool it needs is attached and refuses up front. Nothing else would: `global-setup` does that arithmetic for iOS only, so an over-subscribed Android run currently reaches the tests and fails each one with `Invalid actual capability given`, which reads as a suite bug rather than a missing emulator. The tier worker counts are the arithmetic that fills the pool, NOT measurements. Nothing above one worker has been timed on Android and the iOS numbers do not transfer — a simulator is a process tree, an emulator is a QEMU VM. `simulatorsRequired` becomes `devicesRequired` now it serves both. `print_tier.ts` output is unchanged, so the CI workflow that parses it is unaffected.
The two runners were the same script twice from argument parsing to the pass loop, differing only in what they arrange before the run: the iOS one creates and deletes a simulator pool, the Android one can only check that an already-booted one is there. `parallel_shared.ts` takes the half that was duplicated — flag parsing, validation, the tier listing, spawning Playwright and forwarding signals to it, and the one-invocation-per-device-class loop — and each runner keeps only its own devices. Platform flags are declared rather than parsed by hand, so `--keep` and `--runtime` stay iOS-only without either runner owning a copy of the loop that reads them. The consolidation also removed a check that had become unreachable: the Android pool guard was re-testing MAX_EMULATORS after the shared validator had already refused anything over it. Verified against the pre-refactor scripts on nine CLI paths — every tier and validation error, both `--flag value` and `--flag=value`, the `--` passthrough, and `--tier ci --grep … --keep --runtime …` together. Output is byte-identical on all of them but one: iOS `--list-tiers` now says "(4 simulators)" where it said "(4 sims)", matching the "simulators needed:" line right above it and the Android runner's wording. That is the only intended behaviour change.
`Delete account from swarm` clicks Clear and then waits on the spinner, but the work it covers restarts the app: on devnet the delete can finish before we look, and when we do catch the spinner it leaves with the window, which Playwright reports as a target-closed error rather than a hidden element. Neither is a failure of the thing under test. The appearance wait was already tolerant, so this adds the other end: `windowMayClose` treats a closed window as the loader being gone. Opt-in, since everywhere else a window disappearing mid-wait is a crash worth failing on.
…er exists `copy_file_to_simulator` kept its own `MEDIA_ROOT` of `run/test/media`, which was renamed to `sample_files`. Nothing pointed at the old path any more, and nothing was left at it either — the folder is gone. It is the only consumer of that constant and it is called from `create_ios_simulators`, for the PDF specifically, because `simctl addmedia` refuses PDFs. So provisioning a fresh simulator pool died on the template simulator with `Source file does not exist: run/test/media/test_file.pdf`, taking `pnpm create-simulators` and `pnpm test-ios-parallel` with it. Nobody had noticed because a warm pool never reaches this path. Now reads `mediaFolder` from `run/constants/testfiles`, which is what `create_ios_simulators` already uses for the images and videos two functions away. Verified against a booted simulator: the copy branch runs and the file that lands is byte-identical to `sample_files/test_file.pdf`.
… iOS The two runners differ in exactly one thing — what they arrange before a run — so that is now the only thing they say. `runParallelSuite` owns parsing, validation, the child environment, the pass loop, cleanup and the exit status; each runner declares its tier table, its device noun and a `prepareDevices` hook. iOS creates the throwaway pool and returns the UDIDs plus a cleanup; Android can only check an already-booted pool is attached, because Appium will not boot an emulator. That folds two near-identical `main`s into one, and with it the iOS runner picks up `--project mobile`, which only the Android one passed. No spec title outside the mobile project carries `@ios`/`@android` today, so it selects the same 185 tests either way — it is what stops a desktop or cross-platform title that grows one from silently widening a mobile run. Also refuse a flag whose value is missing instead of reading it as empty: `--network` and `--grep` would otherwise fall back to their defaults unnoticed, and a silently-defaulted `--network` provisions the pool and runs the whole suite against the wrong network.
mpretty-cyro
approved these changes
Aug 28, 2026
mpretty-cyro
added a commit
that referenced
this pull request
Aug 28, 2026
The only conflict is `copy_file_to_simulator`, where both sides had fixed the same broken fixture path. Main's version stands: it drops the `MEDIA_ROOT` constant and reads `mediaFolder` at the call site, which is where the body already reads it from — keeping the alias would leave it unused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes that only make sense together: eight emulators are no use without a way to spend them, and the runner has nothing to fill above four.
The pool
MAX_EMULATORS4 → 8, andcapabilities_androidnow derives its udid list fromBASE_PORTand that count rather than spelling four out. Growing the pool used to be a two-file change with nothing to catch a half-done one — the comment onMAX_EMULATORSliterally said "capabilities_android.tsdeclares exactly this many udids".The previous ceiling was 6, set off a ~7.5 GB per-emulator estimate. Measured on the 64 GB Linux runner with four up:
Under test load it will be more than that, which is why the tier table says to check
free -gfirst.The runner
pnpm test-android-parallel --tier <conservative|standard|full>mirrorsrun_ios_parallel.ts— one Playwright invocation per device class, becauseDEVICES_PER_TEST_COUNTis a single global per invocation, so one run has to size every worker's pool for the largest spec. AtD=4a@1-devicesspec occupies a worker holding four emulators and using one.For the Android suite that is 81 / 75 / 45 / 8 specs across the four classes.
One difference from the iOS runner, and it matters: this provisions nothing. Appium will not boot an emulator, so the pool has to be up first. The runner checks the udids it needs are attached and refuses up front:
Worth doing because nothing else would.
global-setupdoes theworkers × devicesarithmetic for iOS only, so an over-subscribed Android run today reaches the tests and fails each one withInvalid actual capability given: N— which reads as a suite bug rather than a missing emulator.The tier numbers are not measurements
Stated plainly in the tier table and in the runner's docblock: they are the arithmetic that fills the pool (
devices * workers <= emulators), and nothing above one worker has been timed on Android. The iOS measurements do not transfer — a simulator is a process tree, an emulator is a QEMU VM costing gigabytes — so a first run on any tier is a measurement, and the workers want lowering before a flake gets blamed on the app.Incidental
simulatorsRequired→devicesRequired, now that it serves both platforms.print_tier.tsoutput is byte-identical (verified:ci --sims→12,ci→ the same four lines), so the CI workflow that parses it is unaffected.