perf(ios): match picker images from the page source, and make the masquerading avatar distinguishable - #161
Merged
Merged
Conversation
mpretty-cyro
marked this pull request as ready for review
August 27, 2026 06:30
Bilb
approved these changes
Aug 27, 2026
Bilb
force-pushed
the
perf/ios-picker-image-matching
branch
2 times, most recently
from
August 28, 2026 05:03
8a88605 to
8e6b11f
Compare
`MEDIA_ROOT` named `run/test/media`, which does not exist — fixtures live in `sample_files`, as `mediaFolder` has it. The PDF preload therefore threw and `pnpm create-simulators` could not complete, so the pool could not be rebuilt at all. `create_ios_simulators` checks the PDF exists at its own path before calling this, and this resolved a different one, so the guard passed and the copy failed. Taking the constant from `mediaFolder` leaves one directory for both to agree on.
It was byte-identical to `animated_profile_picture.gif`, and both sit in the simulator's photo library. iOS picks the file by matching pixels, so the two were interchangeable: which one a run uploaded was decided by tree order. A spec that turns on the file being a GIF wearing a `.png` name can therefore have been handed the real GIF, where a refusal proves nothing about reading the bytes. Nothing in the tree separates them either — same `name`, and their picker labels agree down to the minute, both being stamped when `simctl addmedia` ran. The inverse is a per-pixel bijection, so the frames still differ exactly as before and the file still animates. It scores -0.99999994 against the original under `TM_CCOEFF_NORMED`, the matcher's default, against a threshold of 0.85. Note ImageMagick's NCC calls the same pair a perfect 1.0: that metric is not mean-subtracted, so it is invariant to inversion and cannot be used to check this. Requires simulators to be recreated: the library is loaded once, at creation.
`matchAndTapImage` asked XCUITest for the candidate elements and then took a screenshot of each. Both are round-trips WDA serialises, so the cost scaled with the number of candidates. Measured on the profile-picture picker: 4969ms for `//XCUIElementTypeImage`, then 586ms per element screenshot across 23 elements. The tree already carries every rect, and reading the whole page source costs 1352ms — less than a third of the element query alone. So the candidates and their geometry come from one page source, the pixels from one screenshot, and the cropping and matching happen locally. Only 10 of those 23 elements were photographs; the rest belong to the app behind the picker's sheet. Every thumbnail carries `PXGGridLayout-Info`, so the profile-picture call site names it and the app's own images stop being screenshotted and scored. Hence the candidate argument is a type and an optional name rather than a locator: the set is resolved by parsing, not by querying. Two things this is deliberately not doing. It does not consult `visible` — every element in that picker reports `visible="false"`, thumbnails included, so filtering on it would discard the targets, and geometry is the reliable test. And it does not name the candidates in the share-to-Session flow, which is the Photos app's own grid rather than the in-app sheet and has not been measured. The point/pixel scaling correction goes with it. The reference is resized to the candidate's own size, so a match covers the whole crop and its centre is the candidate's centre — where the old arithmetic arrived, with one fewer conversion able to go wrong.
…taken The figures stay — a perf gotcha is exactly what a comment is for — but two of them defended a choice against the alternative instead of saying what holds.
Bilb
force-pushed
the
perf/ios-picker-image-matching
branch
from
August 28, 2026 06:41
0d4e0cf to
0d19e6a
Compare
Bilb
approved these changes
Aug 28, 2026
The only conflict is `copy_file_to_simulator`, where both sides had fixed the same broken fixture path. Main's version stands: it drops the `MEDIA_ROOT` constant and reads `mediaFolder` at the call site, which is where the body already reads it from — keeping the alias would leave it unused.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related changes to how iOS picks an image out of the photo library. Stacked on #160 — both branches
touch
DeviceWrapper.ts, in disjoint regions (dismissCTAthere,matchAndTapImagehere).The masquerading avatar was indistinguishable from the file it masquerades as
animated_profile_picture_as_png.pngwas byte-identical toanimated_profile_picture.gif, and both sitin the simulator's photo library. iOS picks by matching pixels, so which of the two a run uploaded was
decided by tree order. Nothing in the accessibility tree separates them either: same
name, and theirpicker labels agree to the minute, both being stamped when
simctl addmediaran.That matters for
pro_animated_format_bypass, whose claim is that the client reads the bytes rather thanthe extension. Handed the real
.gif, the refusal it asserts is free — any client refuses that — so thespec could pass while testing nothing.
It is now the colour inverse. Inversion is a per-pixel bijection, so the frames still differ exactly as
before and the file still animates; it scores -0.99999994 against the original under
TM_CCOEFF_NORMED, the matcher's default, against a 0.85 threshold.Worth knowing for anyone checking this: ImageMagick's
NCCmetric calls the same pair a perfect 1.0.That metric is not mean-subtracted, so it is invariant to inversion and cannot be used here.
Matching now costs two round-trips instead of N
matchAndTapImageasked XCUITest for the candidates, then screenshotted each one. Both are round-tripsWDA serialises, so cost scaled with the candidate count. Measured on the profile-picture picker:
findElements('xpath', '//XCUIElementTypeImage')getElementScreenshotgetPageSourceThe tree already carries every rect, and reading all of it costs less than a third of the element query
alone. So candidates and geometry come from one page source, pixels from one screenshot, and cropping and
matching happen locally.
Only 10 of those 23 elements were photographs — the rest belong to the app behind the picker's sheet.
Every thumbnail carries
PXGGridLayout-Info, so the profile-picture call site names it. Hence thecandidate argument is a type and an optional name rather than a locator: the set is resolved by parsing,
not by querying.
Two deliberate omissions. It does not consult
visible— every element in that picker reportsvisible="false", thumbnails included, so filtering on it would discard the targets. And it does not namethe candidates in the share-to-Session flow, which is the Photos app's own grid rather than the in-app
sheet and has not been measured.
pnpm create-simulatorswas brokencopy_file_to_simulator.tshadMEDIA_ROOT = run/test/media, a directory that no longer exists — fixtureslive in
sample_files. The PDF preload threw, so the pool could not be rebuilt at all.create_ios_simulatorschecks that PDF exists at its path before calling this, which resolved a different one, so the guard
passed and the copy failed.
Unrelated to the picker in subject, but a changed fixture cannot reach a simulator without it.
Verification
Every device class, both candidate paths:
Image+PXGGridLayout-InfoImageunfilteredCell20 logged invocations across those runs, candidate counts perfectly stable (10 filtered, 11 cell) — so the
filter is not racing the grid load. Android Pass A green, 17/17, including both format-bypass specs, which
reach the inverted fixture by a different route (Android pushes the one file it needs and picks by name,
which is why it was never exposed to the twins).
Two specs each make exactly one
uploadProfilePicturecall and moved 58s → 41s and 45s → 32s.Caveat: the simulator pool was recreated between those readings, so some of that could be a fresher pool.
Requires simulators to be recreated — the photo library is loaded once, at creation.