Skip to content

perf(ios): match picker images from the page source, and make the masquerading avatar distinguishable - #161

Merged
mpretty-cyro merged 5 commits into
mainfrom
perf/ios-picker-image-matching
Aug 28, 2026
Merged

perf(ios): match picker images from the page source, and make the masquerading avatar distinguishable#161
mpretty-cyro merged 5 commits into
mainfrom
perf/ios-picker-image-matching

Conversation

@mpretty-cyro

Copy link
Copy Markdown
Collaborator

Three related changes to how iOS picks an image out of the photo library. Stacked on #160 — both branches
touch DeviceWrapper.ts, in disjoint regions (dismissCTA there, matchAndTapImage here).

The masquerading avatar was indistinguishable from the file it masquerades as

animated_profile_picture_as_png.png was byte-identical to animated_profile_picture.gif, and both sit
in the simulator's photo library. iOS picks by matching pixels, so which of the two a run uploaded was
decided by tree order. Nothing in the accessibility tree separates them either: same name, and their
picker labels agree to the minute, both being stamped when simctl addmedia ran.

That matters for pro_animated_format_bypass, whose claim is that the client reads the bytes rather than
the extension. Handed the real .gif, the refusal it asserts is free — any client refuses that — so the
spec could pass while testing nothing.

It is now the colour inverse. Inversion is a per-pixel bijection, so the frames still differ exactly as
before and the file still animates; it scores -0.99999994 against the original under
TM_CCOEFF_NORMED, the matcher's default, against a 0.85 threshold.

Worth knowing for anyone checking this: ImageMagick's NCC metric calls the same pair a perfect 1.0.
That metric is not mean-subtracted, so it is invariant to inversion and cannot be used here.

Matching now costs two round-trips instead of N

matchAndTapImage asked XCUITest for the candidates, then screenshotted each one. Both are round-trips
WDA serialises, so cost scaled with the candidate count. Measured on the profile-picture picker:

findElements('xpath', '//XCUIElementTypeImage') 4969ms
one getElementScreenshot 586ms, ×23
getPageSource 1352ms

The tree already carries every rect, and reading all of it costs less than a third of the element query
alone. So candidates and geometry come from one page source, pixels from one screenshot, and cropping and
matching happen locally.

Only 10 of those 23 elements were photographs — the rest belong to the app behind the picker's sheet.
Every thumbnail carries PXGGridLayout-Info, so the profile-picture call site names it. Hence the
candidate argument is a type and an optional name rather than a locator: the set is resolved by parsing,
not by querying.

Two deliberate omissions. It does not consult visible — every element in that picker reports
visible="false", thumbnails included, so filtering on it would discard the targets. And it does not name
the candidates in the share-to-Session flow, which is the Photos app's own grid rather than the in-app
sheet and has not been measured.

pnpm create-simulators was broken

copy_file_to_simulator.ts had MEDIA_ROOT = run/test/media, a directory that no longer exists — fixtures
live in sample_files. The PDF preload threw, so the pool could not be rebuilt at all. create_ios_simulators
checks that PDF exists at its path before calling this, which resolved a different one, so the guard
passed and the copy failed.

Unrelated to the picker in subject, but a changed fixture cannot reach a simulator without it.

Verification

Every device class, both candidate paths:

D=1 D=2 D=3
Image + PXGGridLayout-Info
Image unfiltered
Cell

20 logged invocations across those runs, candidate counts perfectly stable (10 filtered, 11 cell) — so the
filter is not racing the grid load. Android Pass A green, 17/17, including both format-bypass specs, which
reach the inverted fixture by a different route (Android pushes the one file it needs and picks by name,
which is why it was never exposed to the twins).

Two specs each make exactly one uploadProfilePicture call and moved 58s → 41s and 45s → 32s.
Caveat: the simulator pool was recreated between those readings, so some of that could be a fresher pool.

Requires simulators to be recreated — the photo library is loaded once, at creation.

@mpretty-cyro
mpretty-cyro marked this pull request as ready for review August 27, 2026 06:30
@Bilb
Bilb force-pushed the perf/ios-picker-image-matching branch 2 times, most recently from 8a88605 to 8e6b11f Compare August 28, 2026 05:03
Base automatically changed from feat/pro-pins-over-limit-after-lapse to main August 28, 2026 06:41
`MEDIA_ROOT` named `run/test/media`, which does not exist — fixtures live in `sample_files`, as
`mediaFolder` has it. The PDF preload therefore threw and `pnpm create-simulators` could not complete,
so the pool could not be rebuilt at all.

`create_ios_simulators` checks the PDF exists at its own path before calling this, and this resolved a
different one, so the guard passed and the copy failed. Taking the constant from `mediaFolder` leaves
one directory for both to agree on.
It was byte-identical to `animated_profile_picture.gif`, and both sit in the simulator's photo library.
iOS picks the file by matching pixels, so the two were interchangeable: which one a run uploaded was
decided by tree order. A spec that turns on the file being a GIF wearing a `.png` name can therefore have
been handed the real GIF, where a refusal proves nothing about reading the bytes.

Nothing in the tree separates them either — same `name`, and their picker labels agree down to the
minute, both being stamped when `simctl addmedia` ran.

The inverse is a per-pixel bijection, so the frames still differ exactly as before and the file still
animates. It scores -0.99999994 against the original under `TM_CCOEFF_NORMED`, the matcher's default,
against a threshold of 0.85. Note ImageMagick's NCC calls the same pair a perfect 1.0: that metric is not
mean-subtracted, so it is invariant to inversion and cannot be used to check this.

Requires simulators to be recreated: the library is loaded once, at creation.
`matchAndTapImage` asked XCUITest for the candidate elements and then took a screenshot of each. Both are
round-trips WDA serialises, so the cost scaled with the number of candidates. Measured on the
profile-picture picker: 4969ms for `//XCUIElementTypeImage`, then 586ms per element screenshot across 23
elements.

The tree already carries every rect, and reading the whole page source costs 1352ms — less than a third
of the element query alone. So the candidates and their geometry come from one page source, the pixels
from one screenshot, and the cropping and matching happen locally.

Only 10 of those 23 elements were photographs; the rest belong to the app behind the picker's sheet.
Every thumbnail carries `PXGGridLayout-Info`, so the profile-picture call site names it and the app's own
images stop being screenshotted and scored. Hence the candidate argument is a type and an optional name
rather than a locator: the set is resolved by parsing, not by querying.

Two things this is deliberately not doing. It does not consult `visible` — every element in that picker
reports `visible="false"`, thumbnails included, so filtering on it would discard the targets, and
geometry is the reliable test. And it does not name the candidates in the share-to-Session flow, which is
the Photos app's own grid rather than the in-app sheet and has not been measured.

The point/pixel scaling correction goes with it. The reference is resized to the candidate's own size, so
a match covers the whole crop and its centre is the candidate's centre — where the old arithmetic
arrived, with one fewer conversion able to go wrong.
…taken

The figures stay — a perf gotcha is exactly what a comment is for — but two of
them defended a choice against the alternative instead of saying what holds.
@Bilb
Bilb force-pushed the perf/ios-picker-image-matching branch from 0d4e0cf to 0d19e6a Compare August 28, 2026 06:41
The only conflict is `copy_file_to_simulator`, where both sides had fixed the
same broken fixture path. Main's version stands: it drops the `MEDIA_ROOT`
constant and reads `mediaFolder` at the call site, which is where the body
already reads it from — keeping the alias would leave it unused.
@mpretty-cyro
mpretty-cyro merged commit d9cf87f into main Aug 28, 2026
1 check passed
@mpretty-cyro
mpretty-cyro deleted the perf/ios-picker-image-matching branch August 28, 2026 06:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants