feat(appium): add an Appium test environment and run conformance e2e on PRs - #207
Closed
brax10ward wants to merge 17 commits into
Closed
brax10ward wants to merge 17 commits into
brax10ward wants to merge 17 commits into
Conversation
Adds a standalone test app so the shared conformance suite in mobile-conformance-tests can run against the React Native wrapper, alongside the native Android and iOS test apps it already drives. The suite drives a test app through a fixed contract that React Native cannot reach on its own, and expo prebuild regenerates android/ and ios/ on every run, so an activity or view controller implementing it cannot be checked in. The config plugin injects the native pieces at prebuild time instead: - Android: a module exposing the launch intent's TRANSACT_* extras to JS, a manifest-declared .TransactCommandReceiver (the suite addresses it by explicit component, which a runtime-registered receiver cannot satisfy), singleTask launch mode, and the package registration in MainApplication. - iOS: an Objective-C module reading the TRANSACT_* process environment variables the suite launches with — Metro inlines process.env at build time, so JS cannot read them. Added to the target's Sources phase, since a source file Xcode does not know about is never compiled and RCT_EXPORT_MODULE would never run. Everything native derives from expo.android.package, so changing the id in app.json is sufficient. The app claims its own identifiers rather than the native test app's, so both install side by side and the suite selects one with ANDROID_APP_PACKAGE / IOS_BUNDLE_ID. Log lines use the same tag and wording as the native Android test app so the suite's waitForLogs matches unchanged. On iOS the specs assert on native UI, so the app renders the PauseStatus element and the Task Completed, handoff and RESPOND! alerts. Requires a standalone build: a dev-client build sits at the dev launcher with no JS loaded once Appium reinstalls the app, because installing resets the storage holding the Metro bundle URL. The eas.json "appium" profile builds one. Known React Native bridge gaps are documented in the README, including the pause/resume and fragment-flow APIs the bridge does not expose. The harness logs each unsupported flow so a failing spec says why rather than timing out.
Without an owner, `eas init` creates the project under whichever personal account is logged in rather than the organization. Matches example/app.json.
The pattern read AGENTS.MD while the file is AGENTS.md. macOS is case-insensitive and core.ignorecase is true, so git matched it locally, but case-sensitive tooling did not: EAS Build saw the file as untracked and failed the upload with "Detected inconsistent filename casing between your local filesystem and git".
Silences the "not set, but it will be required in the future" warning. "local" keeps the version in app config rather than having EAS manage and auto-increment it remotely, which suits a test app that is never submitted to a store.
The "appium" profile set a channel, which requires EAS Update: running a build installed expo-updates and wrote updates.url and runtimeVersion into app.json. None of that is wanted here — Appium reinstalls the app on every run, so there is nothing for OTA updates to do, and expo-updates is what fails the release build in the example app (expo-updates:kspReleaseKotlin). Remove the channel, the update config and the dependency; keep the EAS projectId. Verified the release build and the conformance suite still pass without it.
NSLog only reaches a console attached to the process, so an Appium-driven run —
where Appium owns the launch — showed no harness output at all, which made the
authStatusUpdate failure impossible to diagnose. os_log is readable after the
fact with
xcrun simctl spawn <udid> log show --last 5m \
--predicate 'subsystem == "com.atomicfi.appium"'
NSLog is kept so `simctl launch --console-pty` still shows the same lines.
authStatusUpdate was failing on iOS because two alerts were presented within a second of each other: the custom flow's dismiss confirmation, then "Task Completed". iOS shows only the topmost alert and the spec matches on it by title, so the dismiss alert was on screen but underneath — invisible to XCUITest. Serialize alerts through a queue, presenting the next only once the current one is acknowledged, which is what the native iOS test app does with its nextAlertPresentation. Also present the dismiss alert promptly after hiding rather than waiting on onClose/onCleanup. hideTransact is a hide, not a close: it emits neither callback, and the task keeps running — Transact re-presents itself for the summary shortly after — so waiting on them meant the alert only ever came from a timeout, by which point it could be covered. Verified 3/3 on iOS in ~23s, matching the native test app. The README gains a "Gotchas" section covering those two, plus two traps found along the way: the atomictest:// scheme collides with the native iOS test app, and RN's Linking never receives those URLs because the Expo AppDelegate short-circuits openURL before RCTLinkingManager.
build:local:android / build:local:ios run prebuild then Gradle or xcodebuild directly and print the artifact path. They encode two things that are easy to get wrong by hand: JDK 17 (newer JDKs fail inside the Kotlin Gradle plugin) and a UTF-8 locale for anything invoking CocoaPods (pod install otherwise dies in Ruby's unicode normalization before reading the Podfile). Also add build:eas:*:local alongside the cloud builds. EAS is the better choice when the artifact needs to match CI, since it is the same pipeline — but it archives the project through git, so uncommitted changes are excluded and it is a clean build every time. The direct scripts stay for iterating. Both direct scripts verified end to end, and the resulting artifacts pass specs on iOS and Android. The eas --local Android build also succeeds; its artifact lands as a timestamped file in the app directory, now gitignored.
…work The bridge's Config types handoff as a string and passes the config through verbatim, so send the comma-separated value as-is rather than the array the native SDKs take. This does not make the handoff spec pass: a handoff value has no effect through the bridge on either platform — Transact runs past the handoff point and neither onFinish carrying a handoff nor onCleanup fires, with either shape. Recorded in the README's known gaps for the SDK side to pick up.
Replaces build.yml. That workflow built the example app with EAS on every PR and uploaded the artifact, but nothing ever ran against it — the build either worked or it didn't. Now that the repo has an Appium test environment, a PR can run the shared conformance suite instead. appium-e2e.yml keeps build.yml's lint job unchanged — same workflow name and job id, so existing required status checks keep matching — then builds appium-test-environment for both platforms and hands each upload to mobile-conformance-tests' appium_aws.yml, the same path the native SDK repos use. Each test job names the app it is driving, since the suite defaults to the native test apps, and sets TARGET_APP_KIND so the specs covering APIs this wrapper does not expose report as skipped rather than failing (SDK-370, SDK-784, SDK-785, SDK-786). iOS builds with the appium-device profile: Device Farm runs on real devices, so it needs a signed .ipa rather than the simulator build the local scripts produce.
TEMPORARY: both the reusable-workflow refs and the `ref` inputs point at braxton/sdk-774-wrapper-test-targets instead of main. The `ref` inputs matter as much as the `uses` refs — `ref` decides which mobile-conformance-tests checkout supplies the specs, and target-app.ts plus the React Native skips only exist on that branch. Left on main, the run would build the right app and then execute specs that cannot target it. Revert all six to main once the conformance branch merges.
|
Semgrep found 6
Detected possible user input going into a View Dataflow Graphflowchart LR
classDef invis fill:white, stroke: none
classDef default fill:#e7f5ff, color:#1c7fd6, stroke: none
subgraph File0["<b>appium-test-environment/plugin/withAppiumTestEnvironment.js</b>"]
direction LR
%% Source
subgraph Source
direction LR
v0["<a href=https://github.com/atomicfi/atomic-transact-react-native/blob/9c2de6ad0a0dd3bc73d5e6077a9eb5e61c0d45c3/appium-test-environment/plugin/withAppiumTestEnvironment.js#L151 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 151] config</a>"]
end
%% Intermediate
subgraph Traces0[Traces]
direction TB
v2["<a href=https://github.com/atomicfi/atomic-transact-react-native/blob/9c2de6ad0a0dd3bc73d5e6077a9eb5e61c0d45c3/appium-test-environment/plugin/withAppiumTestEnvironment.js#L151 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 151] config</a>"]
v3["<a href=https://github.com/atomicfi/atomic-transact-react-native/blob/9c2de6ad0a0dd3bc73d5e6077a9eb5e61c0d45c3/appium-test-environment/plugin/withAppiumTestEnvironment.js#L159 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 159] destination</a>"]
end
v2 --> v3
%% Sink
subgraph Sink
direction LR
v1["<a href=https://github.com/atomicfi/atomic-transact-react-native/blob/9c2de6ad0a0dd3bc73d5e6077a9eb5e61c0d45c3/appium-test-environment/plugin/withAppiumTestEnvironment.js#L168 target=_blank style='text-decoration:none; color:#1c7fd6'>[Line: 168] destination</a>"]
end
end
%% Class Assignment
Source:::invis
Sink:::invis
Traces0:::invis
File0:::invis
%% Connections
Source --> Traces0
Traces0 --> Sink
⚪️ This finding does not block your pull request. |
The run failed before any job started: the caller declared only `contents: read`, while appium_aws.yml and upload_aws_test_app.yml both request `id-token: write` for AWS OIDC. A called workflow cannot be granted permissions its caller lacks, so the workflow file was rejected outright.
`gh workflow run` surfaced the real error, which the Actions UI only showed as a generic workflow-file problem: error parsing called workflow -> "...appium_aws.yml@braxton/sdk-774-wrapper-test-targets": workflow was not found. The file is present at that ref — GitHub's reusable-workflow resolver does not handle a branch name containing slashes in `owner/repo/path@ref`. Pin the three `uses` refs to the branch head SHA instead, which is unambiguous. The `ref` inputs stay on the branch name: those are passed to actions/checkout, which handles slashes fine. Still temporary — repoint all of it to main once the conformance branch merges.
The Device Farm approach cannot work from here: this repo is public and mobile-conformance-tests is private, and a public repo cannot consume a private repo's reusable workflows. GitHub reports that as "workflow was not found", which is why the first runs failed before any job started. Run the suite on the runners instead. The private suite is checked out with a read-only deploy key — that direction is allowed — and each job builds the test app locally, boots an emulator or simulator, and runs against it. Two things this buys beyond unblocking CI: - No Apple credentials. A simulator build needs no signing, so the iOS job needs no EXPO_APPLE_* secrets on a public repo. - No Expo token. The local build scripts drive prebuild plus Gradle/xcodebuild directly, so EAS is not in the loop. Also make build:local:android honour an existing JAVA_HOME: it looked the JDK up with /usr/libexec/java_home, which only exists on macOS and would have failed on the Linux runner. Drops build-upload-appium-test-app.yml, which only existed to push builds to Device Farm.
The driver install failed on both jobs: 'uiautomator2' cannot be installed because the server version it requires (^3.0.0-rc.2) does not meet the currently installed one (2.19.0) `appium driver install <name>` resolves to the newest driver, which now wants an Appium 3 server. Pin the server and both drivers to the versions this suite is verified against locally: appium 2.19.0, uiautomator2 4.2.7, xcuitest 9.10.1.
Both e2e jobs reached the run step and failed for environment reasons rather than
test failures.
Android never started an emulator: the runner's sdkmanager is too old for the
current SDK metadata ("SDK XML version 4"), so the emulator download failed with
"Error on ZipFile unknown archive". Install current command-line tools first.
iOS built the app, booted the simulator and started Appium, then timed out
creating a session — it was compiling WebDriverAgent, which takes longer than the
120s timeout, three times over. Build WDA in its own step, and raise
WDIO_CONNECTION_RETRY_TIMEOUT to 5 minutes on both jobs for the cold first
session.
The setup-android action's default package list includes the legacy `tools` package, which Google has removed from the SDK repository, so sdkmanager exited 1 and the emulator never started. Only platform-tools and emulator are needed. The iOS job now also tells the driver its simulator is intentionally headless, so it stops restarting the simulator and timing out on the reboot.
14 tasks
Contributor
Author
|
Closed automatically by GitHub when the head branch was renamed Same branch and same commits, continued in #208. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear Link
SDK-774
Summary
Adds an Appium test environment app so the shared conformance suite in
mobile-conformance-testscan run againstthe React Native wrapper, and wires it into CI on pull requests.
Came out of SDK-774 (Gen Digital reported the direct-deposit keypad missing from the Android
accessibility tree) — confirming whether that was wrapper-specific needed a way to drive this SDK
with the same suite that drives the native ones.
appium-test-environment/is a standalone Expo app in its own workspace. No UI beyond a statusline: the suite launches it with
TRANSACT_*parameters and it presents Transact with that config.An Expo config plugin injects what React Native can't reach, and
android//ios/areregenerated by every prebuild so none of it can be checked in:
.TransactCommandReceiver(the suite addresses it by explicit component, which aruntime-registered receiver cannot satisfy),
singleTasklaunch mode, and the packageregistration in
MainApplication.TRANSACT_*process environment variables the suitelaunches with — Metro inlines
process.envat build time, so JS cannot read them. It is added tothe target's Sources phase, since a file Xcode doesn't know about is never compiled and
RCT_EXPORT_MODULEwould never run.Everything native derives from
expo.android.package, so changing the id inapp.jsonis enough.The app claims its own identifiers rather than the native test app's, so both install side by side
and the suite picks one with
ANDROID_APP_PACKAGE/IOS_BUNDLE_ID.CI —
appium-e2e.ymlreplacesbuild.yml. That workflow built the example app on every PR anduploaded the artifact, but nothing ever ran against it. The replacement keeps its lint job verbatim
(same workflow name and job id, so required status checks keep matching), then builds this app for
both platforms with EAS and runs the conformance suite on Device Farm — the same path the native SDK
repos use.
Conformance results
actionsLaunchdeeplinkdepositAmountModalA11yverifyauthStatusUpdatedeferredPaymenthandoffpauseTransactc1PaylinkFlowEvery failure traces to a React Native bridge gap rather than the harness, and each is tracked with
acceptance criteria that include re-enabling the skipped spec:
hideTransactis iOS-only, so Android cannot dismiss Transact.onDataRequestresponse only round-trips on iOS.handoffvalue has no effect on either platform. The one that isn't a knowniOS-only API: the config is accepted and silently ignored.
c1PaylinkFlowis skipped permanently rather than ticketed:FRAGMENT_FLOWis only ever needed onthe native SDKs.
Before merging
CONFORMANCE_REPO_PULL_KEY,MOLECULAR_BANK_API_KEY,MOLECULAR_BANK_API_SECRET(none were needed bybuild.yml).EXPO_TOKENand theEXPO_APPLE_*set already exist.braxton/sdk-774-wrapper-test-targetsin
mobile-conformance-tests, then revert the six temporary refs in9c2de6afrom thatbranch back to
main.com.atomicfi.AppiumTestEnvironment.rn— theappium-deviceprofileneeds a provisioning profile that may not exist in the Expo project yet. This is the likeliest
first-run failure; running only the Android jobs gives a faster signal.
Type of change
Checklist:
ticketed gaps are in the table above; the remaining failures are skipped and tracked.
appium-test-environment/README.mdcovers the contract, both build paths, CI, known gaps anda "Gotchas" section for the traps found along the way.
simulator. Device Farm runs on real hardware, so the first CI run is also the first
physical-device run.
Verification note
The CI workflows themselves are unverified: workflows only execute on GitHub, so the EAS builds
on runners, the reusable-workflow wiring, and the Device Farm handoff are all unexercised. Both
files pass
actionlint.