Resolve the loaded native crash handler and contain startup failures

[melekr]

Summary

Improve Android native crash-handler startup for Unity applications where Backtrace native library is loaded directly from an APK or an Android App Bundle ABI split.

The SDK now resolves the native library selected by Android's linker, while retaining extracted-library, installed split, and base-APK fallbacks.

This PR also incorporates the runtime attribute-containment changes from #300 and strengthens native initialization, JNI cleanup, ANR handling, shutdown, child-process environment construction, and crash-handler failure handling.

Docs

saucelabs/sauce-docs#3640

Problem

The previous Unity implementation constructed the crash-handler library path from the base APK, nativeLibraryDir, and a device-preferred ABI.

That can produce an incorrect path when:

• Android loads native libraries directly from an APK
• an Android App Bundle stores the library in an ABI configuration split
• a 32-bit Unity process runs on a 64-bit-capable device
• nativeLibraryDir or Unity activity metadata is unavailable
• inherited child-process environment variables already contain SDK-reserved keys.

The native bridge also had several lifecycle and interop risks:

• native void exports were declared as returning bool
• JNI local references were retained longer than necessary
• a native bridge could return false after partially initializing native state
• optional native attribute failures could escape into the Unity application or interrupt later attribute updates
• ANR and shutdown failures could prevent later cleanup stages.

What changed

Native-library resolution

• Resolve the loaded libbacktrace-native.so path through Android's native linker.
• Treat a structurally valid linker-selected path as authoritative.
• Fall back to an existing extracted native library.
• Resolve installed ABI configuration splits from ApplicationInfo metadata.
• Fall back to the historical base-APK path when required.
• Compare split candidates globally and reject ambiguous matches.
• Avoid opening or parsing APK archive contents.
• Keep optional linker, filesystem, and package-metadata discovery nonfatal.

Running-process ABI

• Resolve the ABI of the running Unity process rather than using the device's first preferred ABI.
• Use the same process ABI for native-library selection and the device.abi report attribute.
• Preserve the existing unsupported 32-bit x86 native-capture policy.
• Continue managed reporting when native capture is unavailable.

Native initialization and JNI

• Correct P/Invoke return types, Boolean marshalling, and calling conventions.
• Delete JNI local references deterministically.
• Attempt all JNI cleanup operations even if one deletion fails.
• Roll back partial native state when:
  • the native bridge completes and returns false;
  • managed post-initialization setup fails;
  • JNI cleanup fails after native activation.
• Keep rollback failures and rollback diagnostics nonfatal.
• Preserve the original initialization outcome when rollback also fails.

Dynamic native attributes

• Retain attributes in the managed client before optional native propagation.
• Prevent native attribute failures from escaping into the Unity game.
• Continue processing later SetAttributes(...) entries after one native write fails.
• Keep native capture enabled after an isolated attribute failure.
• Emit sanitized diagnostics without logging attribute keys, values, exception messages, paths, or submission URLs.

ANR and shutdown lifecycle

• Contain JVM attach, detach, attribute, and dump failures in the ANR worker.
• Retry transient attachment failures.
• Restore error.type=Crash after every ANR dump attempt.
• Run native disable and Java watcher cleanup stages independently.
• Clear watcher references during shutdown.
• Keep shutdown failures nonfatal.

Crash-handler child process

• Reject missing or blank native-library paths.
• Contain native-library loading and native-dispatch failures.
• Exit nonzero when the child cannot process a crash.
• Avoid logging handler arguments, submission URLs, application attributes, attachment paths, or resolved native-library paths.

Child-process environment

• Preserve unrelated inherited environment variables.
• Replace SDK-reserved variables rather than emitting duplicate keys:
  • CLASSPATH
  • BACKTRACE_UNITY_CRASH_HANDLER
  • LD_LIBRARY_PATH
  • ANDROID_DATA

CI and packaging

• Build Android players on the supported baseline and newer Unity versions.
• Build an ARM64 IL2CPP App Bundle under Unity 6.
• Verify that the App Bundle contains:
  • libil2cpp.so;
  • libbacktrace-native.so;
  • the Java BacktraceCrashHandler class in a base-module DEX file.
• Validate the packaged macOS framework layout.
• Expand lifecycle, resolver, ABI, JNI, attribute, ANR, shutdown, and child-process tests.

Compatibility

• No public API removals.
• No required consumer code changes.
• No serialized configuration migration.
• Traditional APK installations remain supported.
• Android App Bundle and split-APK installations are supported.
• Managed Unity reporting remains available when optional native setup fails.
• Dynamic attributes remain available to managed reports when optional native propagation fails.
• The minimum supported Android API and existing 32-bit x86 policy are unchanged.

C.I

• Unity 2021.3 EditMode
• Unity 2021.3 PlayMode
• Unity 2022.3 EditMode
• Unity 2022.3 PlayMode
• Unity 6000.3 EditMode
• Unity 6000.3 PlayMode
• Unity 2019.4 Android player build
• Unity 2022.3 Android player build
• Unity 6000.3 ARM64 IL2CPP App Bundle build