gluten-package.jar bundles duplicate DefaultAllocationManagerFactory that breaks Arrow 16.0.0+ CheckAllocator

[wangyum]

Description

The shaded `gluten-package.jar` bundles `org.apache.arrow.memory.DefaultAllocationManagerFactory.class` (and the `unsafe/` / `netty/` subpath variants in newer Arrow versions) because the shade plugin's relocation excludes `org.apache.arrow.memory.**` from relocation but still includes the class files in the bundle:

```xml

org.apache.arrow
${gluten.shade.packageName}.org.apache.arrow

org.apache.arrow.memory.**
...


```

This bundled class is redundant — the same class is always available on the runtime classpath via the upstream `arrow-memory-unsafe` / `arrow-memory-netty` jar (a declared dependency of `gluten-arrow`). Worse, starting with Arrow 16.0.0, `org.apache.arrow.memory.CheckAllocator.check()` inspects the URL of `DefaultAllocationManagerFactory.class` and throws

```
java.lang.IllegalStateException: Unknown allocation manager type to infer.
Current: file:/.../gluten-package_2.12-1.7.0-SNAPSHOT.jar!/org/apache/arrow/memory/DefaultAllocationManagerFactory.class
at org.apache.arrow.memory.CheckAllocator.check(CheckAllocator.java:57)
```

when the URL does not contain `memory-core`, `memory-unsafe`, or `memory-netty` (which it does not when the class is loaded from inside `gluten-package.jar`).

When this manifests

`CheckAllocator`'s URL check was added in Arrow 16.0.0. The bundled class lives at the CORE path `org/apache/arrow/memory/DefaultAllocationManagerFactory.class` in Arrow <= 15.x, and at subpaths (`unsafe/` / `netty/`) in Arrow >= 16.x. So the failure occurs when all of the following are true:

1. Gluten is built with `arrow.version` <= 15.x (default for spark-3.3/3.4/3.5 profiles → bundles the CORE-path class)
2. The runtime Arrow version on the Spark classpath is >= 16.0.0 (which has the new URL-aware `CheckAllocator`)
3. `arrow.allocation.manager.type` is not explicitly set (the default `Unknown` case falls through to `CheckAllocator.check()`)

This combination happens when a Spark distribution ships a newer Arrow than what Gluten was built against. For example, a downstream Spark 3.5 fork that upgrades Arrow from 12.0.1 to 18.3.0 will trigger this on every Gluten query, because Gluten's `spark-3.5` profile still uses `arrow.version=15.0.0`.

Why upstream CI does not currently catch this

• Spark 3.3/3.4/3.5 community → Arrow 12.0.1 → `CheckAllocator` predates the URL check → works
• Spark 4.0/4.1 → `spark-4.0` profile overrides `arrow.version` to 18.1.0 → gluten-package bundles the `unsafe/`-subpath class → URL contains `/org/apache/arrow/memory/unsafe/` → matches → works

So the issue is silent on upstream today but breaks for any custom Spark distribution that bumps Arrow past 16.0.0 without also bumping Gluten's `arrow.version`.

Reproducer

```
java.lang.ExceptionInInitializerError
at org.apache.gluten.memory.arrow.alloc.ArrowBufferAllocators.(ArrowBufferAllocators.java:44)
at org.apache.spark.shuffle.CelebornColumnarBatchSerializerInstance.(...)

Caused by: java.lang.IllegalStateException: Unknown allocation manager type to infer.
Current: file:/.../gluten-package_2.12-1.7.0-SNAPSHOT.jar!/org/apache/arrow/memory/DefaultAllocationManagerFactory.class
at org.apache.arrow.memory.CheckAllocator.check(CheckAllocator.java:57)
at org.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:80)
at org.apache.arrow.memory.AllocationManager.(AllocationManager.java:39)
at org.apache.arrow.memory.BaseAllocator.(BaseAllocator.java:46)
at org.apache.arrow.memory.RootAllocator.(RootAllocator.java:43)
at org.apache.gluten.memory.arrow.alloc.ArrowBufferAllocators.(ArrowBufferAllocators.java:44)
```

Proposed fix

Exclude `DefaultAllocationManagerFactory.class` from the shaded bundle so the classloader always finds it in the upstream Arrow jar — whose URL contains the recognized `memory-unsafe` or `memory-netty` substring. The class is functionally identical (same `FACTORY` field providing an `AllocationManager.Factory`); the only behavioral change is that Gluten no longer forces the unsafe allocator and lets Arrow's own default selection apply (which is `netty` per `DefaultAllocationManagerOption.java:80`).

Alternative mitigation: set `-Darrow.allocation.manager.type=Unsafe` (or `Netty`) on driver/executor JVM options to bypass `CheckAllocator.check()` entirely. This works as a runtime workaround but requires every Spark submission to set the JVM option.

The build-time shade exclusion is preferable because:

• It's a one-time fix — no JVM options needed in every Spark submission
• Works for driver and executor uniformly
• Targets the root cause (redundant bundled class) instead of papering over it