Repository navigation
[KYUUBI #7511][SPARK] Start Spark Connect in the Spark SQL engine - #7670
alexandrefimov wants to merge 2 commits into
Conversation
github-actions
Bot
added
kind:documentation
|
The one red check, Could someone re-run it? The |
78ba18e to
9c595bc
Compare
The Spark SQL engine can serve Spark Connect clients next to its Thrift frontend. Connect is started by configuration only - the plugin goes into spark.plugins and the bound endpoint is read back reflectively - because io.grpc is relocated in this module and a compile-time reference to SparkConnectService does not compile here. That also keeps the default spark-3.5 build free of Connect classes. Which engines serve Connect is decided in SparkProcessBuilder, where the values are the server's own. kyuubi.engine.spark.connect.enabled is immutable, but the engine rebuilds its KyuubiConf from every spark.kyuubi.* entry it is handed, so the prefixed copy of that key is dropped from the engine's command line. An engine that would not run as the session user - any share level but USER, or doAs disabled - starts without Connect rather than failing, because a plan submitted over Spark Connect runs as the user the engine runs as, not as the client's user_id. With Connect on, the engine still refuses to start when the runtime cannot authenticate its clients - Spark 3.5 serves Connect with no authentication at all - or when no token is configured: an endpoint that authenticates nobody is worse than no endpoint. The Spark 4.0 requirement is a capability probe rather than a version comparison.
9c595bc to
f7ec0a3
Compare
|
The only failing check is Spark 4.3 / JDK 25: |
Why are the changes needed?
Start Spark Connect alongside the Spark SQL engine's Thrift frontend as the first engine-side slice of Kyuubi issue 7511. Connect is disabled by default and controlled by the server. It requires authentication support and a token. Engines outside USER share level or with doAs disabled continue with Thrift only.
Discovery, Connect-aware lifecycle and generated per-engine tokens remain follow-ups. Deployments must use a Spark build containing the SPARK-58658 Config RPC fix.
How was this patch tested?
After merging master, Spark 3.5.8 / Scala 2.12 / JDK 17 and Spark 4.0.3 / Scala 2.13 / JDK 21 each passed 11 launcher tests, three server configuration tests and the generated settings check. The configuration tests used minimal Spark-home fixtures.
dev/reformatpassed. Full process-launch tests and a live Connect smoke still need a Spark distribution.Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Codex