Ballista 54.1.0 Release

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I would like to release Ballista 54.1.0 to pick up DataFusion 54.1.0 and also ship any new improvements to Ballista since the 54.0.0 release.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[milenkovicm]

I believe 54.0.0 could pick up DataFusion 54.1

[andygrove]

There have been a number of correctness bug fixes since the 54.0.0 release, so I would like to create another Ballista release soon.

Given that our major version number is tied to DataFusion major version, the next release should be 54.1.0 (or 54.0.1).

I am thinking that we should just merge all changes from main into branch-54, possibly excluding #2045, because that is technically a breaking API change.

We would also need to decide whether to wait for the DataFusion 54.1.0 release. This would help fix more correctness issues exposed by TPC-DS.

WDYT @milenkovicm @phillipleblanc @avantgardnerio

[phillipleblanc]

Merging ~all changes from main and waiting for DataFusion 54.1 makes sense to me.

[milenkovicm]

Merging all from main makes sense.

In theory we don't need to wait for DataFusion 54.1 as it could be updated after release, only if we want to keep same version number semantics

[andygrove]

The DataFusion 54.0.1 54.1.0 release vote already started, so I'll wait for that release and then start the Ballista 54.1.0 release process.

[andygrove]

I back-ported some fixes for correctness issues:

• fix: [branch-54] disable the SortMergeJoinExec broadcast conversion by default #2130
• fix: [branch-54] map a task's partition through UnionExec when restricting file scans (backport of #2070) #2131
• fix: [branch-54] keep EmptyExec partition count intact across stage serialization (backport of #2062) #2132

I also bumped the DataFusion dependency to 54.1.0

@phillipleblanc @milenkovicm @avantgardnerio Should we back-port anything else before releasing 54.1.0?

[milenkovicm]

do we still want to release 54.1 or we wait for datafusion 55 ?

[phillipleblanc]

I would vote for a 54.1.0 release with the correctness fixes that Andy mentioned above.

[andygrove]

Agreed on shipping 54.1.0 with the correctness fixes rather than waiting for DataFusion 55.

I went back through everything that has landed on main since branch-54 was cut (133 commits) and triaged the bug fixes. Six of them cherry-pick onto branch-54 cleanly with no adaptation at all, so I have opened them as draft backport PRs:

• fix: [branch-54] enable the executor client pool by default (backport of #2069) #2234 (backport of fix: enable the executor client pool by default #2069) enable the executor client pool by default. client_ttl defaults to 0, which disables pooling, so the shuffle-read path opens a new TCP connection per fetch and a single shuffle-heavy query exhausts the host's ephemeral ports. This one only shows up once you have a second executor, which is every real deployment. The SF1 TPC-DS gate went from 11 OK / 77 FAILED to 88 OK / 0 FAILED.
• fix: [branch-54] keep GROUP BY-less aggregates when propagating empty stages (backport of #2194) #2235 (backport of fix(scheduler): keep GROUP BY-less aggregates when propagating empty stages #2194) keep GROUP BY-less aggregates when propagating empty stages. Silent wrong answer under AQE, a dropped grand-total row.
• fix: [branch-54] bind the gRPC listener before registering with the scheduler (backport of #2225) #2236 (backport of fix(executor): bind the gRPC listener before registering with the scheduler #2225) bind the gRPC listener before registering with the scheduler. Startup race that gives ECONNREFUSED on registration and kills the executor.
• fix: [branch-54] send the shutdown notification before dropping the notifier (backport of #2107) #2237 (backport of fix: executor hanging on ctrl+c #2107) send the shutdown notification before dropping the notifier. Executor hangs on ctrl+c.
• fix: [branch-54] spill sort shuffle writer when the memory pool rejects a reservation grow (backport of #2061) #2238 (backport of fix: spill sort shuffle writer when the memory pool rejects a reservation grow #2061) spill the sort shuffle writer when the memory pool rejects a reservation grow. The writer was buffering through a rejection and under-reporting usage to the shared pool.
• fix: [branch-54] see through Shared wrapper when classifying task IO errors (backport of #2119) #2239 (backport of Retryable IO errors misclassified when wrapped in DataFusionError::Shared #2119) classify task IO errors through find_root(). Retryable IO errors wrapped in DataFusionError::Shared were being marked non-retryable, so a transient object store blip fails the whole job.

All six are unmodified git cherry-pick -x of the original commits. cargo fmt --check is clean and cargo check --workspace --all-targets builds with no warnings on all six applied together. I left the test runs to CI.

One thing worth a look before merging: #2234 changes a default value (--client-ttl from 0 to 30s). It is the highest impact fix of the six, but it is a behaviour change rather than a pure bug fix, so it is the one I would most expect people to have an opinion on.

There are a few more fixes on main that I deliberately did not propose, because they do not cherry-pick and would need real rework:

• fix(scheduler): execute null-aware anti joins in one task #2188 (null-aware anti joins, so distributed NOT IN returns wrong results). I confirmed the bug is present on branch-54, but the fix is around 800 lines across the static planner, AQE join selection and config. If we want it in 54.1.0 it needs a purpose-built minimal version.
• fix: Various fixes for AQE join selection #2124 and feat: make broadcast_join_threshold_bytes/rows authoritative under AQE and the static planner #2086 (AQE join selection). These form a chain and collide with broadcast_sort_merge_join_enabled from fix: [branch-54] disable the SortMergeJoinExec broadcast conversion by default #2130, which main has since deleted.
• fix(scheduler): fail running jobs when all executors are lost #2212 (fail running jobs when all executors are lost). Adds a new config key and depends on the chaos harness from test: add HA chaos harness that drives the scheduler's fault-tolerance paths on a real cluster #2026 that is not on branch-54.
• feat(executor): default to a bounded, auto-sized memory pool #2160 and fix: propagate sort shuffle memory_limit_per_task_bytes to executors #2090 (executor memory pool defaults and propagating memory_limit_per_task_bytes). Both rebase without much trouble, but feat(executor): default to a bounded, auto-sized memory pool #2160 changes default memory behaviour, which felt like more than a patch release should carry. Happy to add them if others disagree.

Two things turned out not to be needed:

• fix: Enable more TPC-DS queries #2064 (INTERSECT / EXCEPT dropping NULLs). The actual fix was in DataFusion and branch-54 already ships 54.1.0, so this is fixed there. Only the regression test would port over.
• fix: stop every task running every union branch at low partition counts #2186 (every task running every union branch). That is a consequence of multi-partition tasks from Multi-partition tasks: partition_slice plumbing + SortShuffleWriter refactor #2038, which is main only. branch-54 runs one partition per task and is not exposed.

@milenkovicm @phillipleblanc @avantgardnerio does this look like the right set, and is there anything from the second list you would want pulled in before we cut the release?

[andygrove]

Blog post PR: apache/datafusion-site#202

[andygrove]

This release was shipped