feat: Column statistics support v1 - #1916
Conversation
andygrove
left a comment
andygrove
left a comment
There was a problem hiding this comment.
Thanks for starting this. Collecting shuffle-time column statistics is a solid foundation for smarter broadcast and AQE decisions, and folding null counts across the input batches is a clean place to gather them. I left a few notes. The main one is that the collected stats are currently dropped at the scheduler, so it might be worth landing a consumer or a storage hook alongside the collection, or gating it behind a config until then. martin-g's note on python/Cargo.lock looks right, and it appears to be what is failing the Test Python Release jobs, since pyballista fails to compile against a mismatched DataFusion version. Reverting that lockfile to main and regenerating only if a dependency really changed should get those green.
This review was prepared with the help of an LLM.
|
Thank you . Let me address the PR comments and update here |
|
Created issue for part 2 : #1960 ( followup) |
# Conflicts: # python/Cargo.lock
# Conflicts: # ballista/core/proto/ballista.proto # ballista/core/src/execution_plans/shuffle_writer.rs # ballista/core/src/execution_plans/sort_shuffle/writer.rs # ballista/core/src/serde/generated/ballista.rs # ballista/executor/src/execution_engine.rs # ballista/executor/src/lib.rs # ballista/scheduler/src/scheduler_server/mod.rs # ballista/scheduler/src/state/execution_stage.rs # ballista/scheduler/src/test_utils.rs
3 participants
Which issue does this PR close?
Closes #1827
Introduced new
TaskColumnStatsproto to compute null count , distinct values (TBD) for each column at the partition / stage level. Follow up PRs should leverage this information to make better decisions like broadcast / AQE etcRationale for this change
What changes are included in this PR?
Are there any user-facing changes?