Create a new roadmap for remainder of 2026

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
There has been an increase in contributions from multiple contributors and companies recently. I think it would be a good idea to discuss a roadmap for the remainder of 2026 (~4 months) so that we communicate to the community what we are working on and where we are looking for help (contributing, reviewing, testing, etc).

Describe the solution you'd like
Let's use this issue to discuss what everyone plans on workong on over the next few months, then we can create (or update) a roadmap section in the contributor guide.

Describe alternatives you've considered

Additional context

[andygrove]

I don't have specific features that I plan on working on. I am mostly testing with TPC-H @ SF1TB and using that to drive my contributions (bug fixes, perf optimizations, etc). I'm also planning on helping with PR reviews to keep momentum going.

@milenkovicm @martin-g @phillipleblanc @avantgardnerio @villebro @akshaychitneni @sandugood would you mind commenting on the issue with some brief notes on areas you plan on working on in the near future?

[avantgardnerio]

I would like to finish up work on parallel windows, and also dive into benchmarks to see which if any could be improved given the RangeRepartitioning primitives now in place. Particularly anything with:

1. a plan where Spark goes faster & uses range partitioning
2. any plan with "hot keys" in hash partitioning - low cardinality or skewed data

These would be obvious candidates for improvements by switching to range re-partitioning, though there might be others. I'd also like to continue helping with manageability & HA, as well as general performance improvements. (it's always fun to see a benchmark go faster :)

Also, I'd like to upstream into DataFusion and arrow some of the operators and enhancements that belong there.

[milenkovicm]

I dont have a plan at the moment, waiting for weather to turn back to normal operations here:)

something on top of my head that i would like to work on

• [EPIC] Improve Cluster Observability #1426 I'd like to have a look at (per stage) flamegraphs, I'd need to try few things to see if this is going to be something feasible
• once feat(python): integrate Ballista through DataFusion FFI #2252 lands would focus on python cleanup and improvements

open for other suggestions

[avantgardnerio]

Speaking of Python, are Python UDFs possible / on the roadmap? Might be good to consider other ones such as wasm too.

[milenkovicm]

once python support lands we can add them, i think (but i can be wrong) datafusion-python already has serde functionality for udf, if not i have working prototype https://github.com/milenkovicm/ballista_python

[andygrove]

Speaking of Python, are Python UDFs possible / on the roadmap? Might be good to consider other ones such as wasm too.

Not entirely related, but there is PR to add support for Rust UDFs (dynamically loaded) to Comet. Maybe useful here too.
apache/datafusion-comet#4459

[villebro]

@andygrove as per this comment, I'd like to work on

• scheduler HA (horizontally scalable, shared state in pluggable backend)
• introducing a HA result proxy service (also horizontally scalable)

[milenkovicm]

@villebro maybe related to your interests, I was thinking to create a "datafusion gateway" which could route queries based on pluggable policies to ballista or datafusion (running on k8s) i did not think i through 😀

[akshaychitneni]

I am planning to contribute to df.cache - #1829 based on my related work with kubeflow data cache and interested to work on kubernetes related improvements

[sandugood]

I am planning to contribute to df.cache - #1829 based on my related work with kubeflow data cache and interested to work on kubernetes related improvements

Is it parallel or colliding with #1993?

[sandugood]

I don't have a particular plan at the moment, but I'd be glad to pick up some of the tasks/issues that might arise.

However, I think it would be nice if Ballista could support different clients for connectivity, such as DBeaver, BI systems, or Python via Flight SQL (so it's related to #2298, I guess). Having Ballista only work with Rust code is somewhat limiting, to say the least.

+1 on @milenkovicm cluster observability

[akshaychitneni]

I am planning to contribute to df.cache - #1829 based on my related work with kubeflow data cache and interested to work on kubernetes related improvements

Is it parallel or colliding with #1993?

It should be parallel/complimentary. I see .cache() as reusing results from executors memory within a session similar to datafusion MemTable path. Let me know if you think otherwise

[Dandandan]

Nothing specific from me - I am considering to add prefetching to DataFusion apache/datafusion#23492 which should help the subpar performance on S3

[villebro]

@villebro maybe related to your interests, I was thinking to create a "datafusion gateway" which could route queries based on pluggable policies to ballista or datafusion (running on k8s) i did not think i through 😀

@milenkovicm love the idea! also related, I've been thinking about adding the option to introduce tiered result storage down the road for perf/redundancy tuning. Some that come to mind:

• in worker memory
• on worker disk
• object storage (large result sets)
• Valkey-like distributed cache (small result sets)
• etc

With a dedicated result proxy it would be easy to introduce these incrementally as needed, as the client facing interface wouldn't change. Probably not high priority now, but maybe later..