Moving the DataFusion integration into a separate repository

[gabotechs]

Continuing the #2992 discussion in an issue as requested by @andygrove, so that updates get sent to the mailing list.

Citing the original discussion description:

---

I’d like to explore decoupling the development lifecycle of iceberg-datafusion from iceberg-rust by moving the existing DataFusion integration into a crate maintained in a separate repository.

As discussed in datafusion-contrib/datafusion-distributed#599, we are exploring incorporating the integration into datafusion-distributed, making native distributed Iceberg support a core offering of that project. Another option worth exploring would be to create a dedicated repository, such as datafusion-contrib/datafusion-iceberg.

Regardless of its eventual home, the goal would be to allow the community to advance the Iceberg+DataFusion integration without adding review pressure for iceberg-rust maintainers. The integration could evolve at its own pace and be maintained by a different or broader group of contributors.

Some reasons why this may be worth considering:

• There appears to be a vision of making iceberg-rust engine-agnostic (feat(datafusion-ballista) Add Initial Integration for Ballisa Datafusion #2613 (comment), cc @blackmwk).
• Several integration-related PRs have had difficulty receiving review attention in this repository, such as feat(datafusion): Add opt-in eager file scan planning with output partitioning #2671 and feat(datafusion): report IcebergTableScan metrics #2521.
• Other DataFusion-related projects have expressed interest in hosting an Iceberg integration, including Add Iceberg integration datafusion-ballista#2174 and [EPIC] Iceberg integration datafusion-contrib/datafusion-distributed#599.

I’d like to hear from the community (particularly the iceberg-rust maintainers) about the possibility of moving iceberg-datafusion out of the iceberg-rust repository. This could allow projects such as Ballista and datafusion-distributed to share common functionality and contribute to the integration independently of the iceberg-rust release and review lifecycle.

[gabotechs]

RE: Could we just create a new repo that is outside the core project, at least initially? Like apache/datafusion-iceberg?

My reason to think it's better to have in datafusion-contrib is that a lot of the people that would be willing to drive the effort (sponsored by their employer) are not part of the Apache DataFusion Committee, and therefore, development speed would be hurt.

Having it under the apache organization is more likely to result in the same situation we have today in this repository: lack of review power, and we are at risk to be back to where this discussion started.

[alamb]

It seems we have a new repo: https://github.com/datafusion-contrib/datafusion-iceberg

🎉

[milenkovicm]

what would be the reason not to host it under apache/datafusion-iceberg ? would it be less complicated to move it from apache to apache now than from datafusion-contrib to apache in the future?

@andygrove has valid point in #2992 (reply in thread)

[alamb]

It would be fine from my perspective to put it in apache/datafusion-iceberg -- though that will come with the same datafusion restructions (only committers have access, we have to manage it via .asf.yaml)

However, the provenance is definitely clearer

[andygrove]

If we move it out of ASF then want to bring it back in later, we need to go through the ASF IP Clearance process, which is quite a lot of work for the PMC. Keeping it within ASF avoids that.

[andygrove]

It would be fine from my perspective to put it in apache/datafusion-iceberg -- though that will come with the same datafusion restructions (only committers have access, we have to manage it via .asf.yaml)

However, the provenance is definitely clearer

I don't understand this comment. Why would only committers have access? Anybody can access and contribute to ASF projects. It is true that PRs would need to be approved by committers.

[andygrove]

It seems we have a new repo: https://github.com/datafusion-contrib/datafusion-iceberg

🎉

Was there a vote in either the Iceberg or DataFusion PMC for moving an existing ASF project outside of the ASF? It seems like something that should involve consensus from the respective communities?

[andygrove]

Thanks for moving the discussion to an issue @gabotechs. I appreciate that you moved the original context over, but we lost some of the following discussion. As I posted on the original discussion in #2992 (reply in thread), moving the ASF code to a non-ASF repo could actually exclude some of the existing contributors from continuing to contribute, so I don't believe that we can just go ahead and do this without a vote.

You mentioned earlier that moving the repo to a non-ASF repo would allow more people (who are not Apache DataFusion committers) to contribute, but you did not include the fact that it would also exclude some existing DataFusion committers from continuing to contribute, and it is ok that you were not aware of this information.

My opinion is that we will need a formal vote on this issue.

[gabotechs]

what would be the reason not to host it under apache/datafusion-iceberg? would it be less complicated to move it from apache to apache now than from datafusion-contrib to apache in the future?

Pretty much what my message above #3029 (comment)

moving the ASF code to a non-ASF repo could actually exclude some of the existing contributors from continuing to contribute, so I don't believe that we can just go ahead and do this without a vote.

My reason to think this is not a problem, is that in datafusion-contrib any admins are free to give write access to the repo to anyone they find appropriate, including any existing contributor, while this is not true in apache.

but you did not include the fact that it would also exclude some existing DataFusion committers from continuing to contribute, and it is ok that you were not aware of this information.

Wouldn't it be as easy as just adding any DataFusion committers who want to contribute just write access to the repository?

My opinion is that we will need a formal vote on this issue.

👍 this sounds reasonable

[dannycjones]

@blackmwk mentioned this in the discussion #2992 (comment)

this repo will still need to depend on iceberg-datafusion integration for data driven integration tests

I fear that moving out of iceberg-rust will result in the integration testing no longer being part of the loop of development, and we'll only discover faults in iceberg-rust during release candidates or later.

[blackmwk]

My opinion is that we will need a formal vote on this issue.

+1. I think we should have a vote on both iceberg community and datafusion community. And before that, it's better to raise a discussion on both community's dev list so that more people could be aware of.

As I said earlier, I'm leaning toward to move it to apache repo so that we could have better maintaince.

[gabotechs]

It seems we have a new repo: https://github.com/datafusion-contrib/datafusion-iceberg

I've moved this repository to private to avoid confusion in the community until this is sorted out.

I fear that moving out of iceberg-rust will result in the integration testing no longer being part of the loop of development

I imagine this can be addressed by keeping the current (or a reduced) integration here for testing purposes, which (at least I) was considering it as priced in in any outcome.

I'd say keeping the DataFusion integration here is also an option. In that case, something like granting some DataFusion PMC members committer access to this project so that they can help with reviews would be ideal.

[blackmwk]

I fear that moving out of iceberg-rust will result in the integration testing no longer being part of the loop of development

This also reminds a caveat for me: when we want to develop sime data integraiton tests, we need to create pr in another repo.

[gabotechs]

This also reminds a caveat for me: when we want to develop sime data integraiton tests, we need to create pr in another repo.

Other Apache projects use a CODEOWNERS file for enforcing different parts of the codebase to be reviewed by at least one of the specified members. For example https://github.com/apache/airflow/blob/main/.github/CODEOWNERS.

If the general preference is to keep it under ASF, and there's still value to maintain the DataFusion integration in iceberg-rust, inviting the most active DataFusion PMC members to also be committers of this project, and restricting which PRs they can approve with a CODEOWNERS file sounds like an attractive option.

[dannycjones]

If you're able to join, it may be worth raising as part of the Iceberg Rust community sync which is at 3pm UTC today. (sync notes: https://docs.google.com/document/d/1_1yhzaWOr28aURogoe7a8LONdLRDXfo88jYPHbLHIg4/)