[EPIC] Move Ballista to new arrow-ballista repo

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This is a long-term goal and there are some steps to get there but I would like to discuss this with the community.

Rationale for this

• Decouple release process for DataFusion and Ballista
• Allow each project to have top-level documentation and user guides that are targeting the appropriate audience
• Reduce issue tracking and PR review burden for DataFusion maintainers who are not as interested in Ballista
• Help avoid accidental circular dependencies being introduced between the projects (such as datafusion-cli crate has circular dependency #2433)
• Helps formalize the public API for DataFusion that other query engines should be using

Steps

• Create new arrow-ballista repo
• Push current arrow-datafusion code to new repo
• Remove DataFusion code from new repo - Remove DataFusion crates datafusion-ballista#1
• Set up CI in such a way that we are aware when changes are made to DataFusion that break compatibility with Ballista so that we can ask the contributor to avoid the breaking change, or prepare a PR against Ballista to accommodate the change
• Update the contributing guide and GitHub PR templates

Design / Discussion Document

• https://docs.google.com/document/d/1jNRbadyStSrV5kifwn0khufAwq6OnzGczG4z8oTQJP4/edit?usp=sharing

[andygrove]

@alamb @tustvold Here is another proposal that I would like to get your opinion on

[yjshen]

Cc @liukun4515 @yahoNanJing @mingmwang @thinkharderdev @gaojun2048 @realno as I could remember.

[realno]

The proposal looks reasonable. +1. Looking forward to seeing the public API design.

[EricJoy2048]

That's great!

[tustvold]

No objections from me, but I'm possibly not the right person to ask... As you allude to, we will need to be more careful making breaking changes to DataFusion, but nothing insurmountable.

Reduce issue tracking and PR review burden for DataFusion maintainers who are not as interested in Ballista

We will need to ensure there is still sufficient review capacity for Ballista to thrive, I don't have a good feel for if this is a concern or not.

Allow each project to have top-level documentation and user guides that are targeting the appropriate audience

I like this a lot, the audiences for the projects are likely rather different 👍

[thinkharderdev]

Sounds like a good idea. I'm happy to spend time helping review PRs for Ballista if review capacity is an issue.

[alamb]

I agree this would be a good step -- and help Ballista and DataFusion both to mature. I am fully supportive.

Thank you for the offer @thinkharderdev

[andygrove]

Thanks for the encouraging feedback.

I started a design doc where we can discuss the finer details. https://docs.google.com/document/d/1jNRbadyStSrV5kifwn0khufAwq6OnzGczG4z8oTQJP4/edit?usp=sharing

[andygrove]

@liukun4515 @yahoNanJing @mingmwang Please take a look and let us know if you have feedback. We have started a vote on the mailing list to move forward with this proposal.

[yahoNanJing]

Hi @andygrove, sorry for late response. I'm not opposed to move Ballista to a new repo. However, I still have some concerns.

1. How to manage some features which needs changes in both datafusion and ballista, like configuration refactoring?
2. For some features, like Morsel-driven parallel execution, it leverages the partition for morsel splitting. However, the partition is also used for task splitting in Ballista. The same concept may have conflicts between datafusion and ballista, if ballista is moved out and new features are introduced to datafusion. How should we avoid that?