[C++] Move Acero out of libarrow

[wjones127]

Describe the enhancement requested

libarrow_acero? libacero?

The core libarrow is largely stable while Acero remains experimental. It would be nice to eventually let these libraries evolve somewhat independently, like arrow-rs and datafusion.

• [C++] Second block of refactoring to move acero out of libarrow #34630

Component(s)

C++

[wjones127]

libarrow_dataset depends on Acero, and is tightly integrated with it. Should it be moved into libacero? or renamed libacero_dataset?

[kou]

I prefer to libacero because Gandiva uses libgandiva not libarrow_gandiva.
ADBC also uses libadbc_* not libarrow_adbc_*.

[jorisvandenbossche]

What part would we want to move to a separate library?
Right now Acero lives in arrow/compute/exec/, right? Would we only move that, or the full of arrow/compute ? (xref ARROW-8891 - "Split non-cast compute kernels into a separate shared library")

[rok]

How usable are compute kernels now without Acero? How usable is Acero without kernels?

[jorisvandenbossche]

How usable is Acero without kernels?

Acero would then of course have to depend on libarrow (or libarrow_compute) for the kernels.

How usable are compute kernels now without Acero?

Very usable (if you are OK with the numpy paradigm of having eager functions for all operations). The current integration in pandas to work with arrow data is built on top of plain compute kernels.

[rok]

So then another option is to split further to libkernels, libacero. I don't have an opinion on the split though :).

[paleolimbot]

I would love to see this split, although the ability to distribute it separately (e.g., so that a separate R or Python 'acero' package could exist) is something I hope this would be a step in the direction of.

[wjones127]

What part would we want to move to a separate library?
Right now Acero lives in arrow/compute/exec/, right? Would we only move that, or the full of arrow/compute ? (xref ARROW-8891 - "Split non-cast compute kernels into a separate shared library")

Yes, just arrow/compute/exec/. Splitting compute functions is its own issue and much messier.

[westonpace]

libacero will be easier than libkernels (there are still parts of the core lib that depend on compute kernels, I think) though both seem valid.

The only gray area I can see is group-by. There is currently a standalone group-by implementation that runs outside of Acero, that is marked internal, that I am trying to get rid of.

My vote would be that "group-by" is a part of Acero. Note, the hash_ aggregate kernels themselves could remain as part of compute/kernels. Aggregations would still be usable outside of Acero.

If that's the case then you could also move arrow/compute/row to Acero (and probably move it to arrow/compute/exec/row while you are at it).

I prefer to libacero because Gandiva uses libgandiva not libarrow_gandiva.
ADBC also uses libadbc_* not libarrow_adbc_*.

Both datasets and substrait will need to depend on libacero. It would be a bit weird to have libarrow_dataset -> libacero -> libarrow_compute. I'm ok with the weirdness though.

[wjones127]

Both datasets and substrait will need to depend on libacero. It would be a bit weird to have libarrow_dataset -> libacero -> libarrow_compute. I'm ok with the weirdness though.

Yeah that's why I was contemplating renaming libarrow_dataset to libacero_dataset.

[wjones127]

I spent some time looking at the necessary changes, and it occured to me with have a bit of a mess with namespaces as well. We introduced a library called libarrow_substrait and that contains the namespace arrow::engine. Nothing else uses that namespace. Acero (i.e. the code in src/arrow/compute/exec all lives in the arrow::compute namespace.

Although I'd like to have this issue somewhat limited in scope, it would be nice to know where we are trying to go with the library structure. I created a document outlining the current structure, an a couple ideas of a slightly more sane structure:

https://docs.google.com/document/d/1IcQfqFlvMgM3GreNbAvax0_2XKFHRdmcs9nqTPQlhOs/edit?usp=sharing

I welcome any comments or alternative suggestions.

[westonpace]

Thanks for the detailed analysis and the suggestions. engine echos back to a time when we were thinking there might be potentially more formats like Substrait that could drive the engine. It was an early standin for "acero" as the name wasn't agreed at the time. I'm fine getting rid of the term "engine" and merging substrait & acero and agree that is a good idea.

I'm a little hesitant about lumping "datasets" with Acero. Datasets is somewhat heavyweight, it requires parquet, orc, csv, filesystems, etc. None of those things are relevant to the core Acero. One could imagine Acero being used without any need for file I/O. For example, flight input, apply some compute operations, flight output.

So my preference would be second chart you drew that has libacero and libacero_dataset.

[icexelloss]

For what it's worth, as a Acero user I currently don't use dataset and don't see myself using it in the near future. So I am +1 not putting dataset into libacero and have a seperate so files for it.