[Java][Dataset] Implement Datasets Java API

[asfimport]

Porting following C++ Datasets APIs to Java:

• DataSource
• DataSourceDiscovery
• DataFragment
• Dataset
• Scanner
• ScanTask
• ScanOptions

Reporter: Hongze Zhang / @zhztheplayer
Assignee: Hongze Zhang / @zhztheplayer

Related issues:

• [JAVA][C++]Support Parquet Read and Write in Java (is part of)

PRs and other links:

• GitHub Pull Request apache/arrow#7030

_{Note: This issue was originally created as ARROW-7808. Please see the migration documentation for further details.}

[asfimport]

Liya Fan / @liyafan82:
Thanks for opening this issue.
Personally, I would like to see these features added to Java code base.
But I am not sure if the community are willing to support them.

[asfimport]

Hongze Zhang / @zhztheplayer:
The major goal is to adopt existing C++ file formats in Java code (maybe via JNI bridge). To have the Datasets APIs implemented in Java is the most reasonable approach as users would be able to access any of the layers defined in C++ Datasets API. Could others in the community please reconfirm this proposal (I recall that we have a discussion for this months ago)? And I can file PRs after then.

[asfimport]

David Li / @lidavidm:
We are interested in bridging the C++ implementation of Parquet in particular to Java. But I agree, supporting JNI bindings to a large API might be difficult.

[asfimport]

Hongze Zhang / @zhztheplayer:
Yes the C++ Parquet reader is useful for a lot of users including us. Now some new features are only available with Datasets, for example, ARROW-6952. With Java API bridged Java users would be able to use the functionalities in their code easily without going down to JNI/C++.

[asfimport]

Micah Kornfield / @emkornfield:
I think the mail list conversation decided on try to do a very small shim 1 or 2 methods not a full 1-1 JNI mapping?

[asfimport]

Hongze Zhang / @zhztheplayer:
I am not pretty sure but based on the mail discussion I would think of mapping 1 or 2 methods via JNI is not final solution but something we can get started with. And, as for format Parquet, users may need access to different Datasets layers such as DataFragments for Parquet files, ScanTasks for RowGroups, even one may need to decide if C++ level post-scan filter should be enabled/disabled, if partition filter should be applied, and so on. One or two methods can not cover all of this.

And maintaining a JNI-based Datasets API may not be a heavy workload, because on Java side, things are just mirrored to some basical Datasets concepts like DataSource, DataFragment, and should keep away from re-implementing low-level logic like scaning, projecting, filtering, etc. But everything in C++ could be available in Java which is important to many users.

[asfimport]

Micah Kornfield / @emkornfield:
@fsaintjacques  I think you weighed in the mailing list discussion, what are your thoughts on this?

[asfimport]

Ben Kietzman / @bkietz:
The dataset API is not stable; a full 1:1 mapping will be more work to maintain. For example https://issues.apache.org/jira/browse/ARROW-7886 would remove Source and SourceFactory altogether, which would necessitate refactoring both the JNI binding and the Java which uses it. I recommend exposing only classes which are directly useful for a minimal use case, then exposing classes as they become necessary in follow ups.

@fsaintjacques's recommendation on the mailing list would be an excellent starting point. Alternatively, I recommend following the initial R binding work: https://github.com/romainfrancois/arrow/blob/9dfba2ea8949a0a0a17393976a97d3a34dc63d39/r/R/dataset.R
This minimally exposes Source, Dataset, Scanner, and the corresponding factories. Scans result in a materialized Table (so ScanTasks, Fragments, etc may remain hidden) and take full advantage of predicate/projection push down.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
I concur with @bkietz  here, prefer a high-level approach, e.g. it should start with the factory, dataset and getting the result of a scan, then you can expose more and more of the inner classes, instead of 1-1 with all the classes.

Also note that a JNI bridge means one more maintenance burden; we already did a bunch of renames and it was cumbersome to do it for each "native" binding too, e.g. DataSource is now called Source.

[asfimport]

Hongze Zhang / @zhztheplayer:
Thanks guys for your suggestions! And so sorry for such a late reply. I've been busy on some other stuffs and now I am continuing to work on this. Actually in my organization we have been maintaining an runnable implementation[1] for several months, it may not be completely ready for making an upstream PR but still be showing my main designs.

I see your suggestion with the preference of high-level approach, with which actually I agree. In my current implementation, there might be some classes that look like something "lower level" in Java, such as DataFragment[2], or ScanTask[3], but further developers don't ever have to make implementations for specific source formats - we have NativeDataFragment[4] or NativeScanTask[5] to cover all cases. The same design is applied to DataSource[6][7] so we only have to bridge c++ DataSourceDiscovery implementations in further development. Here is an example[8] from us to add an arrow::dataset::SingleFileDataSource and use it from Java.

And I know in the newest upstream code C++ API has been reworked a lot (the remove of DataSource, rename of DataSourceDiscovery and so on). So there should be some extra work to me to make things match during rebasing.

Sorry again for the delay, and any thoughts please let me know. Thanks.

[1] https://github.com/zhztheplayer/arrow-1/commits/ARROW-7808
[2] https://github.com/zhztheplayer/arrow-1/blob/b2e98bfaf24565a6b23ecd36b9c3c2c264be51f8/java/dataset/src/main/java/org/apache/arrow/dataset/fragment/DataFragment.java
[3] https://github.com/zhztheplayer/arrow-1/blob/b2e98bfaf24565a6b23ecd36b9c3c2c264be51f8/java/dataset/src/main/java/org/apache/arrow/dataset/scanner/ScanTask.java
[4] https://github.com/zhztheplayer/arrow-1/blob/b2e98bfaf24565a6b23ecd36b9c3c2c264be51f8/java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeDataFragment.java
[5] https://github.com/zhztheplayer/arrow-1/blob/b2e98bfaf24565a6b23ecd36b9c3c2c264be51f8/java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeScanTask.java
[6] zhztheplayer@b2e98bf#diff-deea6cb88ea63d76f71b7b4cfd173206
[7] https://github.com/zhztheplayer/arrow-1/blob/b2e98bfaf24565a6b23ecd36b9c3c2c264be51f8/java/dataset/src/main/java/org/apache/arrow/dataset/jni/NativeDataSource.java
[8] zhztheplayer@7cb13b9

[asfimport]

Hongze Zhang / @zhztheplayer:
I've temporarily marked this with target release 3.0.0 as most of the codes are finished with reviewing except for JNI-based memory pool part. We can put off to later version any time we think is needed.

[asfimport]

Micah Kornfield / @emkornfield:
Issue resolved by pull request 7030
#7030