ARROW-4219: [Rust] [Parquet] Initial support for arrow reader.

[liurenjie1024]

Initial support of arrow reader, which reads parquet into arrow record batch.

[liurenjie1024]

@sunchao @andygrove @nevi-me @paddyhoran Please take a look when you are available.

[andygrove]

LGTM so far but I have a couple of questions in this review

[andygrove]

We have a very similar trait in DataFusion, only it inherits Sync + Send too so that it can be passed between threads. Maybe we could do that here too? See https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/physical_plan/mod.rs#L44-L50

[liurenjie1024]

I don't think it's safe to make this trait as Send+Sync because the reader used some unsafe data(e.g. MutableBuffer).

[andygrove]

OK, makes sense.

[andygrove]

What is returned if there are no more batches? Shouldn't this return Result<Option<RecordBatch>> ?

[liurenjie1024]

Currently if we reached EOF, it returns zero length array. But I think it's better to change return type to Result<Option<RecordBatch>> . Will fix it.

[liurenjie1024]

@andygrove I need to upload some generated files to arrow-testing repo for this integration test. How am I supposed to do that? Should I submit another PR in arrow-testing project?

[github-actions]

https://issues.apache.org/jira/browse/ARROW-4219

[andygrove]

@liurenjie1024 If you are adding Parquet files then you can PR against https://github.com/apache/parquet-testing but please note the comments in the last PR merged there which suggests we look into alternate approaches: apache/parquet-testing#9

[liurenjie1024]

@andygrove I need to upload not only parquet file, but also json files and proto files. Here is my approach to do integration test: I use protobuf to define schema, then generate both json and parquet file with that schema using java code. Then I use rust code to load parquet and json file to compare them. I guess maybe it's better to upload them into arrow-testing library?

[andygrove]

@liurenjie1024 I think it would be good to start a discussion about this on the mailing list. I'm not sure what advice to give on this.

[alippai]

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

[liurenjie1024]

@andygrove Sorry for late reply. I've submitted a PR in arrow-testing repo: apache/arrow-testing#11
Let's discuss there.

[liurenjie1024]

@andygrove This is ready for review now.

[nevi-me]

LGTM

[nevi-me]

The new ParquetFileArrowReader is faster than the existing row based iteration, looks stable for my dataset (but, the Utf8 datatype is missing)

@liurenjie1024 is this an intentional limitation for now? (

arrow/rust/parquet/src/arrow/array_reader.rs

Lines 219 to 222 in 8b915b3

	(arrow_type, _) => Err(general_err!(
	"Reading {:?} type from parquet is not supported yet.",
	arrow_type
	)),

)

Also, my apologies for the late review.

[liurenjie1024]

@nevi-me No, it just takes time to implement. Will implement in future.

[andygrove]

LGTM. Thanks @liurenjie1024