[C++] RecordBatchFileReader performance really bad in S3

[asfimport]

We are using RecordBatchFileWriter to write Arrow type directly to S3 using the S3FileSystem, then using RecordBatchFileReader to read from S3. The write is pretty efficient, write a 50MB finishes within 0.2s. But reading that file is taking 30s, which is definitely too long. Then I did several tests:

1. I tried to use S3FileSystem to read the file into bytes, it's only taking 1s. which somehow makes me believe it's an issue with RecordBatchFileReader
2. Half the size (around 25MB), with RecordBatchFileReader took 17s, without RecordBatchFileReader took 0.28s
3. Double the size (around 100MB), with RecordBatchFileReader took 61s, without RecordBatchFileReader took 2.3s
4. I tried to get all bytes using S3FileSystem first, then create a reader from the bytes. Then read all context from the reader, it's only taking 0.1s. 

Reporter: Lingkai Kong
Assignee: Weston Pace / @westonpace

Related issues:

• [C++] Enable fine grained IO for async IPC reader (duplicates)

PRs and other links:

• GitHub Pull Request #11535

_{Note: This issue was originally created as ARROW-14429. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
From what I recall, the reader seeks around and reads each record batch and its header individually. This means separate requests to S3, for which the latency can add up. This is especially true if the record batches are relatively small or there are a lot of them, and especially true if you're reading from a different S3 region (=higher latency). As you've found, reading all the bytes up front helps.

There are optimizations for this in the datasets project - is reading the file through that interface faster? (You will have to enable the async scanner.)

[asfimport]

David Li / @lidavidm:
Also we can and should probably optimize this in the first place - at the very least, we should be able to read each record batch together with its metadata header, instead of separately like we currently do.

[asfimport]

Lingkai Kong:
@lidavidm  Is there a way to configure how large a record batch is? If we can make a record batch large I guess it would mean fewer requests to S3

Also, haven't tried the datasets project. I will give it a try to see if that is faster or not. 

Thanks for your help!

[asfimport]

David Li / @lidavidm:
[~lingkai2] the size of the record batches in the file is determined when you write the file. How is the file being written?

Also I would note that if you are reading the entire file, and the file is relatively small, it will be hard to do much better than just reading the entire thing into memory first.

[asfimport]

David Li / @lidavidm:
Oh, I forgot to mention, but arrow::io::BufferedInputStream may also help.

I forgot that the IPC reader needs a RandomAccessFile, not a BufferedInputStream, so that won't apply here.

[asfimport]

David Li / @lidavidm:
[~lingkai2] does the Datasets reader perform better?

Also see the pull request; while the API is less convenient, the generator-based reader may be the fastest for you.

[asfimport]

Lingkai Kong:
I realize why the read from RecordBatchFileReader is really slow now. That's because when I write, each batch size is very small. Now I call combine_chunk before I write to DBFS and the read become much faster now. 

And for our case, the file might be quite large so we can't load all the content into memory first

[asfimport]

David Li / @lidavidm:
Thanks for the update.

As mentioned, the PR should speed things up somewhat as well. If you are accessing random record batches in the file, this is the best it can do. However, if you are sequentially scanning through all the record batches in the file, the record batch generator in the PR is likely the fastest - it will read from S3 in chunks to avoid the latency of making requests for each record batch. (It does use more memory though - please follow up if you find that to be an issue.)

[asfimport]

Krisztian Szucs / @kszucs:
@lidavidm @westonpace is this resolved by https://issues.apache.org/jira/browse/ARROW-14577 ?

[asfimport]

David Li / @lidavidm:
Yes, we can close this. [~lingkai2] please report if you do not see an improvement.