ARROW-11797: [C++][Dataset] Provide batch stream Scanner methods #9589

bkietz · 2021-02-26T21:15:26Z

No description provided.

github-actions · 2021-02-26T21:15:47Z

https://issues.apache.org/jira/browse/ARROW-11797

westonpace

Here's a quick look. I think my main concern is the potential race condition in Pop. The other comments are more of "I'm about to replace this with something else so lets not worry"

westonpace · 2021-02-26T21:24:18Z

cpp/src/arrow/dataset/scanner.cc

I'll be replacing this with something better so I don't know how much we care to worry but this is not ideal. For example, with parquet, this would fetch metadata for every file in the scan before starting to read any individual file. It introduces more latency than necessary.

Also, I'm not sure how this will interact with parquet preloading.

westonpace · 2021-02-26T21:26:38Z

cpp/src/arrow/dataset/scanner.cc

Again, what I'm working on will work around this so maybe not stress at the moment but there's no back-pressure here. If the batch consumer is not fast enough and the dataset is larger than RAM the system will run out of RAM.

It's a bit odd because you are fixing ARROW-11800 here (though without pressure) and then I'll be breaking it again with my implementation (the first pass of my impl will have back pressure and parse loaded buffers a bit more serially)

cpp/src/arrow/dataset/scanner.cc

westonpace · 2021-02-26T21:41:03Z

cpp/src/arrow/dataset/scanner.h

Why both Scan(visitor) and ToBatches? Couldn't you just do ToBatches().Visit(visitor)?

The difference is ToBatches().Visit(visitor) would invoke visitor exclusively on the thread which called Visit() whereas Scan(visitor) invokes visitor in the scan's thread pool. This method is speculative; I'm not sure we'd want to provide that but I included it as an example

lidavidm · 2021-04-07T18:58:49Z

I've updated this in case we change how to proceed with ARROW-7001:

Rebased against master
Added Python/R deprecation warnings for Scan(), since either way, we're planning to remove ScanTask in 5.0
Changed the iterator so that scan tasks are materialized and executed iteratively, to better handle larger-than-memory datasets
Removed Scan(visitor) to reduce API surface (we can add it if the need comes up)

westonpace

This is very cool. Now I just need to provide the V2 :)

westonpace · 2021-04-07T20:50:16Z

cpp/src/arrow/dataset/scanner.cc

Style nit: member variables should be at the bottom of a struct:

https://google.github.io/styleguide/cppguide.html#Declaration_Order

westonpace · 2021-04-07T20:53:40Z

cpp/src/arrow/dataset/scanner.cc

I'm not entirely certain it is safe to modify iteration_error outside the mutex. What happens if Pop is accessing it at the same time?

westonpace · 2021-04-07T20:58:37Z

cpp/src/arrow/dataset/scanner.cc

This is also not safe to do outside the mutex.

westonpace · 2021-04-07T20:59:52Z

cpp/src/arrow/dataset/scanner.cc

Nit: best to relinquish the mutex before calling notify

westonpace · 2021-04-07T21:02:25Z

cpp/src/arrow/dataset/scanner.cc

I think I'm just not seeing it but what causes the above loop to exit if there is an error scanning?

Eventually we'll run out of scan tasks/batches, since an error in getting the next scan task won't stop the current scan tasks from eventually completing. So really it's "all scan tasks drained (and maybe we didn't start all of them due to a failure)"

westonpace · 2021-04-07T21:07:54Z

cpp/src/arrow/dataset/scanner.h

Nit: If I see ToBatches I expect RecordBatchVector. I had named it ScanBatches but I don't feel too strongly on this point.

You have a point. I didn't want to take ScanBatches since your method has a different signature. But maybe that could be ScanIndexedBatches or something.

Hmm, the issue I ran into was that ScanBatches was used by FileSystemDataset::Write and it needed the fragment info in order to have access to the fragment's partition expression. So at a minimum I needed to return "record batch & partition it came from".

I think there was some discussion (either on the ML or some JIRA/PR) about the benefit of keeping the fragment available as the user might want to know where the batch came from.

Can you modify the ScanBatches here to return a RecordBatch/Fragment pair? I could align my ScanBatches with that (PositionedRecordBatch is overkill).

westonpace · 2021-04-07T21:09:28Z

cpp/src/arrow/dataset/scanner_test.cc

Consider making a parameterized test instead so it is easier to trace failures if needed?

westonpace · 2021-04-07T21:10:31Z

cpp/src/arrow/dataset/scanner_test.cc

Is this just testing a scan of one scan task & one batch? It seems we would want to test more than that.

MakeScanner generates a union dataset of 2 InMemoryDatasets each of which repeats the batch 16 times so we should have 32 scan tasks total.

westonpace · 2021-04-07T21:11:44Z

python/pyarrow/tests/test_dataset.py

At the very least we should create a JIRA to migrate these over to the new scan. Although I'm wondering if we want to just do that now because that would give us a lot more coverage of the non-deprecated path.

I'll do it now.

westonpace · 2021-04-07T21:12:48Z

r/R/dataset-scan.R

Nit: Does this comment still make sense?

nealrichardson · 2021-04-08T15:26:24Z

r/src/dataset.cpp

Maybe this is a naive question but why are there two versions of TakeRows?

Hmm, Ben split it into two functions (a pure-Arrow implementation, and the R binding) presumably for convenience. But maybe we can just port the implementation into the C++ library and expose it to Python as well?

nealrichardson · 2021-04-08T15:28:38Z

r/R/dataset-scan.R

This is nice but probably not necessary, this was a (de facto) internal method. If it were me I'd just delete it, but since you've already done this, might as well keep it. Can you please just make a JIRA to delete this so we don't forget to clean this up after the release?

We've been using ARROW-11782 so I'll note that in this PR and update the issue.

lidavidm · 2021-04-09T22:43:12Z

@westonpace @bkietz this should be ready now, if either of you wants to take another look.

lidavidm · 2021-04-12T18:16:23Z

I restored the Scan(Visitor) overload that Ben pointed out in ARROW-12288, updated to use TaggedRecordBatch.

lidavidm · 2021-04-13T13:36:00Z

Rebased (unfortunately, I had to squash all commits or else the rebase would've been a pain).

westonpace · 2021-04-13T18:21:26Z

cpp/src/arrow/dataset/scanner.cc

Is there a JIRA for pushing down the index predicate into the scan?

I filed ARROW-12369.

westonpace · 2021-04-13T18:22:31Z

cpp/src/arrow/dataset/scanner.cc

Would you not want to skip empty arrays (where length == 0)?

Thanks, that also exposed a bug (or well, poor error message) if all indices were out-of-bounds.

cpp/src/arrow/dataset/scanner.cc

lidavidm · 2021-04-15T01:08:03Z

I am not sure why the JNI test is having so much trouble but it passes locally under Docker.

westonpace · 2021-04-15T01:30:23Z

FYI, there are probably some things we could do to improve the JNI build (ARROW-11633)

bkietz requested a review from nealrichardson February 26, 2021 21:15

github-actions bot added Component: C++ Component: R labels Feb 26, 2021

westonpace reviewed Feb 26, 2021

View reviewed changes

bkietz force-pushed the 11797-Provide-Scanner-methods-t branch 2 times, most recently from 522aca4 to 08aec2f Compare March 1, 2021 20:36

lidavidm force-pushed the 11797-Provide-Scanner-methods-t branch from 90818b2 to ba8c952 Compare April 7, 2021 18:56

westonpace reviewed Apr 7, 2021

View reviewed changes

github-actions bot added the Component: Python label Apr 7, 2021

lidavidm marked this pull request as ready for review April 8, 2021 13:05

lidavidm mentioned this pull request Apr 8, 2021

ARROW-9731: [C++][Python][R][Dataset] WIP: Port "head" into C++ #9854

Closed

nealrichardson reviewed Apr 8, 2021

View reviewed changes

lidavidm mentioned this pull request Apr 9, 2021

ARROW-12288: [C++] Create Scanner interface #9947

Closed

lidavidm force-pushed the 11797-Provide-Scanner-methods-t branch 4 times, most recently from 8e494c9 to 52f146e Compare April 9, 2021 21:44

lidavidm force-pushed the 11797-Provide-Scanner-methods-t branch from 70c7bb8 to 5e885cd Compare April 12, 2021 18:15

lidavidm force-pushed the 11797-Provide-Scanner-methods-t branch 2 times, most recently from 2f09670 to 67ed5a8 Compare April 13, 2021 13:35

lidavidm force-pushed the 11797-Provide-Scanner-methods-t branch from 67ed5a8 to dc97bb1 Compare April 13, 2021 14:22

westonpace reviewed Apr 13, 2021

View reviewed changes

bkietz and others added 2 commits April 14, 2021 18:00

ARROW-11797: [C++][Dataset] Provide batch stream Scanner methods

ebc169c

ARROW-11797: [C++][Dataset] Add missing virtual

ae91bfd

ARROW-11797: [C++][Dataset] Handle out-of-bounds indices in TakeRows

ad14ea5

lidavidm force-pushed the 11797-Provide-Scanner-methods-t branch from 65f72a2 to ad14ea5 Compare April 14, 2021 22:00

lidavidm closed this in d575858 Apr 15, 2021

This was referenced Apr 15, 2021

[C++][Dataset] Provide Scanner methods to yield/visit scanned batches #27648

Closed

[R] Fix guard around dataset___Scanner__TakeRows #28193

Closed

ARROW-11797: [C++][Dataset] Provide batch stream Scanner methods #9589

ARROW-11797: [C++][Dataset] Provide batch stream Scanner methods #9589

Uh oh!

Conversation

bkietz commented Feb 26, 2021

Uh oh!

github-actions bot commented Feb 26, 2021

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Apr 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Apr 9, 2021

Uh oh!

lidavidm commented Apr 12, 2021

Uh oh!

lidavidm commented Apr 13, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Apr 7, 2021 •

edited

Loading