ARROW-9731: [C++][Python][R][Dataset] WIP: Port "head" into C++ #9854
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
Depends on ARROW-7001/#9607 so not quite ready yet. @westonpace, I made commits here to deprecate uses of Scan() in Python/R which you might want to just cherry-pick into your own PR instead. |
lidavidm
force-pushed
the
arrow-9731
branch
2 times, most recently
from
65e2ba8 to
6bcd208
Compare
ARROW-7001: First stab at converting datasets logic to async ARROW-7001: Fixed a bunch of .result()'s in unit tests that weren't really valid (returning a reference to something then deleted) ARROW-7001: Missed on change during rebase ARROW-7001: Renamed ScanSync to Scan and ExecuteSync to Execute to preserve the old mirror APIs until the public bindings can be removed ARROW-7001: Added a few more mirror APIs to get python build working ARROW-7001: WIP ARROW-7001: Various WIP ARROW-7001: WIP ARROW-7001: First stab at converting datasets logic to async ARROW-7001: Fixed a bunch of .result()'s in unit tests that weren't really valid (returning a reference to something then deleted) ARROW-7001: Renamed ScanSync to Scan and ExecuteSync to Execute to preserve the old mirror APIs until the public bindings can be removed ARROW-7001: Added a few more mirror APIs to get python build working ARROW-7001: WIP ARROW-7001: Various WIP ARROW-7001: Minor fixes to get semantics right ARROW-7001: Cleanup ARROW-7001: Fixing some compile errors after rebase ARROW-7001: Fixing errors from rebase ARROW-7001: Added a test for reordering datasets. Removed old concept of splittable. Fixed bug where file errors may not pass through ARROW-7001: Somewhere in the rebasing I lost the 1-arg ScannerBuilder constructor. Added it back in and created a unit test for it for good measure. ARROW-7001: Removing a ... to see if it removes illegal instruction on mac ARROW-7001: Fixed a potential memory issue in the preserve ordering test ARROW-7001: lint ARROW-7001: Changed from using optional<bool> which isn't allowed to just returning the scan task in Scanner::ToTableAsync::table_building_task ARROW-7001: Removed the forced transfer as it was not truly doing anything ARROW-7001: The CSV scan task was doing a read on the CPU thread pool and it was preventing the async chain from getting setup immediately slowing things down. In addition, the later readahead buffers need to be larger to prevent the CPU thread from idling when things arrive out of order. ARROW-7001: Need to put the impl for Scanner::ToTable in the cc file so it ends up in the so ARROW-7001: Added a reordering test ARROW-7001: Added ordering to scanner ARROW-7001: Converted Future<Generator> to Generator ARROW-7001: File readahead was not working correctly and to fix it required quite an overhaul of the scanner but, on the bright side, performance is better on I/O bound tasks ARROW-7001: Fix failing unit test ARROW-7001: Cleaned up lint. Deprecated the old Scan method. Reworked existing logic to adapt ARROW-7001: Removing unused code detected by build ARROW-7001: Moved some code around between header/impl to make MSVC happy. Fixed up a memory leak in a unit test caused by a circular shared_ptr reference
…R code. To address in ARROW-11782 ARROW-7001: Removed incorrect comment from MakeMappedGenerator ARROW-7001: Fixed a regression present when reading IPC fully buffered in memory ARROW-7001: Made the InMemoryDataset creation methods consistent. ARROW-7001: Adding back in (hopefully legacy) constructor for InMemoryScanTask needed by cglib
Member
Author
|
I'll replace this once ARROW-11797/#9589 lands. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.