Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
825befe
Initial async-for-loop and when-all implementations provided by bkietz
westonpace Dec 18, 2020
e6a3e53
WIP commit to review
westonpace Jan 4, 2021
5317ca1
WIP moving away from oop to purely functional approach
westonpace Jan 5, 2021
00c8d63
Synchronous transform works now
westonpace Jan 5, 2021
66f5520
Checking in WIP for quick review
westonpace Jan 5, 2021
e1b096a
Implementation appears to work now, time to clean things up
westonpace Jan 6, 2021
d4f1818
Various cleanup
westonpace Jan 6, 2021
f4039bd
More cleanup
westonpace Jan 6, 2021
8e307c3
More cleanup
westonpace Jan 6, 2021
ffb1d0e
There was a signed/unsigned mismatch that snuck through somehow and w…
westonpace Jan 6, 2021
b1f3a90
Lint errors
westonpace Jan 6, 2021
1e28089
There was a potential issue where a task group could finish its tasks…
westonpace Jan 6, 2021
bcf9568
Removed dataset-csv-scan-example.cc which was a macro-ish micro-bench…
westonpace Jan 7, 2021
eaea1a8
Commented out table reader tests that need thread pool until I figure…
westonpace Jan 8, 2021
ee19f65
Got rid of MakeSharedCallable in favor of lambdas. Added loops insid…
westonpace Jan 8, 2021
9df3fcb
Fixed a comment
westonpace Jan 8, 2021
b894a9e
Passing AsyncContext into TableReader::Make. Also, improved the benc…
westonpace Jan 8, 2021
6e22116
Simplified some of the transform logic. Removed loops where an if wo…
westonpace Jan 11, 2021
eb83c90
Changed unit tests to use lambda capture instead of shared pointers f…
westonpace Jan 12, 2021
78a0c2c
simplify Loop with CheckForTermination
bkietz Jan 11, 2021
547903d
Exposing AsyncContext on python (was causing failing builds since las…
westonpace Jan 12, 2021
52ac2b5
Fixed a bug in future where a callback wasn't being held onto. Added…
westonpace Jan 14, 2021
fd876ab
Lint
westonpace Jan 14, 2021
8775fef
It appears my initial guess that it was safe to clear todo_ unlocked …
westonpace Jan 14, 2021
befb97f
After making a copy of the list to purge I forgot to purge the copy i…
westonpace Jan 14, 2021
bb88655
Moved CSV reader benchmark to run on the existing slow input stream
westonpace Jan 26, 2021
2ed5031
Removed reader benchmark as it doesn't really add value
westonpace Jan 26, 2021
00d15da
Implementing PR WIP
westonpace Jan 27, 2021
d9fcc3a
Addressing code review from Antoine
westonpace Jan 28, 2021
fbedde9
Changing VisitAsyncGenerator around to remove copies of the generator
westonpace Jan 28, 2021
582e149
Update cpp/src/arrow/util/future.h
westonpace Jan 28, 2021
0fa129b
Fixed a lint error and a PR comment I had missed
westonpace Jan 28, 2021
b931094
Added move per PR review
westonpace Jan 28, 2021
5153b4b
Removed the synchronous readahead queue as it was truly an asynchrono…
westonpace Jan 29, 2021
6599bc8
Race condition in the TransferGnerator test
westonpace Jan 29, 2021
3113cd1
Added more description of TransferGenerator
westonpace Jan 29, 2021
7a8508e
Had to disable the move-accessor for a future because there is no saf…
westonpace Jan 29, 2021
66a26ef
Cleaning up the last commit
westonpace Jan 29, 2021
ab6aa6b
... was causing asan to trigger illegal instruction
westonpace Jan 29, 2021
8ea775d
Quick experiment to see if this valgrind complaint has anything to do…
westonpace Jan 30, 2021
dd8d2f1
Revert "Quick experiment to see if this valgrind complaint has anythi…
westonpace Jan 30, 2021
e6e2607
Reproduced segmentation fault from python. Put in a fix.
westonpace Jan 31, 2021
f61c7e8
Added MoveResult to future and changed existing moves (there was only…
westonpace Feb 1, 2021
732ba30
Missed an <iostream> leftover from debugging
westonpace Feb 1, 2021
e160b69
Removed threaded table reader
westonpace Feb 4, 2021
abc7926
Addressing PR comments
westonpace Feb 9, 2021
2b16727
Rebase and move 'background' I/O onto the I/O thread pool
westonpace Feb 10, 2021
2078da8
Had to rollback moving to the io executor. Need to address ARROW-115…
westonpace Feb 10, 2021
90740c3
Fixing flaky test cases
westonpace Feb 10, 2021
7ccd3a7
One of the stress tests was taking too long for one of the build systems
westonpace Feb 10, 2021
b2989b4
Upping test tolerance to see if the failure is a deadlock or just a s…
westonpace Feb 11, 2021
fbe4dfb
Created IterationTraits<CSVBlock> so that util::optional<CSVBlock> is…
westonpace Feb 11, 2021
7d9fbba
lint
westonpace Feb 11, 2021
f096126
simplify Loop slightly
bkietz Feb 15, 2021
b1200bd
Update cpp/src/arrow/util/future.h
westonpace Feb 15, 2021
4f0e764
Update cpp/src/arrow/csv/reader.cc
westonpace Feb 15, 2021
c29c19e
Merge pull request #4 from bkietz/10183-async-csv
westonpace Feb 15, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions c_glib/arrow-glib/reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1592,6 +1592,7 @@ garrow_csv_reader_new(GArrowInputStream *input,

auto arrow_reader =
arrow::csv::TableReader::Make(arrow::default_memory_pool(),
arrow::io::AsyncContext(),
arrow_input,
read_options,
parse_options,
Expand Down
1 change: 1 addition & 0 deletions cpp/examples/minimal_build/example.cc
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ Status RunMain(int argc, char** argv) {
ARROW_ASSIGN_OR_RAISE(
auto csv_reader,
arrow::csv::TableReader::Make(arrow::default_memory_pool(),
arrow::io::AsyncContext(),
input_file,
arrow::csv::ReadOptions::Defaults(),
arrow::csv::ParseOptions::Defaults(),
Expand Down
1 change: 0 additions & 1 deletion cpp/src/arrow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,6 @@ set(ARROW_SRCS
util/future.cc
util/int_util.cc
util/io_util.cc
util/iterator.cc
util/logging.cc
util/key_value_metadata.cc
util/memory.cc
Expand Down
3 changes: 2 additions & 1 deletion cpp/src/arrow/csv/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@ add_arrow_test(csv-test
column_builder_test.cc
column_decoder_test.cc
converter_test.cc
parser_test.cc)
parser_test.cc
reader_test.cc)

add_arrow_benchmark(converter_benchmark PREFIX "arrow-csv")
add_arrow_benchmark(parser_benchmark PREFIX "arrow-csv")
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/csv/column_decoder.cc
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ class ConcreteColumnDecoder : public ColumnDecoder {
auto chunk_index = next_chunk_++;
WaitForChunkUnlocked(chunk_index);
// Move Future to avoid keeping chunk alive
return std::move(chunks_[chunk_index]).result();
return chunks_[chunk_index].MoveResult();
}

protected:
Expand Down
Loading