ARROW-16000: [C++][Python] Dataset: Alternative implementation for adding transcoding function option to CSV scanner#13820

Merged

lidavidm merged 31 commits intoapache:masterfrom

joosthooz:ARROW-16000-duplicate-python-readoptions

Sep 6, 2022

Commits on Aug 26, 2022

Added field to CsvFragmentScanOptions that holds an optional transform function
joosthooz
committed
WIP wrapping a trancoder around all input streams of a dataset
joosthooz
committed
Added input stream wrapping to CsvFileFormat::CountRows too
joosthooz
committed
Moved make_streamwrap_func into io.pxi, removed duplicated code
joosthooz
committed
Use UpperCamelCase in function name
joosthooz
committed
Additional name change, formatting fix
joosthooz
committed
Use GetReadOptions(), because it overrides the use_threads field
joosthooz
committed
Only add a transcoder for csv files
joosthooz
committed
Processed some review comments regarding documentation
joosthooz
committed
Moved encoding parameter from dataset() into CsvFileFormat
joosthooz
committed
Removed a left-over occurrence of the workaround encoding parameter
joosthooz
committed
Setting default encoding to utf8
joosthooz
committed
Always creating a default_fragment_scan_options
joosthooz
committed
Using a different way of checking the file format
joosthooz
committed
Added documentation about added encoding parameter
joosthooz
committed
Implementaed an alternative way of passing the encoding.
joosthooz
committed
Added python-specific encoding field to CsvFileformat.
joosthooz
committed
Removed encoding from CsvFragmentScanOptions
joosthooz
committed
Instead a python encoding field, now passing around a Python ReadOptions object
joosthooz
committed
Removed redundant parentheses
joosthooz
committed
Added ARROW_PYTHON_EXPORT to new function
joosthooz
committed
Always creating default_fragment_scan_options is not needed anymore, now we're setting the transcoder in the read_options setter
joosthooz
committed
Formatting
joosthooz
committed
Added python tests for transcoding dataset.
joosthooz
committed
Removed failing UTF16 test
joosthooz
committed
Revert "Removed failing UTF16 test"
joosthooz
committed
Added a test for a csv file with non-utf8 characters in the column names
joosthooz
committed
Removed subchecks in encoding tests.
joosthooz
committed
Renamed read_options_py to _read_options_py
joosthooz
committed
Removed unused test data
joosthooz
committed
Handling codec aliases by using codecs.lookup
joosthooz
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ARROW-16000: [C++][Python] Dataset: Alternative implementation for adding transcoding function option to CSV scanner#13820

ARROW-16000: [C++][Python] Dataset: Alternative implementation for adding transcoding function option to CSV scanner#13820
lidavidm merged 31 commits intoapache:masterfrom
joosthooz:ARROW-16000-duplicate-python-readoptions

Commits on Aug 26, 2022

Added field to CsvFragmentScanOptions that holds an optional transform function

WIP wrapping a trancoder around all input streams of a dataset

Added input stream wrapping to CsvFileFormat::CountRows too

Moved make_streamwrap_func into io.pxi, removed duplicated code

Use UpperCamelCase in function name

Additional name change, formatting fix

Use GetReadOptions(), because it overrides the use_threads field

Only add a transcoder for csv files

Processed some review comments regarding documentation

Moved encoding parameter from dataset() into CsvFileFormat

Removed a left-over occurrence of the workaround encoding parameter

Setting default encoding to utf8

Always creating a default_fragment_scan_options

Using a different way of checking the file format

Added documentation about added encoding parameter

Implementaed an alternative way of passing the encoding.

Added python-specific encoding field to CsvFileformat.

Removed encoding from CsvFragmentScanOptions

Instead a python encoding field, now passing around a Python ReadOptions object

Removed redundant parentheses

Added ARROW_PYTHON_EXPORT to new function

Always creating default_fragment_scan_options is not needed anymore, now we're setting the transcoder in the read_options setter

Formatting

Added python tests for transcoding dataset.

Removed failing UTF16 test

Revert "Removed failing UTF16 test"

Added a test for a csv file with non-utf8 characters in the column names

Removed subchecks in encoding tests.

Renamed read_options_py to _read_options_py

Removed unused test data

Handling codec aliases by using codecs.lookup