[C++][Dataset] Support Latin-1 encoding

[asfimport]

In ARROW-15992 a user is reporting issues with trying to read in files with Latin-1 encoding.  I had a look through the docs for the Dataset API and I don't think this is currently supported.

Reporter: Nicola Crane / @thisisnic
Assignee: Joost Hoozemans / @joosthooz

Related issues:

• [R] Supper encoding options for CSVs in open_dataset (blocks)

PRs and other links:

• GitHub Pull Request #13709
• GitHub Pull Request #13820

_{Note: This issue was originally created as ARROW-16000. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
AFAIK, support for encodings isn't done in C++ at all (the C++ CSV reader has no such option, nor does Datasets) - it's done in the bindings by providing a file object wrapper that converts the encoding on the fly (e.g.

arrow/r/src/io.cpp

Lines 368 to 374 in 778b177

	std::shared_ptr<arrow::io::InputStream> MakeReencodeInputStream(
	const std::shared_ptr<arrow::io::InputStream>& wrapped, std::string from) {
	arrow::io::TransformInputStream::TransformFunc transform(
	ReencodeUTF8TransformFunctionWrapper{from});
	return std::make_shared<arrow::io::TransformInputStream>(std::move(wrapped),
	std::move(transform));
	}

). This works for reading individual files, but probably doesn't work so well for datasets…I wonder if what we need is a filesystem wrapper that'll decode different encodings.

Or I suppose we could just hardcode this into the CSV Datasets support.

[asfimport]

Nicola Crane / @thisisnic:
@lidavidm OK, thanks for pointing this out; I've dug deeper into this and it looks like there's a function, {}MakeReencodeInputStream{}, in the R package which handles this for the single file CSV reading.  Perhaps it might be possible to reuse that function for Datasets.

[asfimport]

Joost Hoozemans / @joosthooz:
Hi,

I've come across a similar issue when using pyarrow.dataset on non-utf8 data. Using read_csv works, because a transcoding is performed on the fly using python codecs. But when using multi-file datasets, doing it that way would mean everything would have to go through a single python interpreter. And then it will still not work for other language bindings. Can we add transcoding on the C++ side?

[asfimport]

David Li / @lidavidm:
@westonpace @pitrou what do you think about (optionally?) depending on something like icu to support transcoding? It's a rather heavyweight dependency for just CSV, I suppose, but bindings at the Python/R layer mean we're subject to interop overhead and the interpreter lock.

[asfimport]

Antoine Pitrou / @pitrou:
I would want to know first if there's actual contention due to the Python GIL and/or interpreter overhead.

In C++ the basic building block is TransformInputStream: https://arrow.apache.org/docs/cpp/api/io.html#transforming-input-wrapper , https://github.com/apache/arrow/blob/master/cpp/src/arrow/io/transform.h (sadly, it seems this lacks docstrings; my bad). It should be easy for a normally skilled C++ developer to use it to wrap their transcoding library of choice (some might want to use ICU, others libiconv, etc.).

I think it would be ideal if we offered an optional header-only that would wrap ICU in a TransformInputStream, without actually requiring ICU to be present when compiling Arrow. Perhaps this can be through templates?

Also datasets needs to grow a dedicated configuration option to wrap all input streams, perhaps.

[asfimport]

David Li / @lidavidm:
Sounds reasonable. Maybe we should implement the option in Python/R first, and based on that, see if we need to push it down further. I wonder if the option should be at the filesystems layer (probably not?), dataset layer (probably, maybe as part of CsvFileFormat or CsvFragmentReadOptions), or CSV options layer (possibly) though.

[asfimport]

David Li / @lidavidm:
Ah, but we're going to need some sort of C++ side hook to even make this possible.

[asfimport]

Antoine Pitrou / @pitrou:
CsvFragmentReadOptions sounds like a reasonable place to me.

[asfimport]

Weston Pace / @westonpace:
I agree with Antoine's suggestion of CsvFragmentReadOptions. This is essentially the same problem we have with compression too right? We can auto-detect compression based on the file extension but if the compression doesn't match the file extension (or the file extension doesn't indicate compression) we have no way of wrapping the stream with a decompression transform. It sounds like this solution might solve both problems.

[asfimport]

David Li / @lidavidm:
The comparison to compression is a good one. I agree it seems reasonable to put it there too.

[asfimport]

Joost Hoozemans / @joosthooz:
Thanks everyone for the advice. What makes CsvFragmentScanOptions the preferred place over csv.ReadOptions? CsvFragmentScanOptions right now doesn't directly store any properties itself, it only carries a csv.ConvertOptions and csv.ReadOptions. And compression and encoding sound like properties of a whole file, not a fragment (although I don't know if that is what Fragment means here).

Would it make sense as first attempt for me to add a TransformInputStream to CsvFragmentScanOptions or ReadOptions? Because then I can create 1 in python in the same way read_csv does it (with MakeTransformInputStream, with a callback into en/decode functions in a python library). Then we can see if there is a performance problem. Then later we could add functionality that creates a TransformInputStream in c++ world with a callback to some external library as Antoine suggested

[asfimport]

Antoine Pitrou / @pitrou:
When using the CSV reader directly, no additional option is needed since you can pass whichever InputStream you want.
It's only with datasets that the user doesn't create the InputStream themselves, so some option must be made available to customize the stream wrapping strategy.

[asfimport]

Antoine Pitrou / @pitrou:
So to be clear, I'm suggesting something like this:
https://gist.github.com/pitrou/2c99e971c01d324fdf72b632ffacae7d

[asfimport]

Weston Pace / @westonpace:
For more context. A fragment is a term introduced by the datasets API. The goal of the datasets API is to read data in from a collection on independently scannable fragments (in practice, fragment usually equals file).

The scanning process has its own set of options, ScanOptions, (e.g. use_threads, batch size, projection, etc.) which are independent of the file format. Each file reader has its own set of options (e.g. ReadOptions) and it's completely unaware of any dataset scanner.

Now, pretend you want to scan a collection of CSV files with a custom delimiter (e.g. |). It doesn't make sense for delimiter to be a property of scan options because it is specific to CSV.

As a result, we have ScanOptions::fragment_scan_options. This is an interface that each format provides an implementation for, which can be provided for the scan.

So, to read a CSV file with a custom delimiter, you just create ParseOptions with the correct delimiter. To read a dataset of CSV files with a custom delimiter you first create scan options for the scan itself, and then a ParseOptions with the custom delimiter, and then link them via ScanOptions::fragment_scan_options.

[asfimport]

Weston Pace / @westonpace:
Also, an arbitrary "input stream transformation callback" could also be a property of ScanOptions now that I think about it. This might be more useful once we support scanning JSON files. This could be used if the JSON files are compressed to add a decompressor to the stream.