[C++][Dataset] Revisit File discovery failure mode

[asfimport]

Currently, the default FileSystemFactoryOptions::exclude_invalid_files will silently ignore unsupported files (either IO error, not of the valid format, corruption, missing compression codecs, etc...) when creating a FileSystemSource.

We should change this behavior to propagate an error in the Inspect/Finish calls by default and allow the user to toggle exclude_invalid_files. The error should contain at least the file path and a decipherable error (if possible).

Reporter: Francois Saint-Jacques / @fsaintjacques
Assignee: Francois Saint-Jacques / @fsaintjacques

Related issues:

• [Python][Dataset] Non-existent files are silently dropped in pa.dataset.FileSystemDataset (is related to)
• [C++][Python][Dataset] Provide an option to toggle validation and schema inference in FileSystemDatasetFactoryOptions (is related to)

_{Note: This issue was originally created as ARROW-7673. Please see the migration documentation for further details.}

[asfimport]

Krisztian Szucs / @kszucs:
@fsaintjacques So in 0.16 we plan to provide best effort readers, and aiming this ticket to 1.0.0, right?

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
This has been refactored/fixed in ARROW-8058:

In [40]: da.dataset("/home/fsaintjacques/datasets/nyc-tlc/csv/2016", format="csv")                                                           
Out[40]: <pyarrow._dataset.FileSystemDataset at 0x7fef446b2930>

In [41]: da.dataset("/home/fsaintjacques/datasets/nyc-tlc/csv/2016", format="parquet")                                                       
...
OSError: Could not open parquet input source '/home/fsaintjacques/datasets/nyc-tlc/csv/2016/01/data.csv': Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

In [42]: da.dataset("/home/fsaintjacques/datasets/nyc-tlc/parquet/2016", format="parquet")                                                   
Out[42]: <pyarrow._dataset.FileSystemDataset at 0x7fef447ad7f0>