[Python][Dataset] Non-existent files are silently dropped in pa.dataset.FileSystemDataset

[asfimport]

When passing a list of files to the constructor of pyarrow.dataset.FileSystemDataset, all files that don't exist are silently dropped immediately (i.e. no fragments are created for them).

Instead, I would expect that fragments will be created for them but an error is thrown when one tries to read the fragment with the non-existent file.

Reporter: Uwe Korn / @xhochy
Assignee: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [C++][Dataset] Revisit File discovery failure mode (relates to)

PRs and other links:

• GitHub Pull Request #7468

_{Note: This issue was originally created as ARROW-8283. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@nealrichardson I am not fully sure it is a duplicate. At least from the higher level picture, the other talks about discovery (in which case you shouldn't end up with non-existing paths) while this issue about when manually creating a dataset of a list of paths. The fix might still be related though

[asfimport]

Neal Richardson / @nealrichardson:
Oh, alright, I'll reopen, and we can close together if it is the same fix.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
@jorisvandenbossche This filtering is done on the python side. This is due to the fact that the FileSystemDataset constructor accepts a path_or_selector. The C++ variant only accepts a vector of fragments, maybe we should align both?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Ah, indeed. We maybe could still accept a list of paths, if we then only turn them into a FileFragment without checking. In any case we should probably avoid the filesystem interaction with filesystem.get_file_info(..).

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Correct, we should not touch get_file_info when we have a list of paths.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Issue resolved by pull request 7468
#7468