ValueError: Keyword 'validate_schema' is not yet supported with the new Dataset API

[asfimport]

Documentation for PyArrow 6.x and 7.x both indicate that the validate_schema argument is supported in the ParquetDataset class. Yet passing that argument to an instance results in:

ValueError: Keyword 'validate_schema' is not yet supported with the new Dataset API

Code:

parquet_dataset = pyarrow.parquet.ParquetDataset(
    path_or_paths=paths,
    validate_schema=validate_schema,
    filesystem=filesystem,
    partitioning=partitioning,
    use_legacy_dataset=False,
)

Docs link:

https://arrow.apache.org/docs/6.0/python/generated/pyarrow.parquet.ParquetDataset.html

https://arrow.apache.org/docs/7.0/python/generated/pyarrow.parquet.ParquetDataset.html 

Reporter: Abderrahmane Jaidi

Related issues:

• [Python][Dataset] Expose schema inference / validation options in the factory (duplicates)

_{Note: This issue was originally created as ARROW-17308. Please see the migration documentation for further details.}

[asfimport]

Alenka Frim / @AlenkaF:
Yes, the new Dataset API doesn't yet support the validate_schema keyword, but the work is in progress and is tracked with ARROW-8221 (as it is a duplicated Jira will be closing this issue).

The default in ParquetDataset is still the legacy API and so the keyword validate_schema should work (without supplying the new FileSystem, see:

arrow/python/pyarrow/parquet/core.py

Lines 1717 to 1718 in 9621210

	if isinstance(filesystem, FileSystem):
	use_legacy_dataset = False

, which is relevant for 9.x and I think it was the same in 6.x, could be wrong though)

[asfimport]

Alenka Frim / @AlenkaF:
Maybe it is worth mentioning that even if the validate_schema argument is not supplied, the Arrow C++ implementation validates the schema for all the fragments of the dataset and returns a validation error in case of mismatch.