[Python] Switch default and deprecate use_legacy_dataset=True in ParquetDataset

[asfimport]

The ParquetDataset() constructor itself still defaults to use_legacy_dataset=True (although using specific attributes or keywords related to that will raise a warning). So a next step will be to actually deprecate passing that and switching the default, and then only afterwards we can remove the code.

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Joris Van den Bossche / @jorisvandenbossche

PRs and other links:

• GitHub Pull Request #14052
• GitHub Pull Request #15269

_{Note: This issue was originally created as ARROW-16728. Please see the migration documentation for further details.}

[asfimport]

Raúl Cumplido / @raulcd:
@jorisvandenbossche was your idea for this one to be done on two different releases?
First - DeprecationWarning and switch default to use_legacy_dataset=False

Second - Remove possibility of using use_legacy_dataset=True

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Yes, indeed (but so the two first items together in one release (switching default + deprecating the option, and then only in a later release actually remove the option).

We should certainly do that first step in 10.0

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Issue resolved by pull request 14052
#14052

[0x26res]

Now that this change is effective in 11.0, we get this warning when loading data with use_legacy_dataset=True.

FutureWarning: Passing 'use_legacy_dataset=True' to get the legacy behaviour is deprecated as of pyarrow 11.0.0, and the legacy implementation will be removed in a future version.

I'm in the process of migrating to use_legacy_dataset=False, but was wondering what differences to expect between the 2 implementations. Is this documented somewhere?

I have noticed one significant difference in behaviour. The legacy implementation would complain if the parquet schema are heterogeneous. The new implementation will try to convert all files to the schema of the first file it found (or the schema argument when provided).

Are there other differences to expect?