[Python] Deprecate the legacy ParquetDataset custom python-based implementation

[asfimport]

To be able to remove the custom python implementation (ARROW-15868), we first need to deprecate the various aspects.

This issue is meant as a parent issue to keep an overview of the different tasks.

Reporter: Joris Van den Bossche / @jorisvandenbossche

Subtasks:

• [Python] Start with deprecating ParquetDataset custom attributes
• [Python] Start to raise deprecation warnings when using use_legacy_dataset=True in parquet.py
• [Python] Start raising deprecation warnings for ParquetDataset keywords that won't be supported with the new API
• [Python] ParquetDataset deprecation: change Deprecation to FutureWarnings
• [Python] Deprecate the (common_)metadata(_path) attributes of ParquetDataset
• [Python] Change use_legacy_dataset default and deprecate no-longer supported keywords in parquet.write_to_dataset
• [Python] Support row_group_size/chunk_size keyword in pq.write_to_dataset with use_legacy_dataset=False
• [Python] Switch default and deprecate use_legacy_dataset=True in ParquetDataset

Related issues:

• [Python] Remove the legacy ParquetDataset custom python-based implementation #31303 (blocks)
• [Python] Long-term fate of pyarrow.parquet.ParquetDataset #25775 (is a child of)
• [Python] Remove the test usage of legacy dataset #20233 (is required by)

_{Note: This issue was originally created as ARROW-16119. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
We included many additional deprecations in pyarrow 8.0.0, but we still need to deprecate / switch the use_legacy_dataset keyword in ParquetDataset(..) constructor itself (we are first deprecating the parameters and properties that are enabled by the legacy dataset constructor).

[asfimport]

Alenka Frim / @AlenkaF:
Sorry for closing it, I misunderstood!

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
The ParquetDataset() constructor itself still defaults to use_legacy_dataset=True (although using specific attributes or keywords related to that will raise a warning). So a next step will be to actually deprecate passing that and switching the default, and then only afterwards we can remove the code (-> opened ARROW-16728).

[AlenkaF]

As the ParquetDataset already defaults to use_legacy_dataset=False

arrow/python/pyarrow/parquet/core.py

Lines 1807 to 1809 in f7947cc

	# otherwise the default is already False
	else:
	use_legacy_dataset = False

I think we can now close this issue @jorisvandenbossche

[jorisvandenbossche]

Indeed, and we also warn when specifically choosing the legacy one.