[Python] Start with deprecating ParquetDataset custom attributes

[asfimport]

As a first step for ARROW-9720, we should start with deprecating attributes/methods of pq.ParquetDataset that we would definitely not keep / are conflicting with the "dataset API".

I am thinking of the pieces attribute (and the ParquetDatasetPiece class), the partitions attribute (and the ParquetPartitions class).

In addition, some of the keywords are also exposed as properties (memory_map, read_dictionary, buffer_size, fs), and could be deprecated.

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [Python] Mention alternatives in deprecation message of ParquetDataset attributes (relates to)

PRs and other links:

• GitHub Pull Request #10549

_{Note: This issue was originally created as ARROW-13074. Please see the migration documentation for further details.}

[asfimport]

Lance Dacey / @ldacey:
 Any idea if this includes the partition_filename_cb function? I am still using that pretty extensively to write my "final" datasets that Power BI connects to for visualization since it allows me to overwrite each partition.

[asfimport]

Weston Pace / @westonpace:
@ldacey Does "basename_template" provide the functionality you seek?  It's currently documented as "A template string used to generate basenames of written data files. The token '{i}' will be replaced with an automatically incremented integer. If not specified, it defaults to "part-{i}." + format.default_extname"  I seem to recall there was some earlier discussion of this on a related Jira but I don't remember which one.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@ldacey the current PR is only about the ParquetDataset class, so for reading parquet, not for writing parquet

Let's keep discussion about partition_filename_cb for ARROW-12365 (the issue @ldacey created specifically for that topic; there was also discussion at ARROW-10695 on this topic)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@ldacey can you post the comment (that you deleted again) about the issues you ran into with basename_template in ARROW-12365 ?

[asfimport]

Lance Dacey / @ldacey:
Sure Joris, I posted it and then I read that you said to keep the discussion separate so I tried to be sneaky and delete it before you noticed

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 10549
#10549

[asfimport]

Richard Shadrach:
Is my understanding correct that ParquetDataset.partitions has been deprecated but with no current alternative (as mentioned in ARROW-9720)?

 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
There is actually some replacement (but not 1:1) if you use the Datasets API, see ARROW-8655 / #10661 (comment). That added a partitioning attribute with similar information.
Can you check if that gives sufficient information for your use case?

Now, we should certainly better document this / mention it in the deprecation message (I worked on both changes simultaneously, but should afterwards have updated the deprecation message that an alternative exists). We should maybe also expose the partitioning attribute on ParquetDataset (if constructed with use_legacy_dataset=False), as I did for the filesystem/files/fragments attributes.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I opened ARROW-13525 to track this follow-up.

[asfimport]

Richard Shadrach:

 Can you check if that gives sufficient information for your use case?

Thanks @jorisvandenbossche  - it does. Warnings showed up on pandas CI , here is my proposed fix: pandas-dev/pandas#42873.