[C++][Dataset][Python] Add a callback to visit file writers just before Finish()

[asfimport]

This will fill the role of (for example) metadata_collector or allow stats to be embedded in IPC file footer metadata.

Reporter: Ben Kietzman / @bkietz
Assignee: Ben Kietzman / @bkietz

Related issues:

• [Python] [Dataset] Add metadata_collector option to ds.write_dataset() (fixes)

PRs and other links:

• GitHub Pull Request #10573

_{Note: This issue was originally created as ARROW-10440. Please see the migration documentation for further details.}

[asfimport]

Ben Kietzman / @bkietz:
@jorisvandenbossche

[asfimport]

Lance Dacey / @ldacey:
Will this change allow us to get a list of the blob paths which were saved as file fragments?

I am currently using fs.glob() to find a list of files which were just saved using a specific basename_template as a work around.

    pattern = filesystem.sep.join([output_path, f"**{base_template}-*"])
    files = filesystem.glob(
        pattern,
        details=False,
        invalidate_cache=True,
    )

However, with the legacy write_to_dataset(), I am able to use the metadata_collector and then create a list of the file paths like this, which is more convenient (I do not have to worry about generating unique/predictable basename templates).

    files = []
    for piece in collector:
        files.append(filesystem.sep.join([output_path, piece.row_group(0).column(0).file_path]))

I require the lists of blobs to pass along to other Airflow tasks to either read as a dataset, or I generate a list of filters from the paths.

[asfimport]

Lance Dacey / @ldacey:
Can someone confirm if this issue would cover my use case or if I should add a separate feature request issue? My goal is to simply be able to retrieve the list of fragment paths which were saved using the ds.write_dataset() function.

I believe it does since I am using the metadata_collector argument to gather this information with the legacy dataset, but let me know if this is different. thanks!

[asfimport]

Weston Pace / @westonpace:
Yes, it should be able to satisfy your use case, now that you've mentioned the need here.  Right now a FileWriter only has the schema which wouldn't have the path being written but it seems reasonable that it could have a path accessor as well.    @bkietz can you confirm?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
( @ldacey since I asked you on the other issue about creating new issues: this one actually already covers the metadata_collector question, I think?)

I suppose a generic callback could indeed solve your use case. But on the other hand: getting the file names of the files that were just written also seems like a generally useful thing we could maybe expose in some other way without needing the generic callback?

@bkietz did you envision this also to be exposed in Python?

[asfimport]

Ben Kietzman / @bkietz:
Sorry for the late reply.

@ldacey @westonpace Yes, I think it's totally reasonable to provide const std::string& FileWriter::path(). The caveat is that currently FIleWriter is agnostic about what kind of stream it's writing into; for example it may be writing into a Buffer rather than to disk which would clearly not be associated with any path. The path property could be added to FileFormat::MakeWriter and supplied alongside the output stream whenever appropriate.

@jorisvandenbossche we could also expose this in Python, in which case the general callback could be used to provide simpler functionality when appropriate

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 10573
#10573

[asfimport]

Weston Pace / @westonpace:
Hmm, this doesn't quite fill the role of metadata_collector.  The metadata_collector had to be called "after" finish and not "before" finish.  Otherwise the parquet schema is not accessible in the arrow:: parquet::FileWriter.

 

Looking at the description the reason for calling in "before" finish was to allow inserting extra data into the IPC stream?  In that case I think we will need two callbacks.  Thoughts?