write_parquet() / write_csv_arrow() cannot stream a dataset object back to S3

[asfimport]

It seems we cannot send a dataset (table) object opened with open_dataset() into write_parquet() or write_csv_arrow().  I realize that the former is a 'multi-file' representation and the latter two are not, but as both represent a single table (single schema) it seems like this should still work.  (It also looks like write_csv_arrow() gives us more control over formatting details than write_dataset() )

Reporter: Carl Boettiger / @cboettig

Related issues:

• [R] [C++] Allow me to write_parquet() from an arrow_dplyr_query (duplicates)
• [R] Enable write_csv_arrow to take Dataset or arrow_dplyr_query as input (duplicates)

_{Note: This issue was originally created as ARROW-15104. Please see the migration documentation for further details.}

[asfimport]

Nicola Crane / @thisisnic:
Hi @cboettig , thanks for reporting this!  I totally agree that this functionality should be implemented - and we have a couple of tickets open which when completed will allow this, so I'm going to close this ticket as a duplicate (though always good to know that there is user interest in this getting prioritised)

[asfimport]

Carl Boettiger / @cboettig:
Thanks much! sorry for the duplicate and I'll keep an eye on those issues you linked.