RFC: Implement initial support for COPY ... TO ... statement

[alamb]

Which issue does this PR close?

Closes #5654
Closes #5988

Rationale for this change

1. I would like to to make it easier to test out datafusion (both for new comers as well as when I am trying to benchmark all the things)

What changes are included in this PR?

(I'll try and break this up into smaller pieces for easier review but I want to show it all working together)

• Initial parser support for COPY .. TO ... statements
• New LogicalPlan::CopyTo variant
• Planing / Runtime support for copying to Parquet files
  -[ ] Parser tests
  -[ ] Add end user documentation
  -[ ] Properly support writing single parquet files
  -[ ] sqllogictests

I also plan to file follow on tasks (like support for other file formats, options, etc)

Are these changes tested?

Yes

Are there any user-facing changes?

Yes there is a new (documented) statement

[alamb]

Note that this is the mechanism as used by CREATE TABLE AS SELECT (aka LogicalPlan::CreateMemTable). It is different than the mechanism used by INSERT INTO ... ` (added in #5520 by @metesynnada ) that uses an ExecutionPlan.

The difference bothers me, but I can see the benefits of both approaches

[alamb]

I was playing with this more this evening and I think I came up with something that is half way between that I like even better. Will keep iterating and report back

[ozankabak]

I think we have something you may be able to leverage, check this PR out. It extends the ExecutionPlan approach to writing files, I think you can leverage that work here too. With that change, COPY TO and INSERT INTO will use the same ExecutionPlan-based approach -- the only difference would be related to appending vs overwriting.

FYI, if you are curious about timing, we plan to finalize and submit to upstream in a week or so.

[alamb]

Thanks @ozankabak -- I have reviewed https://github.com/synnada-ai/arrow-datafusion/pull/89 and I have thought about how to incorporate the same structure

I really like your idea to use the the same plans for COPY TO and INSERT INTO. After some more thought, I have an idea of how to plan COPY statements using the same plans as an INSERT.

Here is a proposal for a simplified API: #6339

I'll try and bang out a PR shortly

[alamb]

Thank you for the heads up - I will study the PR you mention.

…

On Wed, May 10, 2023 at 5:02 PM Mehmet Ozan Kabak ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In datafusion/core/src/execution/context.rs <#6313 (comment)> : > @@ -450,6 +457,36 @@ impl SessionContext { self.read_batch(record_batch) } + // Execute a COPY TO statement, returning the number of rows I think we have something you may be able to leverage, check this PR <https://github.com/synnada-ai/arrow-datafusion/pull/89> out. It extends the ExecutionPlan approach to writing files, I think you can leverage that work here too. — Reply to this email directly, view it on GitHub <#6313 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADXZMJBIT634DSGELJU3XLXFQF7FANCNFSM6AAAAAAX37ZGKA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alamb]

i have enough feedback for now and I working to add this functionality in pieces, so no need for this PR now