multi-arg transform support

[advancedxy]

Feature Request / Improvement

As discussed in #5626, it would be nice to have multi-arg transform supported in Iceberg, especially for bucket transform.

I wrote up a spec change proposal doc for this improvement: https://docs.google.com/document/d/1aDoZqRgvDOOUVAGhvKZbp5vFstjsAMY4EFCyjlxpaaw/edit?usp=sharing

Quoted background from the doc:

Iceberg uses a transform to produce partitioning value from a source value. Currently the supported transforms are: Years, Months, Days, Hours, Identity, Void, Truncate, Bucket. Since the current spec requires that each partitioning field consists of a source column id in the table’s schema, the above transforms only accept one argument as its input. However, it’s possible and quite common to use multiple arguments to produce a partitioning value, especially for the Bucket transform. Other transforms might require multiple arguments in the future. This document tries to add multi-arg transform support in Iceberg, especially for the bucket transform.

---

I also did a poc version of how multiple arg bucket would be supported in Spark: #8259. Some places are not modified yet, such as UpdatePartitionSpec, TableMetadata related.
I'd like to get feedbacks from the community before going too much further.

If we have reached the consensus that we should support multi-arg transform and the spec changes are stabilized after reviewing. I would update my code accordingly, and extend the Flink engine support.

Query engine

Spark

[advancedxy]

@RussellSpitzer @rdblue would you like to take a look at this? Thanks.

[RussellSpitzer]

My big thoughts on this is we should probably frame it as a full Spec Version update targeting version 3. Left some comments on the doc.

[advancedxy]

My big thoughts on this is we should probably frame it as a full Spec Version update targeting version 3. Left some comments on the doc.

Yeah, I do think we should frame it in the new version. Let me update doc accordingly.

[advancedxy]

@RussellSpitzer I have update the spec change proposal doc. Would you mind to take a look again?

And I believe we should raise more discussion before committing more code, do you think it's necessary to post it in the dev mail list/slack to get more people's attention?

[RussellSpitzer]

It definitely needs more eyes, it probably should get brought up at the next community sync as well Sent from my iPhoneOn Aug 13, 2023, at 10:16 PM, advancedxy ***@***.***> wrote: @RussellSpitzer I have update the spec change proposal doc. Would you mind to take a look again? And I believe we should raise more discussion before committing more code, do you think it's necessary to post it in the dev mail list/slack to get more people's attention? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[RussellSpitzer]

@szehon-ho This is the proposal we were talking about, if you could check it out I would appreciate it.

[szehon-ho]

Yea I left some initial comments yesterday, plan to take another pass. The idea is promising.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'