Skip to content

fix!: Add support for Athena's Iceberg partitioning transforms#4976

Merged
VaggelisD merged 3 commits intomainfrom
vaggelisd/iceberg_partition_properties
Apr 16, 2025
Merged

fix!: Add support for Athena's Iceberg partitioning transforms#4976
VaggelisD merged 3 commits intomainfrom
vaggelisd/iceberg_partition_properties

Conversation

@VaggelisD
Copy link
Collaborator

@VaggelisD VaggelisD commented Apr 15, 2025

Follow up on #4962

This PR introduces the expressions exp.PartitionedByBucket and exp.PartitionedByTruncate which control the generation of Iceberg transforms:

  • Athena's "Hive" engine (used for CREATE TABLE) demands BUCKET(n, col) / TRUNCATE(n, col)
  • Athena's "Trino" engine (used for CTAS) demands BUCKET(col, n) / TRUNCATE(col, n)

Note: The PARTITIONED BY property attempts to parse a schema, so the new expressions are parsed as "constraints" since _parse_schema() ends up parsing _parse_constraint() or _parse_column_def().

@VaggelisD VaggelisD force-pushed the vaggelisd/iceberg_partition_properties branch from b603827 to 6789fd3 Compare April 15, 2025 12:01
@VaggelisD VaggelisD force-pushed the vaggelisd/iceberg_partition_properties branch from 6789fd3 to 9dcd887 Compare April 15, 2025 12:05
@georgesittas georgesittas requested review from erindru and tobymao April 15, 2025 16:37
@georgesittas
Copy link
Collaborator

Let's verify that this fixes the linked SQLMesh issue.

@VaggelisD
Copy link
Collaborator Author

Let's verify that this fixes the linked SQLMesh issue.

If I'm not mistaken this is one part of the fix; The other part would be parsing the partition_by model expression into the new nodes, which will be generated into the proper Athena engine by using the appropriate generator.

@georgesittas
Copy link
Collaborator

I see, makes sense, thanks for clarifying.

Copy link
Collaborator

@erindru erindru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks correct, thanks!

@VaggelisD VaggelisD merged commit 2b7845a into main Apr 16, 2025
6 checks passed
@VaggelisD VaggelisD deleted the vaggelisd/iceberg_partition_properties branch April 16, 2025 07:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants