Fix(athena): Correctly handle PartitionByProperty when it contains Iceberg transforms#4962
Fix(athena): Correctly handle PartitionByProperty when it contains Iceberg transforms#4962
Conversation
…eberg transforms and is used in both CREATE TABLE and CTAS
| # Hive ref: https://docs.aws.amazon.com/athena/latest/ug/querying-iceberg-creating-tables.html#querying-iceberg-partitioning | ||
| # Trino ref: https://docs.aws.amazon.com/athena/latest/ug/create-table-as.html#ctas-table-properties | ||
|
|
||
| def _reorder(e: exp.Expression) -> exp.Expression: |
There was a problem hiding this comment.
i don't think this is right, you just want to alter the generator of the property, not do this complex transform
There was a problem hiding this comment.
This is just reordering the arguments before it hits the func generator.
I didnt think this logic belonged in the generator for exp.Func because it's very specific:
- It only applies to
exp.Funcinstances that are nested inside anexp.Schemathat is itself nested insideexp.PartitionByProperty - It only applies to Athena
- It only applies to two very specific functions that are used in Iceberg partition transforms,
bucketandtruncate
There was a problem hiding this comment.
This looks good to me, although I would clean up a couple of things. If @tobymao's good with getting this in I can do a quick refactor afterwards and share it with you folks.
|
Closing in favour of a different approach that factors Iceberg partition transforms into their own AST nodes so they can be handled as first-class AST nodes instead of anonymous functions. |
This addresses part of SQLMesh issue: TobikoData/sqlmesh#4090
Given the following:
Prior to this PR, when generating against Athena's Trino engine, the value would be truncated:
The same value works correctly in Hive mode:
However, simply fixing the truncation is not enough. The arguments are swapped when running in Hive mode vs Trino Mode. The argument order is documented in the AWS docs here for Hive and here for Trino.
This PR does the following:
bucketandtruncateIceberg transforms to be correct for whatever Athena generator is being used (Hive or Trino)