API: Fix Transform backward compatibility in PartitionSpec

[deniskuzZ]

BaseUpdatePartitionSpec::addField is failing with “Cannot add duplicate partition field” for the re-added DATE type field.

existing.transform().equals(sourceTransform.second()) returns FALSE since the types are different:

existing.transform() = {Dates@11599} “year”
sourceTransform.second() = {Years@11603} “year”

Test

Schema SCHEMA = new Schema(
          required(1, "id", Types.IntegerType.get()),
          required(2, "data", Types.StringType.get()),
          optional(3, "year_field", Types.DateType.get()));

PartitionSpec SPEC = PartitionSpec.builderFor(SCHEMA).year("year_field").build();
table = create(SCHEMA, SPEC);

table.updateSpec()
    .removeField(Expressions.year("year_field"))
    .addField(Expressions.year("year_field"));

[amogh-jahagirdar]

@deniskuzZ Thanks for the contribution! when you get a chance could you update the PR description about what the incompatibility issue (or if there's a github issue a link to that) is and why this change is required?

[deniskuzZ]

@amogh-jahagirdar, thanks for checking! updated PR description and added unit test

[zhangbutao]

I also had a workaround for this issue, see #6482. Not sure which way is more reasonable.

[deniskuzZ]

I also had a workaround for this issue, see #6482. Not sure which way is more reasonable.

I am ok with either solution, however, noticed a potentially no-go comment in your PR and decided to take the same approach as @Fokko in #6220.
Note this is a blocker for Hive Iceberg 1.1.0 migration.

[zhangbutao]

I am ok with either solution, however, noticed a potentially no-go comment in your PR and decided to take the same approach as @Fokko in #6220. Note this is a blocker for Hive Iceberg 1.1.0 migration.

Understood. I am ok with either solution too. Hope to fix this as soon as possible to unblock the Hive-iceberg upgrading.

[deniskuzZ]

@Fokko could you please take a look

[deniskuzZ]

hi @pvary, could you please help with the review?

[Fokko]

Sorry for the late reply, I was traveling the last few days. I like this solution. It is until Iceberg 2.0.0 that we have to keep the lazy and non-lazy versions of initializing a PartitionSpec using a transform.

[deniskuzZ]

Sorry for the late reply, I was traveling the last few days. I like this solution. It is until Iceberg 2.0.0 that we have to keep the lazy and non-lazy versions of initializing a PartitionSpec using a transform.

Thank you!