KAFKA-6290: Support casting from logical types in cast transform#7371
KAFKA-6290: Support casting from logical types in cast transform#7371rhauch merged 1 commit intoapache:trunkfrom
Conversation
rhauch
left a comment
rhauch
left a comment
There was a problem hiding this comment.
Looks great. A few more suggestions below.
rhauch
left a comment
rhauch
left a comment
There was a problem hiding this comment.
Thanks for the adjustments, @ncliang. A few more suggestions below, but otherwise looks good.
Finally, we need some indication of in which versions this should be fixed, since that tells us to which branches this PR, once merged, should be cherry-picked. Since the bug was first reported against 0.11.0.0, Ideally we should probably go back as far as we can, though it's not immediately apparent how far this PR will be easily cherry-picked.
There was a problem hiding this comment.
I think it'd be useful to verify the behavior of casting all of the logical types to strings, just so that we verify the formats (e.g., timestamps, times, and dates should use the ISO 8601 representation) and have some regression tests for the future to help ensure we maintain backward compatibility.
There was a problem hiding this comment.
Done! I moved the castToString tests to their own testcase.
Looking back at release branches, it seems schema was not piped into castValueToType() until 2.1. Before that, we can not easily determine if we are casting from a logical type. So, I'd say easiest is to go back to 2.1. Otherwise, we will have to backport KAFKA-6684 as well.
There was a problem hiding this comment.
If that's the case, then we may need a separate PR to go back further than 2.1. I'd say we'd want to correct this issue (which has irked quite a few people) as far back as 1.0.
There was a problem hiding this comment.
#4820 cherry-picks pretty cleanly onto 1.0. Then, these changes apply cleanly. My vote would be to take both fixes for KAFKA-6684 and this fix as far back as 1.0. WDYT?
There was a problem hiding this comment.
Ping @rhauch . WDYT about taking both KAFKA-6684 and this PR as far back as 1.0?
This PR adds support for cast transforms to cast from logical types. Casting to numeric types will produce the underlying numeric value represented in the desired type. For logical types represented by underlying Java Date class, this means the milliseconds since EPOCH. For Decimal, this means the underlying value. If the value does not fit in the desired target type, it may overflow. Casting to String from Date, Time, and Timestamp types will produce their ISO 8601 representation. Casting to String from Decimal will result in the value represented as a string. e.g. 1234 -> "1234"
|
Retest this, please. |
Adds support for the Connect Cast transforms to cast from Connect logical types, such as DATE, TIME, TIMESTAMP, and DECIMAL. Casting to numeric types will produce the underlying numeric value represented in the desired type. For logical types represented by underlying Java Date class, this means the milliseconds since EPOCH. For Decimal, this means the underlying value. If the value does not fit in the desired target type, it may overflow. Casting to String from Date, Time, and Timestamp types will produce their ISO 8601 representation. Casting to String from Decimal will result in the value represented as a string. e.g. 1234 -> "1234". Author: Nigel Liang <nigel@nigelliang.com> Reviewer: Randall Hauch <rhauch@gmail.com>
Adds support for the Connect Cast transforms to cast from Connect logical types, such as DATE, TIME, TIMESTAMP, and DECIMAL. Casting to numeric types will produce the underlying numeric value represented in the desired type. For logical types represented by underlying Java Date class, this means the milliseconds since EPOCH. For Decimal, this means the underlying value. If the value does not fit in the desired target type, it may overflow. Casting to String from Date, Time, and Timestamp types will produce their ISO 8601 representation. Casting to String from Decimal will result in the value represented as a string. e.g. 1234 -> "1234". Author: Nigel Liang <nigel@nigelliang.com> Reviewer: Randall Hauch <rhauch@gmail.com>
Adds support for the Connect Cast transforms to cast from Connect logical types, such as DATE, TIME, TIMESTAMP, and DECIMAL. Casting to numeric types will produce the underlying numeric value represented in the desired type. For logical types represented by underlying Java Date class, this means the milliseconds since EPOCH. For Decimal, this means the underlying value. If the value does not fit in the desired target type, it may overflow. Casting to String from Date, Time, and Timestamp types will produce their ISO 8601 representation. Casting to String from Decimal will result in the value represented as a string. e.g. 1234 -> "1234". Author: Nigel Liang <nigel@nigelliang.com> Reviewer: Randall Hauch <rhauch@gmail.com>
Adds support for the Connect Cast transforms to cast from Connect logical types, such as DATE, TIME, TIMESTAMP, and DECIMAL. Casting to numeric types will produce the underlying numeric value represented in the desired type. For logical types represented by underlying Java Date class, this means the milliseconds since EPOCH. For Decimal, this means the underlying value. If the value does not fit in the desired target type, it may overflow. Casting to String from Date, Time, and Timestamp types will produce their ISO 8601 representation. Casting to String from Decimal will result in the value represented as a string. e.g. 1234 -> "1234". Author: Nigel Liang <nigel@nigelliang.com> Reviewer: Randall Hauch <rhauch@gmail.com>
Adds support for the Connect Cast transforms to cast from Connect logical types, such as DATE, TIME, TIMESTAMP, and DECIMAL. Casting to numeric types will produce the underlying numeric value represented in the desired type. For logical types represented by underlying Java Date class, this means the milliseconds since EPOCH. For Decimal, this means the underlying value. If the value does not fit in the desired target type, it may overflow. Casting to String from Date, Time, and Timestamp types will produce their ISO 8601 representation. Casting to String from Decimal will result in the value represented as a string. e.g. 1234 -> "1234". Author: Nigel Liang <nigel@nigelliang.com> Reviewer: Randall Hauch <rhauch@gmail.com>
|
Merged back to the |
3 participants
This PR adds support for cast transforms to cast from logical types Date, Time, and Timestamp to
their internal int32 or int64 representations. Any valid cast on their internal representations
would also be applicable. For instance, Time has internal int32 representation, but can be cast to
int64 if desired.
Committer Checklist (excluded from commit message)