fix(java): support FixedSizeList for java LanceField#5509
fix(java): support FixedSizeList for java LanceField#5509majin1102 merged 1 commit intolance-format:mainfrom
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
majin1102
left a comment
majin1102
left a comment
There was a problem hiding this comment.
Provide some context, please take a look
| private final int parentId; | ||
| private final String name; | ||
| private final boolean nullable; | ||
| private final String logicalType; |
There was a problem hiding this comment.
Just a little context
There was a problem hiding this comment.
For lance vector data type, only the logicalType has FixedSizeList's element type, such as fixed_size_list:float:3.
So, when LanceField is converted to arrow Field , the FixedSizeList's children should be constructed from logicalType.
Thanks for your comment. When Lance Schema(Rust) is converted to LanceSchema(Java) and then LanceSchema(java) is converted to Arrow's Schema using LanceSchema.asArrowSchema the FixedSizeList's children is missed. The original issue is lance-format/lance-spark#138 and related PR is #5471 |
fangbo
changed the title
fangbo
force-pushed
the
lance-schema
branch
4 times, most recently
from
6d80731 to
d033bad
Compare
|
@westonpace @jackye1995 This PR is ready. I look forward to your feedback. Thank you. |
fangbo
force-pushed
the
lance-schema
branch
5 times, most recently
from
0a061b3 to
a0e9fe8
Compare
fangbo
force-pushed
the
lance-schema
branch
4 times, most recently
from
bffa003 to
c22aa34
Compare
# Problem Description The java method [Fragment.mergeColumns](https://github.com/lance-format/lance/blob/main/java/src/main/java/org/lance/Fragment.java#L152) returns a [LanceSchema](https://github.com/lance-format/lance/blob/main/java/src/main/java/org/lance/schema/LanceSchema.java). The java LanceField is converted from a rust lance Field. If a field is a vector which arrow type is FixedSizeList with element's type is Float, the java LanceField missed the element's type when converted from rust. But the Merge transaction needs a arrow Schema. So, when java LanceSchema is converted to arrow Schema a [error occurs](lance-format/lance-spark#138). For vector field, the FixedSizeList's element type is only contained in logicalType. This PR use the logicalType to correctly build a arrow Field when converting java LanceSchema to arrow Schema. @westonpace @jackye1995 @majin1102 Could you give some suggestion about this modification? Thank you very much. Co-authored-by: fangbo.0511 <fangbo.0511@bytedance.com>
2 participants
Problem Description
The java method Fragment.mergeColumns returns a LanceSchema.
The java LanceField is converted from a rust lance Field. If a field is a vector which arrow type is FixedSizeList with element's type is Float, the java LanceField missed the element's type when converted from rust. But the Merge transaction needs a arrow Schema. So, when java LanceSchema is converted to arrow Schema a error occurs.
For vector field, the FixedSizeList's element type is only contained in logicalType. This PR use the logicalType to correctly build a arrow Field when converting java LanceSchema to arrow Schema.
@westonpace @jackye1995 @majin1102 Could you give some suggestion about this modification? Thank you very much.