feat: refine record batch project#1014
Conversation
a9dc195 to
28dbba9
Compare
|
Potential improvement: For now, we will trave the whole record in visitor pattern, but in some case we don't need to travese child column. E.g. the map column is passthrough, we don't need type promotion for its child so we hope to return map column directly and stop traversing its child. |
fbcc13d to
8759513
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
This PR:
SchemaWithPartnervisitor to refactorRecordBatchProjectorRecordBatchProjectorinRecordBatchTransformContext
For read or write in iceberg-rust, we interoperate with arrow RecordBatch. There are similar project tasks when reading and write to project RecordBatch:
For now, these tasks are implemented in different structs,
RecordBatchTransform, andRecordBatchProjectorand there are redundant codes. After we introduce theSchemaWithPartnervisitor in #731, we can unify the implementation under the "visitor" framework.The project task is generally separated into two parts:
We can custom accessor to achieve init-default. E.g. when fail to retrieve the array, accessor can construct the array according to init-default. And we also can custom accessor to achieve name-map.
We can customize visitors to do the type of promotion here.
The PR try this work to refactor our
RecordBatchTransform, andRecordBatchProjector. And it also complete #405 (comment) using ArrowProjectionVisitor as @liurenjie1024 mention