-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[Bug](exchange) fix exchange of tablet shuffle send block error #44102
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug](exchange) fix exchange of tablet shuffle send block error #44102
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
HappenLee
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
…he#44102) Problem Summary: the _batching_block is same as block which is used for output. and maybe different with input block in some column about nullable type. so if sink this will cause nullable type not equal as origin. ``` Status VRowDistribution::generate_rows_distribution( vectorized::Block& input_block, std::shared_ptr<vectorized::Block>& block, int64_t& filtered_rows, bool& has_filtered_rows, std::vector<RowPartTabletIds>& row_part_tablet_ids, int64_t& rows_stat_val) ....... // batching block rows which need new partitions. deal together at finish. if (!_batching_block) [[unlikely]] { std::unique_ptr<Block> tmp_block = block->create_same_struct_block(0); _batching_block = MutableBlock::create_unique(std::move(*tmp_block)); } ``` ``` void OlapTableBlockConvertor::_convert_to_dest_desc_block(doris::vectorized::Block* block) { for (int i = 0; i < _output_tuple_desc->slots().size() && i < block->columns(); ++i) { SlotDescriptor* desc = _output_tuple_desc->slots()[i]; if (desc->is_nullable() != block->get_by_position(i).type->is_nullable()) { if (desc->is_nullable()) { block->get_by_position(i).type = vectorized::make_nullable(block->get_by_position(i).type); block->get_by_position(i).column = vectorized::make_nullable(block->get_by_position(i).column); } else { block->get_by_position(i).type = assert_cast<const vectorized::DataTypeNullable&>( *block->get_by_position(i).type) .get_nested_type(); block->get_by_position(i).column = assert_cast<const vectorized::ColumnNullable&>( *block->get_by_position(i).column) .get_nested_column_ptr(); } } } } ```
…he#44102) Problem Summary: the _batching_block is same as block which is used for output. and maybe different with input block in some column about nullable type. so if sink this will cause nullable type not equal as origin. ``` Status VRowDistribution::generate_rows_distribution( vectorized::Block& input_block, std::shared_ptr<vectorized::Block>& block, int64_t& filtered_rows, bool& has_filtered_rows, std::vector<RowPartTabletIds>& row_part_tablet_ids, int64_t& rows_stat_val) ....... // batching block rows which need new partitions. deal together at finish. if (!_batching_block) [[unlikely]] { std::unique_ptr<Block> tmp_block = block->create_same_struct_block(0); _batching_block = MutableBlock::create_unique(std::move(*tmp_block)); } ``` ``` void OlapTableBlockConvertor::_convert_to_dest_desc_block(doris::vectorized::Block* block) { for (int i = 0; i < _output_tuple_desc->slots().size() && i < block->columns(); ++i) { SlotDescriptor* desc = _output_tuple_desc->slots()[i]; if (desc->is_nullable() != block->get_by_position(i).type->is_nullable()) { if (desc->is_nullable()) { block->get_by_position(i).type = vectorized::make_nullable(block->get_by_position(i).type); block->get_by_position(i).column = vectorized::make_nullable(block->get_by_position(i).column); } else { block->get_by_position(i).type = assert_cast<const vectorized::DataTypeNullable&>( *block->get_by_position(i).type) .get_nested_type(); block->get_by_position(i).column = assert_cast<const vectorized::ColumnNullable&>( *block->get_by_position(i).column) .get_nested_column_ptr(); } } } } ```
…he#44102) Problem Summary: the _batching_block is same as block which is used for output. and maybe different with input block in some column about nullable type. so if sink this will cause nullable type not equal as origin. ``` Status VRowDistribution::generate_rows_distribution( vectorized::Block& input_block, std::shared_ptr<vectorized::Block>& block, int64_t& filtered_rows, bool& has_filtered_rows, std::vector<RowPartTabletIds>& row_part_tablet_ids, int64_t& rows_stat_val) ....... // batching block rows which need new partitions. deal together at finish. if (!_batching_block) [[unlikely]] { std::unique_ptr<Block> tmp_block = block->create_same_struct_block(0); _batching_block = MutableBlock::create_unique(std::move(*tmp_block)); } ``` ``` void OlapTableBlockConvertor::_convert_to_dest_desc_block(doris::vectorized::Block* block) { for (int i = 0; i < _output_tuple_desc->slots().size() && i < block->columns(); ++i) { SlotDescriptor* desc = _output_tuple_desc->slots()[i]; if (desc->is_nullable() != block->get_by_position(i).type->is_nullable()) { if (desc->is_nullable()) { block->get_by_position(i).type = vectorized::make_nullable(block->get_by_position(i).type); block->get_by_position(i).column = vectorized::make_nullable(block->get_by_position(i).column); } else { block->get_by_position(i).type = assert_cast<const vectorized::DataTypeNullable&>( *block->get_by_position(i).type) .get_nested_type(); block->get_by_position(i).column = assert_cast<const vectorized::ColumnNullable&>( *block->get_by_position(i).column) .get_nested_column_ptr(); } } } } ```
…e#3656) cherry-pick from doris-master apache#44102
…he#44102) (apache#393) Problem Summary: the _batching_block is same as block which is used for output. and maybe different with input block in some column about nullable type. so if sink this will cause nullable type not equal as origin. ``` Status VRowDistribution::generate_rows_distribution( vectorized::Block& input_block, std::shared_ptr<vectorized::Block>& block, int64_t& filtered_rows, bool& has_filtered_rows, std::vector<RowPartTabletIds>& row_part_tablet_ids, int64_t& rows_stat_val) ....... // batching block rows which need new partitions. deal together at finish. if (!_batching_block) [[unlikely]] { std::unique_ptr<Block> tmp_block = block->create_same_struct_block(0); _batching_block = MutableBlock::create_unique(std::move(*tmp_block)); } ``` ``` void OlapTableBlockConvertor::_convert_to_dest_desc_block(doris::vectorized::Block* block) { for (int i = 0; i < _output_tuple_desc->slots().size() && i < block->columns(); ++i) { SlotDescriptor* desc = _output_tuple_desc->slots()[i]; if (desc->is_nullable() != block->get_by_position(i).type->is_nullable()) { if (desc->is_nullable()) { block->get_by_position(i).type = vectorized::make_nullable(block->get_by_position(i).type); block->get_by_position(i).column = vectorized::make_nullable(block->get_by_position(i).column); } else { block->get_by_position(i).type = assert_cast<const vectorized::DataTypeNullable&>( *block->get_by_position(i).type) .get_nested_type(); block->get_by_position(i).column = assert_cast<const vectorized::ColumnNullable&>( *block->get_by_position(i).column) .get_nested_column_ptr(); } } } } ```
What problem does this PR solve?
Problem Summary:
test case: #44215
the _batching_block is same as block which is used for output.
and maybe different with input block in some column about nullable type.
so if sink this will cause nullable type not equal as origin.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)