Skip to content

[GLUTEN-8846][CH] [Part 4] Add full-chain UT#9256

Merged
baibaichen merged 4 commits intoapache:mainfrom
jlfsdtc:iceberg_ut
Apr 22, 2025
Merged

[GLUTEN-8846][CH] [Part 4] Add full-chain UT#9256
baibaichen merged 4 commits intoapache:mainfrom
jlfsdtc:iceberg_ut

Conversation

@jlfsdtc
Copy link
Copy Markdown
Contributor

@jlfsdtc jlfsdtc commented Apr 8, 2025 •
edited
Loading

What changes were proposed in this pull request?

  1. Add full-chain UT to GLUTEN-8846

  2. fix the partition table query exception. The exception information is as follows

/home/kylin/Documents/ky_clickhouse/contrib/llvm-project/libcxx/include/vector:1436: assertion __n < size() failed: vector[] index out of bounds
2025-04-18 13:57:07.260 <Fatal> SignalListener: ########################################
2025-04-18 13:57:07.260 <Fatal> SignalListener: (version 25.5.1.1, build id: , git hash: getGitHash()) (from thread 41703) (query_id: 6200ee74-beb6-48e4-b263-332295a48488_0) (query: ) Received signal Aborted (6)
2025-04-18 13:57:07.261 <Fatal> SignalListener: 
2025-04-18 13:57:07.594 <Fatal> SignalListener: 0. /home/kylin/Documents/ky_clickhouse/src/Common/StackTrace.cpp:386: StackTrace::tryCapture() @ 0x0000000015a0d84a
2025-04-18 13:57:07.603 <Fatal> SignalListener: 1. /home/kylin/Documents/ky_clickhouse/src/Common/StackTrace.cpp:355: StackTrace::StackTrace(ucontext_t const&) @ 0x0000000015a0e4e0
2025-04-18 13:57:07.613 <Fatal> SignalListener: 2. /home/kylin/Documents/ky_clickhouse/utils/extern-local-engine/Common/GlutenSignalHandler.cpp:130: signalHandler(int, siginfo_t*, void*) @ 0x00000000161af963
2025-04-18 13:57:07.613 <Fatal> SignalListener: 3. ? @ 0x00007e11b6c45330
2025-04-18 13:57:07.613 <Fatal> SignalListener: 4. ? @ 0x00007e11b6c9eb2d
2025-04-18 13:57:07.613 <Fatal> SignalListener: 5. ? @ 0x00007e11b6c4527e
2025-04-18 13:57:07.613 <Fatal> SignalListener: 6. ? @ 0x00007e11b6c288ff
2025-04-18 13:57:07.638 <Fatal> SignalListener: 7. /home/kylin/Documents/ky_clickhouse/contrib/llvm-project/libcxx/src/verbose_abort.cpp:74: ? @ 0x0000000026591c2e
2025-04-18 13:57:07.645 <Fatal> SignalListener: 8. /home/kylin/Documents/ky_clickhouse/contrib/llvm-project/libcxx/include/vector:1436: std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>::operator[][abi:se190107](unsigned long) @ 0x000000000bdf45a8
2025-04-18 13:57:07.659 <Fatal> SignalListener: 9. /home/kylin/Documents/ky_clickhouse/utils/extern-local-engine/Storages/SubstraitSource/FileReader.cpp:63: _ZZNK12local_engine10BaseReader16addVirtualColumnEN2DB5ChunkEmENK3$_0clINS1_21ColumnWithTypeAndNameEEEN3COWINS1_7IColumnEE13immutable_ptrIS7_EERKT_ @ 0x0000000016540111
2025-04-18 13:57:07.674 <Fatal> SignalListener: 10. /home/kylin/Documents/ky_clickhouse/contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<local_engine::BaseReader::addVirtualColumn(DB::Chunk, unsigned long) const::$_0&>()(std::declval<DB::ColumnWithTypeAndName const&>())) std::__invoke[abi:se190107]<local_engine::BaseReader::addVirtualColumn(DB::Chunk, unsigned long) const::$_0&, DB::ColumnWithTypeAndName const&>(local_engine::BaseReader::addVirtualColumn(DB::Chunk, unsigned long) const::$_0&, DB::ColumnWithTypeAndName const&) @ 0x0000000016540068
2025-04-18 13:57:07.688 <Fatal> SignalListener: 11. /home/kylin/Documents/ky_clickhouse/contrib/llvm-project/libcxx/include/__functional/invoke.h:28: ? @ 0x000000001653ffa8
2025-04-18 13:57:07.700 <Fatal> SignalListener: 12. /home/kylin/Documents/ky_clickhouse/contrib/llvm-project/libcxx/include/__algorithm/ranges_transform.h:51: std::ranges::in_out_result<std::__wrap_iter<DB::ColumnWithTypeAndName const*>, std::back_insert_iterator<std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>> std::ranges::__transform::__fn::__unary[abi:se190107]<std::__wrap_iter<DB::ColumnWithTypeAndName const*>, std::__wrap_iter<DB::ColumnWithTypeAndName const*>, std::back_insert_iterator<std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>, local_engine::BaseReader::addVirtualColumn(DB::Chunk, unsigned long) const::$_0, std::identity>(std::__wrap_iter<DB::ColumnWithTypeAndName const*>, std::__wrap_iter<DB::ColumnWithTypeAndName const*>, std::back_insert_iterator<std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>, local_engine::BaseReader::addVirtualColumn(DB::Chunk, unsigned long) const::$_0&, std::identity&) @ 0x000000001653fee9
2025-04-18 13:57:07.713 <Fatal> SignalListener: 13. /home/kylin/Documents/ky_clickhouse/contrib/llvm-project/libcxx/include/__algorithm/ranges_transform.h:102: std::ranges::in_out_result<std::_IfImpl<borrowed_range<T>>::_Select<decltype(std::ranges::__cpo::begin(std::declval<T&>())), std::ranges::dangling>, T0> std::ranges::__transform::__fn::operator()[abi:se190107]<std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::back_insert_iterator<std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>>>, local_engine::BaseReader::addVirtualColumn(DB::Chunk, unsigned long) const::$_0, std::identity>(T&&, T0, T1, T2) const @ 0x000000001653d5a5
2025-04-18 13:57:07.725 <Fatal> SignalListener: 14. /home/kylin/Documents/ky_clickhouse/utils/extern-local-engine/Storages/SubstraitSource/FileReader.cpp:57: local_engine::BaseReader::addVirtualColumn(DB::Chunk, unsigned long) const @ 0x000000001653d471
2025-04-18 13:57:07.738 <Fatal> SignalListener: 15. /home/kylin/Documents/ky_clickhouse/utils/extern-local-engine/Storages/SubstraitSource/FileReader.cpp:260: local_engine::NormalFileReader::pull(DB::Chunk&) @ 0x000000001653f24b
2025-04-18 13:57:07.748 <Fatal> SignalListener: 16. /home/kylin/Documents/ky_clickhouse/utils/extern-local-engine/Storages/SubstraitSource/SubstraitFileSource.cpp:84: local_engine::SubstraitFileSource::generate() @ 0x000000001653917b
2025-04-18 13:57:07.759 <Fatal> SignalListener: 17. /home/kylin/Documents/ky_clickhouse/src/Processors/ISource.cpp:139: DB::ISource::tryGenerate() @ 0x000000001e74abd6
2025-04-18 13:57:07.769 <Fatal> SignalListener: 18. /home/kylin/Documents/ky_clickhouse/src/Processors/ISource.cpp:108: DB::ISource::work() @ 0x000000001e74a919
2025-04-18 13:57:07.778 <Fatal> SignalListener: 19. /home/kylin/Documents/ky_clickhouse/src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) @ 0x000000001e787a20
2025-04-18 13:57:07.787 <Fatal> SignalListener: 20. /home/kylin/Documents/ky_clickhouse/src/Processors/Executors/ExecutionThreadContext.cpp:102: DB::ExecutionThreadContext::executeTask() @ 0x000000001e7875cc
2025-04-18 13:57:07.802 <Fatal> SignalListener: 21. /home/kylin/Documents/ky_clickhouse/src/Processors/Executors/PipelineExecutor.cpp:305: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x000000001e766457
2025-04-18 13:57:07.820 <Fatal> SignalListener: 22. /home/kylin/Documents/ky_clickhouse/src/Processors/Executors/PipelineExecutor.cpp:174: DB::PipelineExecutor::executeStep(std::atomic<bool>*) @ 0x000000001e765fdb
2025-04-18 13:57:07.833 <Fatal> SignalListener: 23. /home/kylin/Documents/ky_clickhouse/src/Processors/Executors/PullingPipelineExecutor.cpp:54: DB::PullingPipelineExecutor::pull(DB::Chunk&) @ 0x000000001e793573
2025-04-18 13:57:07.845 <Fatal> SignalListener: 24. /home/kylin/Documents/ky_clickhouse/src/Processors/Executors/PullingPipelineExecutor.cpp:65: DB::PullingPipelineExecutor::pull(DB::Block&) @ 0x000000001e793601
2025-04-18 13:57:07.857 <Fatal> SignalListener: 25. /home/kylin/Documents/ky_clickhouse/utils/extern-local-engine/Parser/LocalExecutor.cpp:69: local_engine::LocalExecutor::hasNext() @ 0x0000000015f74e0d
2025-04-18 13:57:07.875 <Fatal> SignalListener: 26. /home/kylin/Documents/ky_clickhouse/utils/extern-local-engine/local_engine_jni.cpp:290: Java_org_apache_gluten_vectorized_BatchIterator_nativeHasNext @ 0x000000000bc13428
2025-04-18 13:57:07.875 <Fatal> SignalListener: Integrity check of the executable skipped because the reference checksum could not be read.
2025-04-18 13:57:07.876 <Fatal> SignalListener: Changed settings: max_download_threads = 1, max_parsing_threads = 1, compile_expressions = false, output_format_compression_level = 18446744071562067968, log_processors_profiles = true, join_use_nulls = true, max_bytes_before_external_group_by = 1073741824, prefer_external_sort_block_bytes = 8388608, max_bytes_before_external_sort = 1717986918, max_bytes_ratio_before_external_sort = 0., max_memory_usage = 2147483648, enable_named_columns_in_function_tuple = false, query_plan_merge_filters = false, query_plan_optimize_lazy_materialization = false, short_circuit_function_evaluation = 'disable', remote_filesystem_read_prefetch = false, function_json_value_return_type_allow_nullable = true, function_json_value_return_type_allow_complex = true, session_timezone = 'Asia/Shanghai', enable_adaptive_memory_spill_scheduler = true, input_format_csv_allow_cr_end_of_line = true, input_format_orc_case_insensitive_column_matching = true, input_format_parquet_case_insensitive_column_matching = true, input_format_orc_allow_missing_columns = true, input_format_orc_dictionary_as_low_cardinality = false, input_format_parquet_allow_missing_columns = true, input_format_parquet_enable_row_group_prefetch = false, input_format_csv_trim_whitespaces = false, input_format_orc_skip_columns_with_unsupported_types_in_schema_inference = true, input_format_json_read_bools_as_numbers = false, input_format_json_read_numbers_as_strings = true, date_time_input_format = 'best_effort', date_time_64_output_format_cut_trailing_zeros_align_to_groups_of_thousands = true, output_format_json_quote_64bit_integers = false, output_format_json_quote_denormals = true, output_format_json_escape_forward_slashes = false, output_format_json_skip_null_value_in_named_tuples = true, output_format_parquet_string_as_string = true, output_format_parquet_fixed_string_as_fixed_byte_array = false, output_format_parquet_version = '1.0', output_format_parquet_compression_method = 'snappy', output_format_parquet_use_custom_encoder = false, output_format_orc_string_as_string = true, output_format_orc_compression_method = 'snappy', precise_float_parsing = true, input_format_parquet_import_nested = true, input_format_orc_import_nested = true, fs.s3a.path.style.access = 'true', fs.s3a.connection.ssl.enabled = 'false', fs.s3a.use.instance.credentials = 'false', fs.s3a.connection.maximum = '15', fs.s3a.retry.limit = '20', spark.sql.decimalOperations.allowPrecisionLoss = true, spark_version = '3.3.1', mergetree.insert_without_local_storage = false, mergetree.merge_after_insert = true

How was this patch tested?

Add UTs

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 8, 2025

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 8, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 9, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 10, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 11, 2025

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 14, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 16, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 18, 2025

Run Gluten Clickhouse CI on x86

@jlfsdtc
Copy link
Copy Markdown
Contributor Author

jlfsdtc commented Apr 18, 2025

image

@jlfsdtc jlfsdtc changed the title [CH][draft] iceberg UT [GLUTEN-8846][CH] [Part 4] Add UT from Icerberg Apr 18, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 18, 2025

#8846

@jlfsdtc jlfsdtc changed the title [GLUTEN-8846][CH] [Part 4] Add UT from Icerberg [GLUTEN-8846][CH] [Part 4] Add full-chain UT Apr 18, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 21, 2025

Run Gluten Clickhouse CI on x86

1 similar comment
@jlfsdtc
Copy link
Copy Markdown
Contributor Author

jlfsdtc commented Apr 21, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 21, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 22, 2025

Run Gluten Clickhouse CI on x86

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 22, 2025

Run Gluten Clickhouse CI on x86

1 similar comment
@jlfsdtc
Copy link
Copy Markdown
Contributor Author

jlfsdtc commented Apr 22, 2025

Run Gluten Clickhouse CI on x86

@baibaichen baibaichen merged commit 3318044 into apache:main Apr 22, 2025
8 checks passed
@jlfsdtc jlfsdtc deleted the iceberg_ut branch April 22, 2025 09:17
@zhouyuan
Copy link
Copy Markdown
Member

zhouyuan commented Apr 22, 2025 •
edited
Loading

Cc @jackylee-ch
it looks like Ck backend added the icberge.version now

@baibaichen baibaichen mentioned this pull request Apr 23, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@baibaichen baibaichen baibaichen approved these changes

Assignees

No one assigned

Labels

CLICKHOUSE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jlfsdtc @zhouyuan @baibaichen