[Fix](segment compaction) fix error using of inverted index file writer in segment compaction #43114 #43616
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…er in segment compaction (apache#43114) ### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close apache#12345". Otherwise, use "ref apache#12345" --> <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: apache#41625 Problem Summary: Fix BUAF problem, stack like this ``` stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421 1# 0x00007F5370ABD520 in /lib/x86_64-linux-gnu/libc.so.6 2# pthread_kill at ./nptl/pthread_kill.c:89 3# raise at ../sysdeps/posix/raise.c:27 4# abort at ./stdlib/abort.c:81 5# _gnu_cxx::_verbose_terminate_handler() [clone .cold] at ../../../../libstdc+-v3/libsupc+/vterminate.cc:75 6# _cxxabiv1::_terminate(void ()) at ../../../../libstdc+-v3/libsupc+/eh_terminate.cc:48 7# 0x000055D9E015B681 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 8# 0x000055D9E015B7D4 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 9# 0x000055D9E015BBC6 in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 10# void fmt::v7::detail::buffer::append(char const*, char const*) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 11# char const* fmt::v7::detail::parse_replacement_field, char, fmt::v7::basic_format_context, char> >&>(char const*, char const*, fmt::v7::detail::format_handler, char, fmt::v7::basic_format_context, char> >&) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 12# void fmt::v7::detail::vformat_to(fmt::v7::detail::buffer&, fmt::v7::basic_string_view, fmt::v7::basic_format_args::type>, fmt::v7::type_identity::type> >, fmt::v7::detail::locale_ref) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 13# fmt::v7::detail::vformat[abi:cxx11](fmt::v7::basic_string_view, fmt::v7::format_args) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 14# doris::segment_v2::InvertedIndexDescriptor::get_temporary_index_path[abi:cxx11](std::basic_string_view >, std::basic_string_view >, long, long, std::basic_string_view >) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_desc.cpp:35 15# doris::segment_v2::InvertedIndexFileWriter::open(doris::TabletIndex const*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp:45 16# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)7>::init_bkd_index() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:146 17# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)7>::init() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:116 18# doris::segment_v2::InvertedIndexColumnWriter::create(doris::Field const*, std::unique_ptr >, doris::segment_v2::InvertedIndexFileWriter, doris::TabletIndex const*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:698 19# doris::segment_v2::ScalarColumnWriter::init() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:483 20# doris::segment_v2::SegmentWriter::_create_column_writer(unsigned int, doris::TabletColumn const&, std::shared_ptr const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:258 21# doris::segment_v2::SegmentWriter::_create_writers(std::shared_ptr const&, std::vector > const&) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:307 22# doris::segment_v2::SegmentWriter::init(std::vector > const&, bool) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:276 23# doris::SegcompactionWorker::_do_compact_segments(std::shared_ptr, std::allocator > > >) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 24# doris::SegcompactionWorker::compact_segments(std::shared_ptr, std::allocator > > >) at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segcompaction.cpp:354 25# doris::StorageEngine::_handle_seg_compaction(std::shared_ptr, std::shared_ptr, std::allocator > > >, unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/olap/olap_server.cpp:1121 26# std::_Function_handler, std::shared_ptr, std::allocator > > >)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 27# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be 28# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:499 29# start_thread at ./nptl/pthread_create.c:442 30# 0x00007F5370BA1850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83 172.20.50.120 last coredump sql: last SQL query not found ``` ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
Member
Author
|
run buildall |
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| } | ||
| } | ||
|
|
||
| TEST_F(SegCompactionTest, CreateSegCompactionWriter) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
TEST_F(SegCompactionTest, CreateSegCompactionWriter) {
^Additional context
be/test/olap/segcompaction_test.cpp:851: 264 lines including whitespace and comments (threshold 80)
TEST_F(SegCompactionTest, CreateSegCompactionWriter) {
^
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cherry pick from #43114