[Feature](Row store) support column group with store row format for partial columns of table #34089
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| int32_t rs_column_uid() const { return _rs_column_uid; } | ||
|
|
||
| const std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; } |
Contributor
There was a problem hiding this comment.
warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
Suggested change
| const std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; } | |
| std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; } |
|
|
||
| const std::unordered_set<int32_t> missing_col_uids() const { return _missing_col_uids; } | ||
|
|
||
| const std::unordered_set<int32_t> include_col_uids() const { return _include_col_uids; } |
Contributor
There was a problem hiding this comment.
warning: return type 'const int' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]
Suggested change
| const std::unordered_set<int32_t> include_col_uids() const { return _include_col_uids; } | |
| std::unordered_set<int32_t> include_col_uids() const { return _include_col_uids; } |
eldenmoon
changed the title
Member
Author
|
run buildall |
|
TeamCity be ut coverage result: |
eldenmoon
changed the title
Member
Author
|
run buildall |
TPC-H: Total hot run time: 40423 ms |
TPC-DS: Total hot run time: 188110 ms |
|
TeamCity be ut coverage result: |
xiaokang
reviewed
May 13, 2024
fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
Outdated
Show resolved
Hide resolved
| } | ||
| } | ||
|
|
||
| if (properties != null) { |
Contributor
There was a problem hiding this comment.
put it inside row store branch
fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
Outdated
Show resolved
Hide resolved
xiaokang
reviewed
May 13, 2024
be/src/olap/tablet_schema.h
Outdated
|
|
||
| // groups of this column belongs to | ||
| // contains column ids of group, and encode the group of columns into row store | ||
| std::vector<int32_t> _group_ids; |
be/src/olap/tablet_schema.h
Outdated
| void set_store_row_column(bool store_row_column) { _store_row_column = store_row_column; } | ||
| bool store_row_column() const { return _store_row_column; } | ||
| // indicate if full row store column(all the columns encodes as row) exists | ||
| bool has_full_row_store_column() const { return _store_row_column; } |
Contributor
There was a problem hiding this comment.
consistent name is better
be/src/vec/jsonb/serialize.h
Outdated
| static void block_to_jsonb(const TabletSchema& schema, const Block& block, ColumnString& dst, | ||
| int num_cols, const DataTypeSerDeSPtrs& serdes); | ||
| int num_cols, const DataTypeSerDeSPtrs& serdes, | ||
| const std::unordered_set<int32_t>& group_cids); |
Contributor
There was a problem hiding this comment.
consistent name include_cids is better
| std::unordered_map<uint32_t, uint32_t> _col_uid_to_idx; | ||
| std::vector<std::string> _col_default_values; | ||
| // picked rowstore(column group) column unique id | ||
| int32_t _rs_column_uid; |
Contributor
There was a problem hiding this comment.
what's the meaning of name rs?
| const TabletColumn& target_rs_column = schema.column_by_uid(target_rs_column_id); | ||
| DCHECK(target_rs_column.is_row_store_column()); | ||
| // The full column group is considered a full match, thus no missing cids | ||
| if (target_rs_column.group_ids().empty()) { |
Contributor
There was a problem hiding this comment.
consider that if it's right after light schema chage
eldenmoon
force-pushed
the
rs-group
branch
4 times, most recently
from
6f0854c to
1981d22
Compare
Member
Author
|
run buildall |
1 similar comment
Member
Author
|
run buildall |
|
TeamCity be ut coverage result: |
Member
Author
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40900 ms |
TPC-DS: Total hot run time: 172450 ms |
Member
Author
|
run buildall |
|
TeamCity be ut coverage result: |
Member
Author
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 40387 ms |
TPC-DS: Total hot run time: 173431 ms |
ClickBench: Total hot run time: 30.77 s |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
lxr599
approved these changes
Jun 19, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jun 21, 2024
…artial columns of table (#34089)
eldenmoon
added a commit
that referenced
this pull request
Jun 25, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jun 26, 2024
16 tasks
morrySnow
added a commit
that referenced
this pull request
Feb 14, 2025
### What problem does this PR solve? Problem Summary: 1. Because defer materialization in BE do not support project on scan node, so we use a little trick to do projection on last fragment's OutputExprs. 2. fix core when table use partial column row store, introduced by #34089 thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start
lzyy2024
pushed a commit
to lzyy2024/doris
that referenced
this pull request
Feb 21, 2025
### What problem does this PR solve? Problem Summary: 1. Because defer materialization in BE do not support project on scan node, so we use a little trick to do projection on last fragment's OutputExprs. 2. fix core when table use partial column row store, introduced by apache#34089 thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
### What problem does this PR solve? Problem Summary: 1. Because defer materialization in BE do not support project on scan node, so we use a little trick to do projection on last fragment's OutputExprs. 2. fix core when table use partial column row store, introduced by apache#34089 thrift error, reason=No more data to read.F20250211 12:07:07.526710 15029400 rowid_fetcher.cpp:410] Check failed: tablet->tablet_schema()->has_row_store_for_all_columns() *** Check failure stack trace: *** @ 0x10ddd2830 google::LogMessageFatal::~LogMessageFatal() @ 0x10ddce834 google::LogMessageFatal::~LogMessageFatal() @ 0x102a961a8 doris::RowIdStorageReader::read_by_rowids() @ 0x103938624 std::__1::__function::__func<>::operator()() @ 0x103922390 doris::WorkThreadPool<>::work_thread() @ 0x1039228fc std::__1::__thread_proxy<>() @ 0x18300f2e4 _pthread_start
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...