[fix](build index)Fix build index failed on renamed column #42882
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
qidaye
force-pushed
the
fix_renamed_col_build_index
branch
from
8ae42e1 to
afa6caf
Compare
Contributor
Author
|
run buildall |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
qidaye
force-pushed
the
fix_renamed_col_build_index
branch
from
afa6caf to
f79a854
Compare
Contributor
Author
|
run buiildall |
qidaye
force-pushed
the
fix_renamed_col_build_index
branch
from
f79a854 to
bd43995
Compare
Contributor
Author
|
run buildall |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
approved
Contributor
|
PR approved by anyone and no changes requested. |
starocean999
approved these changes
Nov 1, 2024
github-actions bot
pushed a commit
that referenced
this pull request
Nov 1, 2024
Add `column_unique_ids` in `TOlapTableIndex` thrift struct, to get col_unique_id when building index.
16 tasks
airborne12
pushed a commit
that referenced
this pull request
Nov 5, 2024
Add `column_unique_ids` in `TOlapTableIndex` thrift struct, to get col_unique_id when building index.
airborne12
pushed a commit
that referenced
this pull request
Nov 5, 2024
### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close #12345". Otherwise, use "ref #12345" --> Introduced by #42882 <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: #42882 Problem Summary: ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> #### TEST - Upgrade the BE first, then use build index, and it will core dump. - Manual testing shows that it doesn't core dump after the upgrade. 1. Execute ADD INDEX 2. Upgrade BE 3. Execute BUILD INDEX - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [x] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Nov 6, 2024
### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close apache#12345". Otherwise, use "ref apache#12345" --> Introduced by apache#42882 <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: apache#42882 Problem Summary: ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> #### TEST - Upgrade the BE first, then use build index, and it will core dump. - Manual testing shows that it doesn't core dump after the upgrade. 1. Execute ADD INDEX 2. Upgrade BE 3. Execute BUILD INDEX - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [x] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
16 tasks
airborne12
pushed a commit
that referenced
this pull request
Nov 7, 2024
…43336) ### What problem does this PR solve? This PR is a followup repair on the basis of #42882. Building index on renamed column will produce an empty index file, when `column_idx` is not found. The query is successful because match without inverted index. Move test_match_without_index to fault_injection folder and make it non-concurrent to avoid concurrency issues.
github-actions bot
pushed a commit
that referenced
this pull request
Nov 7, 2024
…43336) ### What problem does this PR solve? This PR is a followup repair on the basis of #42882. Building index on renamed column will produce an empty index file, when `column_idx` is not found. The query is successful because match without inverted index. Move test_match_without_index to fault_injection folder and make it non-concurrent to avoid concurrency issues.
qidaye
pushed a commit
to qidaye/incubator-doris
that referenced
this pull request
Nov 7, 2024
### What problem does this PR solve? <!-- You need to clearly describe your PR in this part: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why this function was added. 4. Which codes were refactored and why this part of the code was refactored. 5. Which functions were optimized and what is the difference before and after the optimization. The description of the PR needs to enable reviewers to quickly and clearly understand the logic of the code modification. --> <!-- If there are related issues, please fill in the issue number. - If you want the issue to be closed after the PR is merged, please use "close apache#12345". Otherwise, use "ref apache#12345" --> Introduced by apache#42882 <!-- If this PR is followup a preivous PR, for example, fix the bug that introduced by a related PR, link the PR here --> Related PR: apache#42882 Problem Summary: ### Check List (For Committer) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason <!-- Add your reason? --> #### TEST - Upgrade the BE first, then use build index, and it will core dump. - Manual testing shows that it doesn't core dump after the upgrade. 1. Execute ADD INDEX 2. Upgrade BE 3. Execute BUILD INDEX - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - Release note <!-- bugfix, feat, behavior changed need a release note --> <!-- Add one line release note for this PR. --> None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [x] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
qidaye
added a commit
to qidaye/incubator-doris
that referenced
this pull request
Nov 7, 2024
…pache#43336) ### What problem does this PR solve? This PR is a followup repair on the basis of apache#42882. Building index on renamed column will produce an empty index file, when `column_idx` is not found. The query is successful because match without inverted index. Move test_match_without_index to fault_injection folder and make it non-concurrent to avoid concurrency issues.
16 tasks
dataroaring
pushed a commit
that referenced
this pull request
Mar 20, 2025
) Issue Number: close #48989 Related PR: #46648 #42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters
qidaye
pushed a commit
to qidaye/incubator-doris
that referenced
this pull request
Mar 20, 2025
…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters
qidaye
pushed a commit
to qidaye/incubator-doris
that referenced
this pull request
Mar 20, 2025
…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters
qidaye
pushed a commit
to qidaye/incubator-doris
that referenced
this pull request
Mar 20, 2025
…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters
qidaye
pushed a commit
to qidaye/incubator-doris
that referenced
this pull request
Mar 20, 2025
…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…che#48988) Issue Number: close apache#48989 Related PR: apache#46648 apache#42882 Problem Summary: We introduced `Index.columnUniqueIds` in 3.0.3, and make sure `Index.columnUniqueIds` is not empty in 3.0.4. But when we upgrade Doris from 3.0.2 and before to 3.0.4, the `Index.columnUniqueIds` will be `null`. Versions 302 and earlier do not have this variable, and gson deserialization will assign it to null. We need to initialize the column unique ids associated with the index correctly. Key Changes Removed stored `columnUniqueIds` from `Index` class: Previously, column unique IDs were stored as a field in the `Index` class Now they're dynamically computed when needed rather than stored redundantly Added dynamic computation method: New method `getColumnUniqueIds(List<Column> schema)` computes IDs at runtime Column IDs are looked up by matching index column names with schema columns Updated method signatures: Changed `toThrift()` to `toThrift(List<Integer> indexColumnUniqueIds)` Changed `toPb()` to `toPb(Map<Integer, Column> columnMap, List<Integer> indexColumnUniqueIds)` Updated all callers to pass column IDs as parameters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Add
column_unique_idsinTOlapTableIndexthrift struct, to get col_unique_id when building index.