[Fix](delete command) Mark delete sign when do delete command in MoW table #35917
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Collaborator
Author
|
run buildall |
TPC-H: Total hot run time: 42255 ms |
TPC-DS: Total hot run time: 170489 ms |
ClickBench: Total hot run time: 30.43 s |
Yukang-Lian
force-pushed
the
Fix-Partial-Update-Delete-Predicate-Delete-Sign
branch
from
d8665f4 to
8387e79
Compare
Collaborator
Author
|
run buildall |
TPC-H: Total hot run time: 39676 ms |
TPC-DS: Total hot run time: 173875 ms |
ClickBench: Total hot run time: 30.32 s |
Yukang-Lian
force-pushed
the
Fix-Partial-Update-Delete-Predicate-Delete-Sign
branch
from
8387e79 to
485badc
Compare
Collaborator
Author
|
run buildall |
2 similar comments
Collaborator
Author
|
run buildall |
Collaborator
Author
|
run buildall |
Yukang-Lian
force-pushed
the
Fix-Partial-Update-Delete-Predicate-Delete-Sign
branch
from
f92a1a3 to
4df1572
Compare
Collaborator
Author
|
run buildall |
1 similar comment
Collaborator
Author
|
run buildall |
TPC-H: Total hot run time: 39823 ms |
TPC-DS: Total hot run time: 170324 ms |
ClickBench: Total hot run time: 30.92 s |
Collaborator
Author
|
run buildall |
TPC-H: Total hot run time: 40628 ms |
TPC-DS: Total hot run time: 170936 ms |
ClickBench: Total hot run time: 30.3 s |
Yukang-Lian
force-pushed
the
Fix-Partial-Update-Delete-Predicate-Delete-Sign
branch
2 times, most recently
from
d2405ba to
ab893b2
Compare
Collaborator
Author
|
run buildall |
morrySnow
reviewed
Jun 19, 2024
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DeleteFromCommand.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/DeleteFromCommand.java
Outdated
Show resolved
Hide resolved
TPC-H: Total hot run time: 39657 ms |
TPC-DS: Total hot run time: 172396 ms |
|
那个版本发布后，修复此问题呢？ |
Contributor
|
@weilai201 2.1.5 |
dataroaring
pushed a commit
that referenced
this pull request
Jun 28, 2024
…table (#35917) ## Proposed changes close #34551 Problem: As shown in the issue above, if a key deleted by a delete statement is written to by updating only certain columns, the data will not display correctly. Reason: The delete statement deletes the data by writing a delete predicate, which is stored in the rowset meta and applied during data retrieval to filter the data. However, partial column updates do not consider the effect of the delete predicate when reading the original data. The imported key should be considered as a new key (since it has already been deleted), but it is actually treated as an old key. Therefore, only some columns are updated, leading to incorrect results. Solution: Consider the delete predicate during partial column updates, but this method will result in reading more columns, as shown in #35766. Thus, in this PR, we change the delete operation in the mow table from writing a delete predicate to writing a delete sign, which effectively resolves the issue.
gavinchou
pushed a commit
that referenced
this pull request
Jul 5, 2024
…d mode (#37151) Problem: The `test_new_partial_update` case fails to run in cloud mode, but it passes in local mode. Reason: In PR #35917, we introduced a new table attribute `enable_light_delete`. When executing schema changes with `alter table set xxx` statements, the local and cloud modes process the logic differently. The cloud mode has its unique processing logic, which was not addressed in the mentioned PR, leading to failures in the cloud environment. Solution: To resolve the issue, we need to complete the missing schema change logic for the cloud mode. Once this is implemented, the problem should be resolved.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Jul 10, 2024
…table (apache#35917) close apache#34551 Problem: As shown in the issue above, if a key deleted by a delete statement is written to by updating only certain columns, the data will not display correctly. Reason: The delete statement deletes the data by writing a delete predicate, which is stored in the rowset meta and applied during data retrieval to filter the data. However, partial column updates do not consider the effect of the delete predicate when reading the original data. The imported key should be considered as a new key (since it has already been deleted), but it is actually treated as an old key. Therefore, only some columns are updated, leading to incorrect results. Solution: Consider the delete predicate during partial column updates, but this method will result in reading more columns, as shown in apache#35766. Thus, in this PR, we change the delete operation in the mow table from writing a delete predicate to writing a delete sign, which effectively resolves the issue.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Jul 10, 2024
…d mode (apache#37151) Problem: The `test_new_partial_update` case fails to run in cloud mode, but it passes in local mode. Reason: In PR apache#35917, we introduced a new table attribute `enable_light_delete`. When executing schema changes with `alter table set xxx` statements, the local and cloud modes process the logic differently. The cloud mode has its unique processing logic, which was not addressed in the mentioned PR, leading to failures in the cloud environment. Solution: To resolve the issue, we need to complete the missing schema change logic for the cloud mode. Once this is implemented, the problem should be resolved.
This was referenced Jul 10, 2024
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Jul 15, 2024
…table (apache#35917) close apache#34551 Problem: As shown in the issue above, if a key deleted by a delete statement is written to by updating only certain columns, the data will not display correctly. Reason: The delete statement deletes the data by writing a delete predicate, which is stored in the rowset meta and applied during data retrieval to filter the data. However, partial column updates do not consider the effect of the delete predicate when reading the original data. The imported key should be considered as a new key (since it has already been deleted), but it is actually treated as an old key. Therefore, only some columns are updated, leading to incorrect results. Solution: Consider the delete predicate during partial column updates, but this method will result in reading more columns, as shown in apache#35766. Thus, in this PR, we change the delete operation in the mow table from writing a delete predicate to writing a delete sign, which effectively resolves the issue.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Jul 15, 2024
…d mode (apache#37151) Problem: The `test_new_partial_update` case fails to run in cloud mode, but it passes in local mode. Reason: In PR apache#35917, we introduced a new table attribute `enable_light_delete`. When executing schema changes with `alter table set xxx` statements, the local and cloud modes process the logic differently. The cloud mode has its unique processing logic, which was not addressed in the mentioned PR, leading to failures in the cloud environment. Solution: To resolve the issue, we need to complete the missing schema change logic for the cloud mode. Once this is implemented, the problem should be resolved.
zhannngchen
pushed a commit
that referenced
this pull request
Jul 15, 2024
dataroaring
pushed a commit
that referenced
this pull request
Jul 17, 2024
…d mode (#37151) Problem: The `test_new_partial_update` case fails to run in cloud mode, but it passes in local mode. Reason: In PR #35917, we introduced a new table attribute `enable_light_delete`. When executing schema changes with `alter table set xxx` statements, the local and cloud modes process the logic differently. The cloud mode has its unique processing logic, which was not addressed in the mentioned PR, leading to failures in the cloud environment. Solution: To resolve the issue, we need to complete the missing schema change logic for the cloud mode. Once this is implemented, the problem should be resolved.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Jul 29, 2024
…table (apache#35917) close apache#34551 Problem: As shown in the issue above, if a key deleted by a delete statement is written to by updating only certain columns, the data will not display correctly. Reason: The delete statement deletes the data by writing a delete predicate, which is stored in the rowset meta and applied during data retrieval to filter the data. However, partial column updates do not consider the effect of the delete predicate when reading the original data. The imported key should be considered as a new key (since it has already been deleted), but it is actually treated as an old key. Therefore, only some columns are updated, leading to incorrect results. Solution: Consider the delete predicate during partial column updates, but this method will result in reading more columns, as shown in apache#35766. Thus, in this PR, we change the delete operation in the mow table from writing a delete predicate to writing a delete sign, which effectively resolves the issue.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Jul 29, 2024
…d mode (apache#37151) Problem: The `test_new_partial_update` case fails to run in cloud mode, but it passes in local mode. Reason: In PR apache#35917, we introduced a new table attribute `enable_light_delete`. When executing schema changes with `alter table set xxx` statements, the local and cloud modes process the logic differently. The cloud mode has its unique processing logic, which was not addressed in the mentioned PR, leading to failures in the cloud environment. Solution: To resolve the issue, we need to complete the missing schema change logic for the cloud mode. Once this is implemented, the problem should be resolved.
xiaokang
pushed a commit
that referenced
this pull request
Jul 31, 2024
gavinchou
pushed a commit
that referenced
this pull request
Sep 13, 2024
In #35917 and #37151, we changed MOW table default delete command from delete predicate to delete sign. It makes sure the correctness during partial update but leads to slowdowns. Actually, if there is no partial update, delete predicate will not lead to data fault. Delete data by delete predicate or delete sign can be controlled by a table property "enable_light_delete". If "enable_light_delete=true", we execute delete command by delete predicate. Otherwise, we execute delete command by delete sign. In p2 cases, there are lots of cases with large data need to delete and do not perform partial column update operations. Therefore, in order to make it faster, we change some cases default create table clause.
dataroaring
pushed a commit
that referenced
this pull request
Oct 9, 2024
In #35917 and #37151, we changed MOW table default delete command from delete predicate to delete sign. It makes sure the correctness during partial update but leads to slowdowns. Actually, if there is no partial update, delete predicate will not lead to data fault. Delete data by delete predicate or delete sign can be controlled by a table property "enable_light_delete". If "enable_light_delete=true", we execute delete command by delete predicate. Otherwise, we execute delete command by delete sign. In p2 cases, there are lots of cases with large data need to delete and do not perform partial column update operations. Therefore, in order to make it faster, we change some cases default create table clause.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Oct 12, 2024
…7689) In apache#35917 and apache#37151, we changed MOW table default delete command from delete predicate to delete sign. It makes sure the correctness during partial update but leads to slowdowns. Actually, if there is no partial update, delete predicate will not lead to data fault. Delete data by delete predicate or delete sign can be controlled by a table property "enable_light_delete". If "enable_light_delete=true", we execute delete command by delete predicate. Otherwise, we execute delete command by delete sign. In p2 cases, there are lots of cases with large data need to delete and do not perform partial column update operations. Therefore, in order to make it faster, we change some cases default create table clause.
Yukang-Lian
added a commit
to Yukang-Lian/doris
that referenced
this pull request
Oct 12, 2024
…7689) In apache#35917 and apache#37151, we changed MOW table default delete command from delete predicate to delete sign. It makes sure the correctness during partial update but leads to slowdowns. Actually, if there is no partial update, delete predicate will not lead to data fault. Delete data by delete predicate or delete sign can be controlled by a table property "enable_light_delete". If "enable_light_delete=true", we execute delete command by delete predicate. Otherwise, we execute delete command by delete sign. In p2 cases, there are lots of cases with large data need to delete and do not perform partial column update operations. Therefore, in order to make it faster, we change some cases default create table clause.
mongo360
pushed a commit
to mongo360/doris
that referenced
this pull request
Dec 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
close #34551
Problem: As shown in the issue above, if a key deleted by a delete statement is written to by updating only certain columns, the data will not display correctly.
Reason: The delete statement deletes the data by writing a delete predicate, which is stored in the rowset meta and applied during data retrieval to filter the data. However, partial column updates do not consider the effect of the delete predicate when reading the original data. The imported key should be considered as a new key (since it has already been deleted), but it is actually treated as an old key. Therefore, only some columns are updated, leading to incorrect results.
Solution: Consider the delete predicate during partial column updates, but this method will result in reading more columns, as shown in #35766. Thus, in this PR, we change the delete operation in the mow table from writing a delete predicate to writing a delete sign, which effectively resolves the issue.