Skip to content

[fix](merge-on-write) fix duplicate key in schema change #25705

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 6 commits into from
Oct 25, 2023
Merged

[fix](merge-on-write) fix duplicate key in schema change #25705

xiaokang merged 6 commits into from
Oct 25, 2023

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Oct 20, 2023 •
edited
Loading

Proposed changes

  1. It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
  2. The remaining NOTREADY tablet in the schema change failure should be dropped.
  3. When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 20, 2023

run buildall

github-actions[bot]
github-actions bot reviewed Oct 20, 2023
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@liaoxin01 liaoxin01 marked this pull request as draft October 20, 2023 12:36
@liaoxin01 liaoxin01 marked this pull request as ready for review October 24, 2023 11:21
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 24, 2023

run buildall

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 24, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Oct 24, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.52 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162279163 Bytes

@apache apache deleted a comment from github-actions bot Oct 25, 2023
zhannngchen
zhannngchen previously approved these changes Oct 25, 2023
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 25, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 25, 2023

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 25, 2023

run buildall

github-actions[bot]
github-actions bot reviewed Oct 25, 2023
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Oct 25, 2023

run buildall

zhannngchen
zhannngchen approved these changes Oct 25, 2023
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Oct 25, 2023

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

doris-robot commented Oct 25, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.62 seconds
stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162172721 Bytes

xiaokang
xiaokang approved these changes Oct 25, 2023
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang merged commit f31c1d8 into apache:master Oct 25, 2023
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Oct 25, 2023
@liaoxin01
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)
4b77cac
@liaoxin01 liaoxin01 mentioned this pull request Oct 25, 2023
Merged
xiaokang pushed a commit that referenced this pull request Oct 25, 2023
@liaoxin01
[fix](merge-on-write) fix duplicate key in schema change (#25705) (#2…
6f8ea15 
…5914)
dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
@liaoxin01
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)
7c3e600 
It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
The remaining NOTREADY tablet in the schema change failure should be dropped.
When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
@liaoxin01
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)…
ce76137 
… (apache#25914)
@xiaokang xiaokang mentioned this pull request Dec 4, 2023
Closed
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@liaoxin01
[fix](merge-on-write) fix duplicate key in schema change (apache#25705)
f1330f9 
It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
The remaining NOTREADY tablet in the schema change failure should be dropped.
When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.
@liaoxin01 liaoxin01 deleted the fix_sc_master branch February 6, 2024 12:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@xiaokang xiaokang xiaokang approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged merge_conflict p0_b p0_w reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@liaoxin01 @doris-robot @xiaokang @zhannngchen