Skip to content

Conversation

@Yukang-Lian
Copy link
Collaborator

@Yukang-Lian Yukang-Lian commented Aug 9, 2023

Proposed changes

Issue Number: close #xxx

In the process of writing data from block to the segment by the partial update, there will be three different positions, block_pos, delta_pos and segment_pos, and their relationship is shown in the figure below. This PR fixes the use of the wrong positions.

block segment
2 -> 0
3 -> 1 *
4 -> 2
5 -> 3
Take the behavior example with *, we will add block 2-5 rows to segment 0-3 rows. Here * row block_pos = 3, segment_pos = 1, delta_pos = 3-2 = 1.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc..

@github-actions
Copy link
Contributor

github-actions bot commented Aug 9, 2023

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian Yukang-Lian force-pushed the Fix_Segment_Writer_Wrong_Pos branch from cccbbb1 to 003a32b Compare August 14, 2023 09:09
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Yukang-Lian
Copy link
Collaborator Author

run p0

@Yukang-Lian Yukang-Lian force-pushed the Fix_Segment_Writer_Wrong_Pos branch from f980b52 to 1183a58 Compare August 16, 2023 08:01
@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.75 seconds
stream load tsv: 545 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 95.8 seconds inserted 10000000 Rows, about 104K ops/s
storage size: 17162116647 Bytes

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 16, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@zhannngchen zhannngchen merged commit 0aa57d1 into apache:master Aug 16, 2023
xiaokang pushed a commit that referenced this pull request Oct 8, 2023
…n merge-on-write unique table (#25071)

* pick "[Fix](Partial update) Fix wrong position using in segment writer #22782"

* pick "[Enhancement](merge-on-write) use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist #24011"

* pick "[Fix](merge-on-write) Correct the alignment process when the existing rows with same key has marked delete sign #24877"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants