[improve](txn insert) Txn load support cloud mode #34721
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 40555 ms |
TPC-DS: Total hot run time: 185951 ms |
gavinchou
reviewed
May 11, 2024
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
2 times, most recently
from
8078ba3 to
296f6a1
Compare
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
Contributor
Author
|
run buildall |
Contributor
|
clang-tidy review says "All clean, LGTM! 👍" |
TPC-H: Total hot run time: 41166 ms |
TPC-DS: Total hot run time: 188753 ms |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
296f6a1 to
1fc1a2b
Compare
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
1fc1a2b to
a493ddf
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| } | ||
| } | ||
|
|
||
| TEST(MetaServiceTest, CommitTxnWithSubTxnTest) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^Additional context
cloud/test/meta_service_test.cpp:1334: 185 lines including whitespace and comments (threshold 80)
TEST(MetaServiceTest, CommitTxnWithSubTxnTest) {
^| } | ||
| } | ||
|
|
||
| TEST(MetaServiceTest, BeginAndAbortSubTxnTest) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^Additional context
cloud/test/meta_service_test.cpp:1521: 118 lines including whitespace and comments (threshold 80)
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41841 ms |
TPC-DS: Total hot run time: 187484 ms |
ClickBench: Total hot run time: 30.74 s |
| } | ||
| } | ||
|
|
||
| TEST(MetaServiceTest, BeginAndAbortSubTxnTest) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST' exceeds recommended size/complexity thresholds [readability-function-size]
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^Additional context
cloud/test/meta_service_test.cpp:1521: 119 lines including whitespace and comments (threshold 80)
TEST(MetaServiceTest, BeginAndAbortSubTxnTest) {
^
Contributor
Author
|
run buildall |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
4c9df64 to
80a107e
Compare
Contributor
Author
|
run buildall |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
2 times, most recently
from
94bc14a to
16a2691
Compare
Contributor
Author
|
run buildall |
TPC-DS: Total hot run time: 182574 ms |
Contributor
Author
|
run fe_ut |
Contributor
Author
|
run performance |
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
8981044 to
afd5858
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41029 ms |
TPC-DS: Total hot run time: 172961 ms |
ClickBench: Total hot run time: 30.23 s |
Contributor
Author
|
run cloud_p0 |
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
mymeiyi
force-pushed
the
txn-insert-cloud-2
branch
from
afd5858 to
029385e
Compare
github-actions
bot
removed
the
approved
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41093 ms |
TPC-DS: Total hot run time: 172097 ms |
ClickBench: Total hot run time: 30.84 s |
Contributor
Author
|
run cloud_ut |
Contributor
Author
|
run p0 |
Contributor
Author
|
run cloudut |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
dataroaring
pushed a commit
that referenced
this pull request
Jun 7, 2024
## Proposed changes ### Purpose The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load. #32980 implements one txn write to one partition more than one rowsets. This pr implements to cloud mode of #32980 ### Implementation #### sub_txn_id see #32980 #### Meta service supports commit txn This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns. One example: Suppose the table, partition, tablet and version info is: ``` -------------------------------------------- | table | partition | tablet | version | -------------------------------------------- | t1 | t1_p1 | t1_p1.1 | 1 | | t1 | t1_p1 | t1_p1.2 | 1 | | t1 | t1_p2 | t1_p2.1 | 2 | | t2 | t2_p3 | t2_p3.1 | 3 | | t2 | t2_p4 | t2_p4.1 | 4 | -------------------------------------------- ``` Now we commit a txn with 3 sub txns and the tablets are: * sub_txn1: t1_p1.1, t1_p1.2, t1_p2.1 * sub_txn2: t2_p3.1 * sub_txn3: t1_p1.1, t1_p1.2 When commit, the partitions version will be: * sub_txn1: t1_p1(1 -> 2), t1_p2(2 -> 3) * sub_txn2: t2_p3(3 -> 4) * sub_txn3: t1_p1(2 -> 3) After commit, the partitions version will be: * t1: t1_p1(3), t1_p2(3) * t2: t2_p3(4), t2_p4(4) #### Meta service support generate sub_txn_id by `begin_sub_txn`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Purpose
The user doc: https://doris.apache.org/zh-CN/docs/dev/data-operate/import/transaction-load-manual
We have supported insert into select(#31666), update(#33034) and delete(#33100) in transaction load.
#32980 implements one txn write to one partition more than one rowsets.
This pr implements to cloud mode of #32980
Implementation
sub_txn_id
see #32980
Meta service supports commit txn
This process is generally the same as commit_txn, the difference is that he partitions version will plus 1 in multi sub txns.
One example:
Suppose the table, partition, tablet and version info is:
Now we commit a txn with 3 sub txns and the tablets are:
When commit, the partitions version will be:
After commit, the partitions version will be:
Meta service support generate sub_txn_id by
begin_sub_txnFurther comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...