[improvement](cloud rebalance) make multiple fe consistent of tablet's backend #40771
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
yujun777
force-pushed
the
fe-consistent-tablet-view
branch
5 times, most recently
from
864a5db to
69901d8
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41702 ms |
TPC-DS: Total hot run time: 199762 ms |
yujun777
force-pushed
the
fe-consistent-tablet-view
branch
from
b530068 to
813d3ea
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 42157 ms |
Contributor
Author
|
run buildall |
yujun777
changed the title
yujun777
changed the title
TPC-H: Total hot run time: 42736 ms |
Contributor
Author
|
run performance |
TPC-H: Total hot run time: 41904 ms |
yujun777
force-pushed
the
fe-consistent-tablet-view
branch
from
3c6112e to
6b000ab
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41642 ms |
TPC-DS: Total hot run time: 196430 ms |
ClickBench: Total hot run time: 33.73 s |
gavinchou
reviewed
Sep 24, 2024
| public static boolean enable_immediate_be_assign = true; | ||
| @ConfField(mutable = true, masterOnly = true, | ||
| description = {"存算分离模式下，当tablet分布的be异常，是否立即映射tablet到新的be上，默认false"}) | ||
| public static boolean enable_immediate_be_assign = false; |
Contributor
There was a problem hiding this comment.
this seems a behavior change
Contributor
Author
There was a problem hiding this comment.
this seems a behavior change
默认要开启新算法。这个开关关了，那还是走旧算法
Contributor
|
Add more background and why we choose the solution and what is the trade off |
yujun777
force-pushed
the
fe-consistent-tablet-view
branch
from
0d86f33 to
58b3daa
Compare
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 41393 ms |
TPC-DS: Total hot run time: 192726 ms |
Contributor
Author
|
run buildall |
yujun777
force-pushed
the
fe-consistent-tablet-view
branch
from
3c96447 to
7a077a8
Compare
Contributor
Author
|
run buildall |
yujun777
force-pushed
the
fe-consistent-tablet-view
branch
from
7a077a8 to
080f139
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
run feut |
1 similar comment
Contributor
Author
|
run feut |
TPC-H: Total hot run time: 40927 ms |
TPC-DS: Total hot run time: 191242 ms |
ClickBench: Total hot run time: 32.75 s |
gavinchou
approved these changes
Sep 27, 2024
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
gavinchou
added
p0_c
dev/3.0.x
and removed
approved
cjj2010
pushed a commit
to cjj2010/doris
that referenced
this pull request
Oct 12, 2024
…s backend (apache#40771) When given a tablet, for multiple frontends, this tablet's backends maybe different. Because when a tablet's backend shutdown, the frontends will immedidately change the tablet to a live backend using hash alg. Because changed no write editlog, so the changed is not aync. It may result in a tablet relocating to different backends for different frontends. To fix this problem, each tablet introduces two be id: primary be id and secondary be id. Only master fe can change primary be id actively, and it will write an editlog. So it will ensure all frontends have the same primary be. For a tablet, if its primary be is alive, then use it as its backend, if its primary be is dead, then can chose a secondary be as its backend. More details: 1. when be dead over 1h(Config.rehash_tablet_after_be_dead_seconds), change none-colocate tablet's primary be id to its secondary be id; 2. don't balance on colocate table, and colocate table don't save their primary be id; 3. colocate table rehash on "alive be + be dead less 1h(Config.rehash_tablet_after_be_dead_seconds)" sets, but if the gotten be is dead, change to an alive be. This both ensure availability and avoid all tablets rehashing when be dead for a short time; 4. colocate tablet rehash on group id + bucket index; 5. rehash prefer to skip the decommission be; 6. 'show backends' will print the right tablet num of each backend; 7. cloud tablet show PrimaryBackendId
amorynan
pushed a commit
to amorynan/doris
that referenced
this pull request
Oct 12, 2024
…s backend (apache#40771) When given a tablet, for multiple frontends, this tablet's backends maybe different. Because when a tablet's backend shutdown, the frontends will immedidately change the tablet to a live backend using hash alg. Because changed no write editlog, so the changed is not aync. It may result in a tablet relocating to different backends for different frontends. To fix this problem, each tablet introduces two be id: primary be id and secondary be id. Only master fe can change primary be id actively, and it will write an editlog. So it will ensure all frontends have the same primary be. For a tablet, if its primary be is alive, then use it as its backend, if its primary be is dead, then can chose a secondary be as its backend. More details: 1. when be dead over 1h(Config.rehash_tablet_after_be_dead_seconds), change none-colocate tablet's primary be id to its secondary be id; 2. don't balance on colocate table, and colocate table don't save their primary be id; 3. colocate table rehash on "alive be + be dead less 1h(Config.rehash_tablet_after_be_dead_seconds)" sets, but if the gotten be is dead, change to an alive be. This both ensure availability and avoid all tablets rehashing when be dead for a short time; 4. colocate tablet rehash on group id + bucket index; 5. rehash prefer to skip the decommission be; 6. 'show backends' will print the right tablet num of each backend; 7. cloud tablet show PrimaryBackendId
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When given a tablet, for multiple frontends, this tablet's backends maybe different. Because when a tablet's backend shutdown, the frontends will immedidately change the tablet to a live backend using hash alg. Because changed no write editlog, so the changed is not aync. It may result in a tablet relocating to different backends for different frontends. To fix this problem, each tablet introduces two be id: primary be id and secondary be id. Only master fe can change primary be id actively, and it will write an editlog. So it will ensure all frontends have the same primary be. For a tablet, if its primary be is alive, then use it as its backend, if its primary be is dead, then can chose a secondary be as its backend.
More details: