Skip to content

[Optimize][Clone] Take version count into consideration when choosing src replica for clone task #6513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
morningman merged 5 commits into from
Aug 30, 2021
Merged

[Optimize][Clone] Take version count into consideration when choosing src replica for clone task #6513

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
30805f6
select src replica with min version count for clone task
Aug 24, 2021
3d91e84
select src replica with min version count for clone task
Aug 26, 2021
4410d81
select src replica with min version count for clone task
Aug 27, 2021
d6239f0
select src replica with min version count for clone task
Aug 27, 2021
4dddb05
select src replica with min version count for clone task
Aug 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions fe/fe-core/src/main/java/org/apache/doris/clone/TabletSchedCtx.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,8 +489,10 @@ public void chooseSrcReplica(Map<Long, PathSlot> backendsWorkingSlots) throws Sc
if (candidates.isEmpty()) {
throw new SchedException(Status.UNRECOVERABLE, "unable to find source replica");
}

// choose a replica which slot is available from candidates.
long minVersionCount = Long.MAX_VALUE;
boolean findSrcReplica = false;
for (Replica srcReplica : candidates) {
PathSlot slot = backendsWorkingSlots.get(srcReplica.getBackendId());
if (slot == null) {
Expand All @@ -499,10 +501,26 @@ public void chooseSrcReplica(Map<Long, PathSlot> backendsWorkingSlots) throws Sc

long srcPathHash = slot.takeSlot(srcReplica.getPathHash());
if (srcPathHash != -1) {
setSrc(srcReplica);
return;
if (!findSrcReplica) {
// version count is set by report process, so it may not be set yet and default value is -1.
// so we need to check it.
minVersionCount = srcReplica.getVersionCount() == -1 ? Long.MAX_VALUE : srcReplica.getVersionCount();
setSrc(srcReplica);
findSrcReplica = true;
} else {
long curVerCount = srcReplica.getVersionCount() == -1 ? Long.MAX_VALUE : srcReplica.getVersionCount();
if (curVerCount < minVersionCount) {
minVersionCount = curVerCount;
setSrc(srcReplica);
findSrcReplica = true;
}
}
}
}

if (findSrcReplica) {
return;
}

throw new SchedException(Status.SCHEDULE_FAILED, "unable to find source slot");
}
Expand Down