[Bug] Lots of version incomplete replica created when decommission BE

[morningman]

Describe the bug
When decommission BE, some of tablets may has lots of replicas generated by clone task.

Why

Give examples of why:

1. Tablet X has 3 replicas on A, B, C 3 BEs.
2. C is decommission, so we choose the Backend D to relocating the new replica,
3. After relocating, Tablet X has 4 replicas: A, B, C(decommision), D(may be version incomplete)
4. D may be version incomplete because the clone task ran a long time, the new version has been published.
5. At the next time of tablet checking, Tablet X's status is still REPLICA_RELOCATING.
   If we don't choose D as dest BE to do the new relocating, it will choose new backend E
   to store the new replicas. So back and forth, the number of replicas will increase forever.

So a better solution is to select D as dest BE again to do the clone task. This may trigger an increment clone task
that can be done more fast.

[francisoliverlee]

@morningman
what's the doris version in the issue, and how do you do to make the need-relocated-tablets relocate to D, not other BEs?
as what i know, relocate-task only can be started by doris itsself, we cound change the task priority, can not deside which BE the need-relocated-tablets can go

[kangkaisen]

@morningman

So a better solution is to select C as dest BE again to do the clone task.

Which should be D?

[morningman]

@morningman
what's the doris version in the issue, and how do you do to make the need-relocated-tablets relocate to D, not other BEs?
as what i know, relocate-task only can be started by doris itsself, we cound change the task priority, can not deside which BE the need-relocated-tablets can go

When I say we choose the Backend D to relocating the new replica, I do not mean to "choose" the D, D just represents any one of Backend being chosen.
The lasted Doris version has this problem.

[morningman]

Which should be D?

I don't catch your meaning...

[morningman]

Which should be D?

I don't catch your meaning...

I bad, it should be D.

[francisoliverlee]

@morningman
what's the doris version in the issue, and how do you do to make the need-relocated-tablets relocate to D, not other BEs?
as what i know, relocate-task only can be started by doris itsself, we cound change the task priority, can not deside which BE the need-relocated-tablets can go

When I say we choose the Backend D to relocating the new replica, I do not mean to "choose" the D, D just represents any one of Backend being chosen.
The lasted Doris version has this problem.

what problem?