Tiflash replica is not available when a tiflash node is down

[jiyfhust]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. depoly a tidb cluster with 2 tiflash node
2. create a table t(id int);
3. let one of the tiflash down
4. alter table t set tiflash replica 1
5. select * from information_schema.TIFLASH_REPLICA; The value of AVAILABLE if always 0，and tiflash can not be used for handing query.

2. What did you expect to see? (Required)

Because there are two tiflash nodes，tiflash should be available even one of them is down.

3. What did you see instead (Required)

Tiflash replica of table t is always unavailable untile the down tiflash node is up again.

4. What is your TiDB version? (Required)

master

[jiyfhust]

tidb/ddl/ddl_tiflash_api.go

Line 368 in 2b3b809

err := helper.CollectTiFlashStatus(store.Store.StatusAddress, tableID, &regionReplica)

When a tiflash node is down，CollectTiFlashStatus return error, then it fails to get tiflash progress.

tidb/ddl/ddl_tiflash_api.go

Line 570 in 2b3b809

if err := d.UpdateTableReplicaInfo(ctx, tb.ID, avail); err != nil {

The tableInfo of table t can't be set available。

So:

1. Is it need to check tiflash node status before CollectTiFlashStatus?
2. If skip CollectTiFlashStatus from the down tiflash node
   • The available state can be set in the case
   • The progress of other table may be <1 when the down tiflash node have region replicas of the table

How to solve it properly?