redis cluster stuck in warming state when there are "two" redis clusters

[ramaraochavali]

We have configured two AWS elastic redis clusters with TLS. After this fix #19541, we were able to use redis in cluster mode.

But now the problem is if we have two such clusters, only one is getting warmed up. The other is stuck in warming. On further debug, we found that the cluster slots command is not returned for the cluster that is stuck in warming. We get upstream_cx_connect_timeout

cluster.aws-redis.mcn-session-store3-mcn-platform.svc.mesh.sfdc.net..6381.upstream_cx_active: 0
cluster.aws-redis.mcn-session-store3-mcn-platform.svc.mesh.sfdc.net..6381.upstream_cx_connect_timeout: 13
cluster.aws-redis.mcn-session-store3-mcn-platform.svc.mesh.sfdc.net..6381.upstream_cx_destroy: 13
cluster.aws-redis.mcn-session-store3-mcn-platform.svc.mesh.sfdc.net..6381.upstream_cx_destroy_local: 13
cluster.aws-redis.mcn-session-store3-mcn-platform.svc.mesh.sfdc.net..6381.upstream_cx_destroy_local_with_active_rq: 13
cluster.aws-redis.mcn-session-store3-mcn-platform.svc.mesh.sfdc.net..6381.upstream_cx_destroy_with_active_rq: 13

But if we configure one of these clusters it works correctly. So either of them if configured work, but not both.

Is this supported? Or any known issues?

[bpo]

@ramaraochavali can you provide details on how you are configuring the two clusters?

[ramaraochavali]

@bpo We are actually checking if AWS elastic cluster is properly configured to return slots or some erratic behaviour on elastic cluster side. Will update this issue once we have some conclusion.

[ramaraochavali]

Also @mattklein123 do you think it is reasonable to extend wait_for_warm_on_init to redis cluster so that in cases such as this we do not stall envoy initialization with one bad external cluster?

[mattklein123]

Also @mattklein123 do you think it is reasonable to extend wait_for_warm_on_init to redis cluster so that in cases such as this we do not stall envoy initialization with one bad external cluster?

Sure SGTM.

[ramaraochavali]

After fixing aws side, we were not able to reproduce this. I will close this and we will follow-up with the warm_on_init for redis cluster