Increase Zookeeper probe timeouts

[lhotari]

Motivation

• 5 seconds seems to be too short a probe timeout on a system with low resources, such as in CI

Modifications

• change default timeout to 30 seconds (which matches the probe's periodSeconds parameter)

[michaeljmarshall]

5 seconds seems to be too short a probe timeout on a system with low resources, such as in CI

Were you seeing CI errors because of the low value? What are the consequences of changing this default? It seems to me that it'd let an unresponsive ZK receive traffic for 25 seconds longer, but zk might have some other mechanism that intercedes first. If it does increase the time to recovery, I am not sure that we should change the default for the chart. Can we update a CI values.yaml instead?

[lhotari]

5 seconds seems to be too short a probe timeout on a system with low resources, such as in CI

Were you seeing CI errors because of the low value? What are the consequences of changing this default? It seems to me that it'd let an unresponsive ZK receive traffic for 25 seconds longer, but zk might have some other mechanism that intercedes first. If it does increase the time to recovery, I am not sure that we should change the default for the chart. Can we update a CI values.yaml instead?

There hasn't been a timeout for k8s <1.20 before #214 . It's better to set a safe default than have a very tight timeout.

[michaeljmarshall]

@lhotari - thanks for the explanation, LGTM