wait and timeout options with Helm-based operator

[jfgosselin]

Feature Request

Describe the problem you need a feature to resolve.

The Helm-based operator does not wait until the Deployments have minimum (Desired minus maxUnavailable) Pods in ready state before marking the release as successful.

Describe the solution you'd like.

Expose a wait and timeout option like the Helm cli and update the CustomResource status accordingly.

--timeout: A value in seconds to wait for Kubernetes commands to complete This defaults to 5m0s
--wait: Waits until all Pods are in a ready state, PVCs are bound, Deployments have minimum (Desired minus maxUnavailable) Pods in ready state and Services have an IP address (and Ingress if a LoadBalancer) before marking the release as successful. It will wait for as long as the --timeout value. If timeout is reached, the release will be marked as FAILED. Note: In scenarios where Deployment has replicas set to 1 and maxUnavailable is not set to 0 as part of rolling update strategy, --wait will return as ready as it has satisfied the minimum Pod in ready condition.

/language helm

[joelanford]

This seems like it would be helpful.

The one concern I have is that Kubernetes controllers generally don't block, waiting for certain conditions to exist. Instead, they make their observations and changes, and then wait until those changes trigger another reconcile loop. The loop then repeats until steady state is reached.

If Helm's --wait conditions could be implemented as I describe above, would that meet your use case?

This has me thinking about --timeout as well. If the above can be implemented, does a timeout like this make sense in the context of an always reconciling operator? I'm not sure it does. With the Helm CLI, after the timeout occurs, the Helm binary exits, presumably leaving the incomplete release resources as is. But the operator is always running, so it will just keep retrying, waiting, timing out, etc. the exact same way every time with an exponential backoff between retries.

[jfgosselin]

@joelanford that will meet my use case. You are right, I don't think the timeout make sense.

[joelanford]

@jfgosselin Cool! Is this something you would be willing to investigate and contribute?

One thing I'm not sure about is whether the Helm libraries expose the same checks that the --wait flag makes without actually having to wait. If it does, it should be possible to incorporate that check into the reconcile loop and possibly tweak or add a status condition.

[drewwells]

This is closely related to #2491. The operator should mark the CR status Ready only after all the resources are ready by k8s's definition of ready.

In Helm, there is an interface that implements ready checking of multiple resources https://github.com/helm/helm/blob/9b42702a4bced339ff424a78ad68dd6be6e1a80a/pkg/kube/interface.go#L38-L44

The hooks that perform the checks are here
https://github.com/helm/helm/blob/193850a9e2c509acf1a499d98e8d23c12c134f11/pkg/action/hooks.go#L58-L83

[joelanford]

@drewwells It looks like WatchUntilReady is used only for hook execution, and it also looks like it is always called, even if --wait is not set. So I think the helm-operator is likely already waiting for hooks.

For --wait functionality, we would want to mimic or re-use the KubeClient.Wait() function. Here's the usage in action.Install.

Unfortunately, it looks like the waiter struct and waiter.waitForResources() function are both unexported and they block, so we can't re-use them in the SDK without actually using the action.Install.Wait value on the struct, which I think we should avoid.

Ideally this block of logic would be extracted and exported into a new IsResourceReady(r Resource) (bool, error) function.

[drewwells]

I agree about the action package, it's very lacking in its flexibility.

You're right about WatchUntilReady. There's another similarly named method that is public and constructs a waiter https://github.com/helm/helm/blob/593fec6868d7a239032cf234b530aee6f5d52708/pkg/kube/client.go#L116-L128 We won't have any errors to show until the Timeout elapses though and possibly no other reconciliations.

What is the objection to using action.Install.Wait flag? I think even using client.Wait here will have limited visibility from within the operator itself.

[joelanford]

@drewwells There are a few reasons I don't think we should use action.Install.Wait.

As I mentioned in my first comment, it is generally an anti-pattern to wait in Kubernetes controllers. The more accepted pattern is to observe the current state of the CR and its resources (e.g. ask is the release ready?), make any changes (e.g. set CR status based on whether the resources are ready), and immediately return. Then if/when any of the status blocks of any of the release resources change, our reconciler will be triggered again, and we can again check if the release is ready. This process repeats until all of the release resource status blocks achieve steady state. If the final steady state is that all of the resources are ready, then the reconciler will set the CR status to reflect that.

With this design the operator can scale to handle a very large number of CRs and their releases being active at the same time.

If the reconciler is allowed to block, the operator will only be able to handle MaxConcurrentReconciles CRs simultaneously. Once that number of reconcilers is waiting, all other events will be queued until an existing CR is finished waiting or it times out.

Another benefit of this design is that the question of whether a release is ready is always checked. If steady state was reached and the release became ready, that doesn't mean it will ALWAYS be ready. If a pod terminates or a service loses its endpoints, the reconciler will be triggered and it will change the CR status from ready to not ready.