Capture and expose debug (log) information on wait timeout

[kingdonb]

If a helm release has wait enabled, the default setting, there is a chance of failure due to something obscure and hard to detect which is only logged in debug mode. Like when a service: LoadBalancer is used on a home lab cluster that doesn't enable LB, opting instead for HostNetworking ingress. Helm will wait for the LoadBalancer service to become ready, which it will never do, and the error message will be context deadline exceeded or some fairly unhelpful message that is not by itself enough to trace back to the source of the problem. You have to enable debug mode.

This is not strictly a Helm Controller bug, since HC faithfully reproduces the behavior of Helm upstream in this regard, in debug operating mode, before a timeout is reached, Helm fails reconciling with this error (and so does helm-controller):

client.go:464: [debug] Looks like there are no changes for Ingress "controller-api-server-ingress-http"
wait.go:53: [debug] beginning wait for 59 resources with timeout of 30s
wait.go:225: [debug] Service does not have load balancer ingress IP address: deis/deis-builder
wait.go:225: [debug] Service does not have load balancer ingress IP address: deis/deis-builder
...
# on like this until "context deadline exceeded"

The helpful information is hidden behind the --debug gateway, which is possible to access in helm-controller now by setting --log-level=debug on the entire helm controller.

So, filed under DX/UX note, I thought wouldn't it be great if with my HelmRelease that fails to reconcile, there was a debug flag in the API that I can use on a per-HelmRelease basis, to enable emitting debug entries into (at least the helm-controller logs?)

In thinking this through, it occurs to me now too that perhaps the most useful place to have the debug logs would be in the kubernetes event stream for the HelmRelease, and not just in the helm-controller logs... I have not tried to enable the log-level debug yet, not sure which of these it does right now. But either way, I can see wanting to enable --debug logs from Helm on just a particular HelmRelease vs doing so on every helm release that hc reconciles across the entire cluster at once.

Related conversation from CNCF slack: https://cloud-native.slack.com/archives/CLAJ40HV3/p1613417136364700?thread_ts=1613335794.255500&cid=CLAJ40HV3

[stefanprodan]

I think we should capture the wait logs and in case of a timeout write them in hc logs as errors.

[hiddeco]

I think we should capture the wait logs and in case of a timeout write them in hc logs as errors.

This is an option, we could also keep a buffer of e.g. the last 3 log lines and write them to the failure event.