When a SIGTERM is sent to the Helm CLI during a deployment / upgrade, it leaves the cluster labels in a PENDING state, blocking further deployments

[jorgeclaro]

When a SIGTERM is sent to the Helm CLI during a deployment / upgrade, it leaves the cluster labels in a PENDING state, blocking further deployments.

We need to execute a helm rollback or uninstall the release to recover from this pending state.
It might be considered acceptable to development environments but not in production.

Output of helm version:

version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

[mattfarina]

Handling a SIGTERM isn't that straight forward for Helm. For example, let's say Helm has rendered the templates and sent them to the cluster but has not heard back. What state are the resources in? Were they accepted? This is not yet known and then a SIGTERM is sent. What should Helm do? There are various different points along the way that Helm would need to recover differently from.

This is certainly something that's open to look at improving. Do you have any suggestions on handling the different situations?

[bacongobbler]

FYI: #9180

[github-actions]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.