Namespace deletion fails if a cassdc exists

[adejanovski]

What happened?

If a namespace deletion is initiated while a cassdc exists in it, it will never go through because of a couple TerminalErrors that will prevent the reconciliation to go through as the finalizer will not be removed.
This is due to secrets being deleted as part of the namespace deletion, before the CassDc can be deleted.

What did you expect to happen?

Namespace deletion should go through with the cassdc being removed despite race conditions in object deletions.

How can we reproduce it (as minimally and precisely as possible)?

Create a cassdc in a specific namespace, then delete the namespace.
Terminal errors will be reported and namespace deletion won't go through.

cass-operator version

1.25.0

Kubernetes version

1.31

Method of installation

No response

Anything else we need to know?

No response

┆Issue is synchronized with this Jira Story by Unito
┆Issue Number: CASS-116

[burmanm]

Some clarifications from the chat to this. While namespace deletion should go through, we can't skip these errors. The process has to go through own its own to correctly delete the resources in correct order.

If there's some resource that gets deleted in incorrect order, that's something that needs to be fixed by the caller. This error scenario was created by an external tool that deleted Secrets before the CassDc object. We might need to investigate if we have to add more finalizers to prevent such issue from happening (but that could create issue with external secrets).

Otherwise the decommission of a Datacenter is not properly handled and it could result in issues in multi-DC configurations.