[RFC-0010] Introduce object-level workload identity for KMS decryption

[matheuscscp]

Depends on: fluxcd/pkg#919

Part of: fluxcd/flux2#5022

Fixes: #324

This PR also introduces a token cache in kustomize-controller for caching the cloud provider access tokens used for interacting with the respective KMS services. This token cache will report the following metrics:

• Counter gotk_token_cache_events_total
  • Help: Total number of cache retrieval events for a Gitops Toolkit resource reconciliation.
  • Labels: event_type, kind, name, namespace, operation. The values of event_type can be cache_miss or cache_hit. The value of operation is decrypt_with_<cloud_provider>.
• Counter gotk_token_cache_requests_total
  • Help: Total number of cache requests partioned by success or failure.
  • Labels: status. The values of status can be success or failure.
• Counter gotk_token_cache_evictions_total
  • Help: Total number of cache evictions.
  • No labels.
• Gauge gotk_token_cached_items
  • Help: Total number of items in the cache.
  • No labels.

I tested this PR in the following setups:

EKS:

• Single-tenant Pod Identity
• Single-tenant IRSA
• Multi-tenant IRSA
• Single-tenant IRSA with the SOPS feature for assuming a role
• Multi-tenant IRSA with the SOPS feature for assuming a role

AKS:

• Single-tenant workload identity
• Multi-tenant workload identity

GKE:

• Single-tenant workload identity
• Multi-tenant workload identity with direct access
• Multi-tenant workload identity with GCP service account impersonation
• .secretRef with GCP service account JSON key (this is unrelated to workload identity but this code path is changing, so I tested that it still works)

[stefanprodan]

LGTM

Thanks @matheuscscp 🏅