Secure metrics endpoint with cntrlr-runtime metrics authz mechanics by anik120 · Pull Request #3660 · operator-framework/operator-lifecycle-manager

anik120 · 2025-09-23T19:15:37Z

Description of the change:

Implements native metrics authentication and authorization for OLM and catalog operators using controller-runtime
filters. Adds TLS support with automatic certificate management via cert-manager, replacing unprotected HTTP metrics
endpoints with authenticated HTTPS endpoints on port 8443.

Motivation for the change:

Current metrics endpoints are unprotected and accessible to anyone with cluster access, creating potential security
risks. This change secures metrics access by requiring proper Kubernetes RBAC authentication and authorization,
following the same pattern used by operator-controller for production deployments.

Architectural changes:

Integrates controller-runtime's WithAuthenticationAndAuthorization filter for metrics endpoints
Adds cert-manager integration for automatic TLS certificate lifecycle management
Implements dynamic certificate watching and reloading using existing filemonitor package
Disables HTTP/2 to mitigate known CVEs, enforcing HTTP/1.1 only
Updates both operators to use HTTPS (port 8443) with client certificate authentication
Maintains fallback to unprotected metrics when TLS is disabled for development scenarios

Testing remarks:

Reviewer Checklist

tmshort · 2025-09-23T20:54:39Z

/lgtm

anik120 · 2025-09-23T22:41:58Z

Looks like I have to make change to the metrics e2e tests coz the current ones are not authenticating themselves which is why they're failing. Great sign that the changes are working, working on the modifications to the e2e tests.....

Signed-off-by: Anik Bhattacharjee <anbhatta@redhat.com>

anik120 · 2025-09-24T14:55:31Z

            KIND_CLUSTER_NAME="kind-olmv0-${i}" \
            KIND_CREATE_OPTS="--kubeconfig=${E2E_KUBECONFIG_ROOT}/kubeconfig-${i}" \
-            HELM_INSTALL_OPTS="--kubeconfig ${E2E_KUBECONFIG_ROOT}/kubeconfig-${i}" \
+            HELM_INSTALL_OPTS="--kubeconfig ${E2E_KUBECONFIG_ROOT}/kubeconfig-${i} --set certManager.enabled=false" \


This presented itself as the easiest way to do all the tests we have for metrics (since these tests are about testing the metrics omitted, eg "creating a subscription emits these metrics", and not the security aspect of the endpoints).

anik120 · 2025-09-24T14:56:11Z

              path: /healthz
-              port: {{ .Values.olm.service.internalPort }}
-              scheme: {{ if .Values.olm.tlsSecret }}HTTPS{{ else }}HTTP{{end}}
+              port: {{ if .Values.certManager.enabled }}{{ .Values.olm.service.internalPortHttps }}{{ else }}{{ .Values.olm.service.internalPort }}{{ end }}


Which means the templates had to be updated to configure different endpoints based on the presence of the cert-manager

anik120 · 2025-09-24T14:57:08Z

+e2e-local: e2e-build kind-create e2e-local-deploy e2e
+
+.PHONY: e2e-local-deploy
+e2e-local-deploy: $(KIND) $(HELM) #HELP Deploy OLM for e2e testing (without cert-manager)


Also had to make a new deploy target to deploy olm without cert-manager for e2e testing

tmshort · 2025-09-24T19:52:27Z

/lgtm

openshift-ci · 2025-10-10T12:00:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: perdasilva

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [perdasilva]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…hanics (operator-framework#3660)" This reverts commit 589a5d0.

**Problem:** PR operator-framework#3660 introduced cert-manager as a hard dependency for OLM deployments, causing installation failures when cert-manager CRDs are not present: error getting resource "olm/olm-cert" with GVK "cert-manager.io/v1, Kind=Certificate": no matches for kind "Certificate" in version "cert-manager.io/v1" This is a breaking change for existing users who don't have cert-manager installed. **Solution:** Make secured metrics endpoints an opt-in feature by setting `certManager.enabled: false` by default in Helm values. Users who want authenticated metrics must explicitly enable cert-manager. **Changes:** - Set `certManager.enabled: false` in `deploy/chart/values.yaml` - Remove `cert-manager-install` dependency from `make run-local` - Remove `--set certManager.enabled=true` override from `make deploy` - Remove automatic cert-manager cleanup from `make undeploy` **Behavior:** - Default (cert-manager disabled): HTTP metrics on port 8080, no authentication - Opt-in (`certManager.enabled: true`): HTTPS metrics on port 8443 with authentication/authorization Fixes the breaking change introduced in operator-framework#3660 while preserving the secured metrics feature for users who want it.

**Problem:** PR #3660 introduced cert-manager as a hard dependency for OLM deployments, causing installation failures when cert-manager CRDs are not present: error getting resource "olm/olm-cert" with GVK "cert-manager.io/v1, Kind=Certificate": no matches for kind "Certificate" in version "cert-manager.io/v1" This is a breaking change for existing users who don't have cert-manager installed. **Solution:** Make secured metrics endpoints an opt-in feature by setting `certManager.enabled: false` by default in Helm values. Users who want authenticated metrics must explicitly enable cert-manager. **Changes:** - Set `certManager.enabled: false` in `deploy/chart/values.yaml` - Remove `cert-manager-install` dependency from `make run-local` - Remove `--set certManager.enabled=true` override from `make deploy` - Remove automatic cert-manager cleanup from `make undeploy` **Behavior:** - Default (cert-manager disabled): HTTP metrics on port 8080, no authentication - Opt-in (`certManager.enabled: true`): HTTPS metrics on port 8443 with authentication/authorization Fixes the breaking change introduced in #3660 while preserving the secured metrics feature for users who want it.

anik120 requested a review from joelanford September 23, 2025 19:15

openshift-ci Bot requested review from ankitathomas and perdasilva September 23, 2025 19:15

anik120 requested review from tmshort and removed request for ankitathomas and perdasilva September 23, 2025 19:15

anik120 force-pushed the native-metrics-authnz branch 3 times, most recently from a62b02f to f63b885 Compare September 23, 2025 20:28

anik120 mentioned this pull request Sep 23, 2025

DO NOT MERGE: demonstrating native metrics authn/authz using controller-runtime #3659

Closed

11 tasks

tmshort reviewed Sep 23, 2025

View reviewed changes

Comment thread deploy/chart/templates/_helpers.tpl Outdated

openshift-ci Bot assigned tmshort Sep 23, 2025

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 23, 2025

anik120 force-pushed the native-metrics-authnz branch from f63b885 to 7704b39 Compare September 24, 2025 13:44

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 24, 2025

anik120 force-pushed the native-metrics-authnz branch from 7704b39 to 9bac784 Compare September 24, 2025 13:51

Secure metrics endpoint with cntrlr-runtime metrics authz mechanics

63ab287

Signed-off-by: Anik Bhattacharjee <anbhatta@redhat.com>

anik120 force-pushed the native-metrics-authnz branch from 9bac784 to 63ab287 Compare September 24, 2025 14:32

anik120 commented Sep 24, 2025

View reviewed changes

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 24, 2025

anik120 mentioned this pull request Oct 2, 2025

WIP: OCPBUGS-59768: Authenticated metrics endpoints using controller-runtime auth filters openshift/operator-framework-olm#1111

Closed

perdasilva approved these changes Oct 10, 2025

View reviewed changes

openshift-ci Bot assigned perdasilva Oct 10, 2025

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 10, 2025

openshift-merge-bot Bot merged commit 589a5d0 into operator-framework:master Oct 10, 2025
14 checks passed

camilamacedo86 added a commit to camilamacedo86/operator-lifecycle-manager that referenced this pull request Oct 23, 2025

Revert "Secure metrics endpoint with cntrlr-runtime metrics authz mec…

c69f771

…hanics (operator-framework#3660)" This reverts commit 589a5d0.

rik-v mentioned this pull request Nov 3, 2025

Unable to install olm 0.36.0 to kind cluster #3675

Closed

anik120 mentioned this pull request Nov 11, 2025

fix: Make cert-manager optional for secured metrics endpoints #3706

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Secure metrics endpoint with cntrlr-runtime metrics authz mechanics#3660

Secure metrics endpoint with cntrlr-runtime metrics authz mechanics#3660
openshift-merge-bot[bot] merged 1 commit intooperator-framework:masterfrom
anik120:native-metrics-authnz

anik120 commented Sep 23, 2025

Uh oh!

Uh oh!

tmshort commented Sep 23, 2025

Uh oh!

anik120 commented Sep 23, 2025

Uh oh!

anik120 Sep 24, 2025 •

edited

Loading

Uh oh!

anik120 Sep 24, 2025

Uh oh!

anik120 Sep 24, 2025

Uh oh!

tmshort commented Sep 24, 2025

Uh oh!

openshift-ci Bot commented Oct 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

anik120 commented Sep 23, 2025

Uh oh!

Uh oh!

tmshort commented Sep 23, 2025

Uh oh!

anik120 commented Sep 23, 2025

Uh oh!

anik120 Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anik120 Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

anik120 Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

tmshort commented Sep 24, 2025

Uh oh!

openshift-ci Bot commented Oct 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

anik120 Sep 24, 2025 •

edited

Loading