Bug 2015793: Run OLM's collect profiles job on the management cluster by awgreene · Pull Request #583 · openshift/hypershift

awgreene · 2021-10-21T13:27:55Z

No description provided.

openshift-ci · 2021-10-21T13:27:57Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2021-10-21T19:25:08Z

@awgreene: This pull request references Bugzilla bug 2015793, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

Bug 2015793: Run OLM's collect profiles job on the management cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

awgreene · 2021-10-21T19:25:30Z

Blocked until openshift/operator-framework-olm#208 is merged into 4.9

awgreene · 2021-10-21T19:26:32Z

/bugzilla refresh

openshift-ci · 2021-10-21T19:26:37Z

@awgreene: This pull request references Bugzilla bug 2015793, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @jianzhangbjz

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

awgreene · 2021-10-21T19:27:17Z

/hold until openshift/operator-framework-olm#208 merges

awgreene · 2021-10-21T19:30:40Z

        - name: profile-collector
          secret:
-            secretName: olm-profile-collector
+            secretName: pprof-cert


Unfortunately this is hardcoded in OLM

awgreene · 2021-10-21T21:58:43Z

/retest

awgreene · 2021-10-21T22:00:28Z

+	// Collect Profiles
+	collectProfilesConfigMap := manifests.CollectProfilesConfigMap(hcp.Namespace)
+	olm.ReconcileCollectProfilesConfigMap(collectProfilesConfigMap, p.OwnerRef, p.OLMImage, hcp.Namespace)
+	if err := r.Create(ctx, collectProfilesConfigMap); err != nil && !apierrors.IsAlreadyExists(err) {


Helpful note: The pprof config shouldn't be modified - it allows users to disable the cronjob and the generated configmaps.

I think we need to decide which user can modify the configmap. If it's the owner of the guest cluster (aka customer), then we should probably create this in the guest cluster and mirror its content back to the control plane. If it's meant to be configured by the cluster provider (aka management), then there likely should be a way to modify it via the HostedCluster API. I wouldn't expect anyone to make changes directly to configmaps in the control plane's namespace.

I believe the owner of the control plane should be responsible for modifying this configMap. It acts as an emergency escape hatch for disabling the cronjob, which runs on the management cluster.

If we want to configure this via the hostedCluster API, we can:

Modify the hosted control plane operator to create/update this configMap.

Introduce a knob on the hostedCluster API to set the value in the configMap.

awgreene · 2021-10-21T22:01:08Z

+	}
+
+	collectProfilesSecret := manifests.CollectProfilesSecret(hcp.Namespace)
+	olm.ReconcileCollectProfilesSecret(collectProfilesSecret, p.OwnerRef, p.OLMImage, hcp.Namespace)


Helpful note: The pprof secret shouldn't be modified - it allows the cronjob to frequently change the pprof secret.

jianzhangbjz · 2021-10-26T02:42:18Z

I test this PR, but failed to create the hosted cluster, details: https://bugzilla.redhat.com/show_bug.cgi?id=2015793#c1

awgreene · 2021-11-03T22:12:03Z

/retest

awgreene · 2021-11-17T19:03:46Z

/retest

Problem: In 4.9, OLM introduce a job that collects the data from the pprof endpoint ever 15 minutes. This job currently runs on the guest cluster but should run on the control plane. Solution: Run the profile collection job on the control plane.

awgreene · 2021-11-18T20:25:54Z

openshift/operator-framework-olm#208 has merged
/unhold

csrwng · 2021-11-19T11:22:41Z

/lgtm

csrwng · 2021-11-19T11:23:16Z

/approve

openshift-ci · 2021-11-19T11:24:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: awgreene, csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2021-11-19T12:01:19Z

@awgreene: All pull requests linked via external trackers have merged:

openshift/hypershift#583

Bugzilla bug 2015793 has been moved to the MODIFIED state.

Details

In response to this:

Bug 2015793: Run OLM's collect profiles job on the management cluster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2021

openshift-ci Bot requested review from alvaroaleman and enxebre October 21, 2021 13:29

awgreene changed the title ~~wip~~ Bug 2015793: Run OLM's collect profiles job on the management cluster Oct 21, 2021

openshift-ci Bot added the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label Oct 21, 2021

openshift-ci Bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Oct 21, 2021

awgreene force-pushed the run-collect-profiles-on-management-cluster branch from 4b7450e to 8fd08bc Compare October 21, 2021 19:26

openshift-ci Bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Oct 21, 2021

openshift-ci Bot requested a review from jianzhangbjz October 21, 2021 19:26

awgreene marked this pull request as ready for review October 21, 2021 19:26

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 21, 2021

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 21, 2021

awgreene force-pushed the run-collect-profiles-on-management-cluster branch from 8fd08bc to 80f5b8c Compare October 21, 2021 19:28

awgreene commented Oct 21, 2021

View reviewed changes

awgreene force-pushed the run-collect-profiles-on-management-cluster branch from 80f5b8c to a909db1 Compare October 21, 2021 19:41

awgreene commented Oct 21, 2021

View reviewed changes

awgreene force-pushed the run-collect-profiles-on-management-cluster branch from a909db1 to b108e6c Compare November 17, 2021 19:58

awgreene force-pushed the run-collect-profiles-on-management-cluster branch from b108e6c to 43de60d Compare November 17, 2021 20:09

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 18, 2021

openshift-ci Bot assigned csrwng Nov 19, 2021

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Nov 19, 2021

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 19, 2021

openshift-merge-robot merged commit e50ea8c into openshift:main Nov 19, 2021

sjenning mentioned this pull request Feb 15, 2022

use CreateOrUpdate for OLM collect profiles resources #1018

Merged

4 tasks

Conversation

awgreene commented Oct 21, 2021

Uh oh!

openshift-ci Bot commented Oct 21, 2021

Uh oh!

openshift-ci Bot commented Oct 21, 2021

Uh oh!

awgreene commented Oct 21, 2021

Uh oh!

awgreene commented Oct 21, 2021

Uh oh!

openshift-ci Bot commented Oct 21, 2021

Uh oh!

awgreene commented Oct 21, 2021

Uh oh!

awgreene Oct 21, 2021

Choose a reason for hiding this comment

Uh oh!

awgreene commented Oct 21, 2021

Uh oh!

awgreene Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

csrwng Nov 4, 2021

Choose a reason for hiding this comment

Uh oh!

awgreene Nov 4, 2021

Choose a reason for hiding this comment

Uh oh!

awgreene Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jianzhangbjz commented Oct 26, 2021

Uh oh!

awgreene commented Nov 3, 2021

Uh oh!

awgreene commented Nov 17, 2021

Uh oh!

awgreene commented Nov 18, 2021

Uh oh!

csrwng commented Nov 19, 2021

Uh oh!

csrwng commented Nov 19, 2021

Uh oh!

openshift-ci Bot commented Nov 19, 2021

Uh oh!

openshift-ci Bot commented Nov 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

awgreene Oct 21, 2021 •

edited

Loading

awgreene Oct 21, 2021 •

edited

Loading