Skip to content

Bug 1957991: install/0000_90_cluster-version-operator_02_servicemonitor: Soften ClusterOperatorDegraded #554

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 1 commit into from
May 7, 2021
Merged

Bug 1957991: install/0000_90_cluster-version-operator_02_servicemonitor: Soften ClusterOperatorDegraded #554

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions install/0000_90_cluster-version-operator_02_servicemonitor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,9 +87,9 @@ spec:
or on (name)
group by (name) (cluster_operator_up{job="cluster-version-operator"})
) == 1
for: 10m
for: 30m
Copy link

@jianlinliu jianlinliu May 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 83 line has such message - "Cluster operator {{ "{{ $labels.name }}" }} has been degraded for 10 minutes". Should be also update "10 minutes" to "30 minutes"?

Copy link
Member Author

@wking wking May 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, nice catch. I've opened #556 to fix.

labels:
severity: critical
severity: warning
Copy link
Contributor

@smarterclayton smarterclayton May 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m ok with this. Warnings and degraded are for the morning. Do we have an estimate of how much churn this removes in the fleet?

Copy link
Member Author

@wking wking May 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poking around in recent Telemetry, the improvement for running clusters is expected to be small but noticeable. I expect impact during install to be larger, but haven't worked up numbers. We should be able to scrape it out of test-case JUnit post-merge.

- alert: ClusterOperatorFlapping
annotations:
message: Cluster operator {{ "{{ $labels.name }}" }} up status is changing often. This might cause upgrades to be unstable.
Expand Down