test: Assess average series rather than max over the test window #25783
Conversation
With the recent increase in cluster metrics, some disruptive tests can trigger errors that result in a burst of cluster_operator_conditions or alerts series that then clear after the disruption. We want to run the full suite after we run a disruption, and in general we are concerned with average over max, so shorten the interval we check to 1h and calculate the average. When looking at telemetry from 4.7 CI clusters, the disruptive tests BRIEFLY peak at 600 series and then fall to 300 almost immediately after. Using the average, the total count is closer to 400 over the hour the tests run and that better represents the desired goal of the test (to limit average load, not spikes). Check the maximum as double the average.
openshift-ci-robot
added
the
approved
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: marun, smarterclayton The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@smarterclayton: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
5 participants
With the recent increase in cluster metrics, some disruptive tests
can trigger errors that result in a burst of
cluster_operator_conditions or alerts series that then clear after
the disruption. We want to run the full suite after we run a
disruption, and in general we are concerned with average over max,
so shorten the interval we check to 1h and calculate the average.
When looking at telemetry from 4.7 CI clusters, the disruptive tests
BRIEFLY peak at 600 series and then fall to 300 almost immediately
after. Using the average, the total count is closer to 400 over the
hour the tests run and that better represents the desired goal of
the test (to limit average load, not spikes). Check the maximum as
double the average.
Resolves failures encountered when attempting to run the disruptive
suite (destroy the cluster and recover) and then the conformance
suite. Subsequent PR will remove the skip on disruptive
@marun, @lilic