test: Don't allow alerts to fire during a test run

The alert test has not been working and a PR was able to merge that
was firing alerts during the test run.

Fix the test logic based on:

1. Remove the filter on KubeAPILatencyHigh - no longer needed in 4.6+
2. Remove the filter on KubePodCrashLooping - no longer needed in 4.6+
3. Sort the metric by number of seconds the alert was firing

The bug that allowed the merge was the filter for KubePodCrashLooping,
to omit a series we must use `unless X`. If we reintroduce a filter we
should use that syntax.

Author: smarterclayton

test/extended/prometheus/prometheus.go

@@ -65,10 +65,9 @@ var _ = g.Describe("[sig-instrumentation][Late] Alerts", func() {
 		testDuration := exutil.DurationSinceStartInSeconds().String()
 
 		tests := map[string]bool{
-			// Checking Watchdog alert state is done in "should have a Watchdog alert in firing state".
-			// TODO: remove KubePodCrashLooping subtraction logic once https://bugzilla.redhat.com/show_bug.cgi?id=1842002
-			// is fixed, but for now we are ignoring KubePodCrashLooping alerts in the openshift-kube-controller-manager namespace.
-			fmt.Sprintf(`count_over_time(ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured|KubeAPILatencyHigh",alertstate="firing",severity!="info"}[%[1]s]) - count_over_time(ALERTS{alertname="KubePodCrashLooping",namespace="openshift-kube-controller-manager",alertstate="firing",severity!="info"}[%[1]s]) >= 1`, testDuration): false,
+			// Invariant: No alerts should have fired during the test run except the known alerts
+			// Returns number of seconds the alerts were firing
+			fmt.Sprintf(`sort_desc(count_over_time(ALERTS{alertname!~"Watchdog|AlertmanagerReceiversNotConfigured",alertstate="firing",severity!="info"}[%[1]s:1s]) > 0)`, testDuration): false,
 		}
 		err := helper.RunQueries(tests, oc, ns, execPod.Name, url, bearerToken)
 		o.Expect(err).NotTo(o.HaveOccurred())