[flaky] TestMetricsExport is super flaky

[vagababov]

metrics/resource_view_test.go - TestMetricsExport flakes most of the time.

/assign @jjzeng-seattle @evankanderson

here's a good example: https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_pkg/1666/pull-knative-pkg-unit-tests/1300475037081407489

[mattmoor]

cc @yanweiguo (community oncall)

[yanweiguo]

I ran the tests hundreds times with my own cluster and could only reproduce the timeout error once. I guess I have to send out a PR to run the CICD to debug.

[evankanderson]

/reopen

I still see issues, though with a different signature. Collecting signatures here.

=== CONT  TestMetricsExport
    resource_view_test.go:334: Created exporter at localhost:12345
    logger.go:130: 2020-09-27T02:15:16.884Z	INFO	metrics/exporter.go:155	Flushing the existing exporter before setting up the new exporter.
    logger.go:130: 2020-09-27T02:15:16.940Z	ERROR	websocket/connection.go:138	Websocket connection could not be established	{"error": "dial tcp: lookup somewhere.not.exist on 10.7.240.10:53: no such host"}
    logger.go:130: 2020-09-27T02:15:16.975Z	INFO	metrics/opencensus_exporter.go:56	Created OpenCensus exporter with config:	{"config": {}}
    logger.go:130: 2020-09-27T02:15:16.975Z	INFO	metrics/exporter.go:168	Successfully updated the metrics exporter; old config: &{knative.dev/serving testComponent prometheus 1000000000 <nil> <nil>  false 19090 false   {   false}}; new config &{knative.dev/serving testComponent opencensus 1000000000 <nil> <nil> localhost:12345 false 0 false   {   false}}

and then:

    resource_view_test.go:370: Timeout reading input
    resource_view_test.go:376: Unexpected OpenCensus exports (-want +got):
          []metrics.metricExtract(Inverse(Sort, []string{
          	"knative.dev/serving/testComponent/global_export_counts<>:2",
          	"knative.dev/serving/testComponent/resource_global_export_count<>:2",
          	`knative.dev/serving/testComponent/testing/value<project="p1",rev`...,
        - 	`knative.dev/serving/testComponent/testing/value<project="p1",revision="r2">:1`,
          }))

[knative-prow-robot]

@evankanderson: Reopened this issue.

Details

In response to this:

/reopen

I still see issues, though with a different signature. Collecting signatures here.

=== CONT  TestMetricsExport
   resource_view_test.go:334: Created exporter at localhost:12345
   logger.go:130: 2020-09-27T02:15:16.884Z	INFO	metrics/exporter.go:155	Flushing the existing exporter before setting up the new exporter.
   logger.go:130: 2020-09-27T02:15:16.940Z	ERROR	websocket/connection.go:138	Websocket connection could not be established	{"error": "dial tcp: lookup somewhere.not.exist on 10.7.240.10:53: no such host"}
   logger.go:130: 2020-09-27T02:15:16.975Z	INFO	metrics/opencensus_exporter.go:56	Created OpenCensus exporter with config:	{"config": {}}
   logger.go:130: 2020-09-27T02:15:16.975Z	INFO	metrics/exporter.go:168	Successfully updated the metrics exporter; old config: &{knative.dev/serving testComponent prometheus 1000000000 <nil> <nil>  false 19090 false   {   false}}; new config &{knative.dev/serving testComponent opencensus 1000000000 <nil> <nil> localhost:12345 false 0 false   {   false}}

and then:

   resource_view_test.go:370: Timeout reading input
   resource_view_test.go:376: Unexpected OpenCensus exports (-want +got):
         []metrics.metricExtract(Inverse(Sort, []string{
         	"knative.dev/serving/testComponent/global_export_counts<>:2",
         	"knative.dev/serving/testComponent/resource_global_export_count<>:2",
         	`knative.dev/serving/testComponent/testing/value<project="p1",rev`...,
       - 	`knative.dev/serving/testComponent/testing/value<project="p1",revision="r2">:1`,
         }))

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[evankanderson]

I'm slightly suspicious, seeing interleaved logging from different tests, if we're seeing side effects of the global monitoring singleton.

Unfortunately, it seems like it's hard to adjust our current prow test infrastructure to run these separately; let me look into doing it via GitHub Actions.